jump to navigation

Friday Philosophy: Be A Hero – OR Be The Best August 26, 2016

Posted by mwidlake in Friday Philosophy, humour, Perceptions, working.
Tags: , , ,

There is a crisis! The database is not responding, the apps can’t work and the business is suffering. Management are doing what management are there for – panicking and demanding “Someone Do Something!!!”.

Step forward a DBA who logs into the server, checks the alert logs, spots what is wrong and fixes it. The database starts processing requests, the applications are all working fine and the business is back on track. What a hero!

The Mantra of the DBA Hero

The Mantra of the DBA Hero

Such situations are not just the preserve of the database and the DBA of course. You get the hero System Administrators who step in and sort out the lack of storage space at 3am. Or the programmers who look at the code that has been running slow for weeks, that others have not been able to fix, and make it run in 5 minutes rather than 5 hours. All heroes who then bask in the gratitude of management and colleagues. Thank goodness for the Hero Developer/DBA/Sys Admin or whatever. You even get articles and advice on how to be The Hero in some quarters. I’ve even seen job ads like “Are YOU our next Developer Hero?!?”.

Only, 9 times out of 10, whatever was wrong should never have occurred. Yes, there are always going to be hard-to-predict failures or unavoidable catastrophes. But the majority of situations I have seen when the database goes seriously wrong, a critical program messes up badly, or a server goes offline, it is down to something that could and should have been spotted before hand – or never set up in the poor manner that it has been. These are things like Archive Redo log areas filling up, an “innocuous” network tweak taking out a major connection or a data processing program that goes wrong if it is run with no data to process. Just a little bit of thought or testing will avoid these sorts of issues.

As you get better at your role, and I mean really, truly better and not just older, you learn about better ways to do things. Either you make mistakes yourself and have to fix them (the best way to learn, even though it does not often feel like it), correcting something someone else did poorly or you read about how to set up systems to be more fault tolerant. You become more experienced with the tools and you grab hold of any new features that are going to make the systems run better. I’d hope we also all learn skills and working practices that help avoid disasters, such as proper testing methodologies (something that we seem to get less and less time & resource for) and proactive rather than reactive monitoring of our systems. If I am owning a database and it unexpectedly runs out of space for the data files or archive redo – I failed. The database did not, I did – as I know how to set up checks for those things.

The best technicians (in my opinion) that I have worked with are all like this. They don’t monitor for things that have gone wrong so much as monitoring for things that are going wrong. Every week or month they will change something that was OK – but it could be better, more resilient. The end result is a much quieter life and a substantially better service provided to the business.

But that’s where the rub is. That’s where things become unfair. When you are being the Best DBA or the Best Developer, things just work without a fuss. There are no disasters that impact the business and thus no need for The Hero. The systems run smooth & fast and management figure you are probably not doing that much. Heck, you seem to be spending all your time tinkering rather than fixing stuff! They often don’t get that the “tinkering” is what stops the disasters and the need for Heroes. That can lead to a lack of appreciation for what you are doing and it is extremely hard to see someone get praise for fixing an issue that they should never have let happen and even getting a pay rise and you get just a “yeah, thanks for, like, keeping the lights on I guess”.

I had this in spades in one role. I turned up and the critical databases would all be going down once or twice a week. People just accepted it. I worked on the problems, got my team together (and trained them!) and improved the service. For a couple of years I was a card-carrying member of the cape and spandex pants club. I was a Hero. We provided more services and incidents became very rare. And then they decided I was not doing enough. No problems were occurring so what did they need me for? After I calmed down from that (it took a few months) I decided I agreed with them and left. But I left behind a fantastic team and rock-solid systems. {It actually took me years to stop resenting the way they handled it, to be fair, but I never stopped being proud of what I did and that team}.

blowing you own trumpet can help - a little

blowing you own trumpet can help – a little

So what do you do when you are being the best you can and not the hero and, as a result, you are fading into the woodwork? Well, I advise people to do several things, some of which you can see from a slide (taken from my “disasters” presentation) shown to the left. Record the number of incidents and how they go down as you improve things. Document improved up-time and better performance (which might be the same response time under higher workloads). Generally blow your own trumpet. However, it never seems to be enough to counteract the prestige people get from being the hero. It’s not “Right” but it just seems to be the way it is. I know some people take the other approach, which is to actually let (or even create?) disasters in which they can be heroes. After all, this is your career.

One fix is to just move on. After all, in the situation I described above I had actually completed my job – I had been hired to put in place a professional service and I did. So it would have been best if we had all been grown up and decided it was job done and time for me to move on. As a contractor/consultant this is a lot easier to do. Turn up at a client, be a hero for a while and then do your real job of making the systems solid. And then move on.

But not everyone has that luxury to move on. There may be few opportunities where you live or you would lose other aspects to your job that are very important (good child care is one example I have seen). Or moving roles might be something that gives you a lot of stress. So if you are “stuck” in your role and you are doing the best that you can, it is massively demoralising to fade into the woodwork as a result. What is the reward for all your work – pride and less interrupted nights are good but not getting the credit you deserve is hard.

But in the end I think you have a choice. Be a Hero or be The Best You Can Be. I have to aim for the latter, I can’t knowingly allow disasters without trying to at least warn management it could happen. And if you decide to be the best you can be perhaps you have to accept that, unless your management is very unusual, it may well mean less respect than you deserve. But *you* will know. I suppose it is a pride thing.

Are you a Hero? Or are you Simply The Best!



1. Narendra - August 26, 2016


That is all I can say….
The worst part is, when “management” forces a labyrinth of procedures and still the requirement for heroes never goes away.
Personally, I am scared that recently I am reaching that stage of frustration/irritation at the state of affairs very quickly…within first 6 months

mwidlake - August 26, 2016

Thanks Narendra – and yes, sometimes companies do not help themselves by making it so hard to change things, eg only allow a change ONCE a disaster has occurred, that you are trapped being a hero as proactive improvements are so hard to get done. That’s a whole new post 🙂

2. Dom Brooks - August 26, 2016

Sadly, if you go around fixing everything before it happens, it will never be fully appreciated.
Fortunately frequently-found change management practices mean that you need it to break before the change can be justified / approved in a timely fashion.

mwidlake - August 26, 2016

So when we all complain about the awful, stifling, Change Management process, we should really be grateful – for the heroic opportunities it provides? 🙂

3. Neil Chandler - August 26, 2016

You just stop it now! Do you know what sort of an afternoon I’ve had? And you write this.

Well the chance of hero-ness improved greatly when I finally had enough and just let all the stupid change through. My, that’s a lot of BasicFile BLOBs in the 12C DB storing all those character messages. Still, you’ll hit your deadlines if I release it…

Heros get bonuses.

[comment are fictional, and do not represent and of my clients past, present or future. OK. I made it up. Honest.]

mwidlake - August 26, 2016

Well, the blog is kind of your fault as you (correctly) pulled me up for a tweet about being a DBA hero – and how the best DBAs do not get the recognition they deserve as they only really need to be heroes (but cometh the hour, cometh the true DBA/sysadmin/developer hero)

4. Piet de Visser - August 26, 2016

I can relate to that, being part of a “response team” at one customer.. But one thing my boss keeps repeating: “Nobody wants to Pay (or budget) for Pro-Active”

mwidlake - August 26, 2016

It something I struggle with on the times I represent a consultancy – “don’t fix what they have not paid us to fix!”. But, but, but…

daithiwalker - August 27, 2016

Been there alright. So frustrating when they only want to pay the minimum but you know they really need so much more. I find it very difficult to let these things go.

5. Jeffrey Kemp - August 27, 2016

I’m not a DBA but your article made me appreciate my current client’s IT leadership a bit more. Whenever there’s a disaster, their first priority is to get the situation under control. Then, the next day after the dust has settled, the questions start getting fired: “what failed, and why?” “what could we have done to avoid this? what can we do now to avoid future failures of this type?”. Usually it gets CC’ed to us developers so we can consider how the app design could be improved.

As you say – I think the most important thing any DBA can do in this area is to make sure they are appropriately “marketing” what they are doing to IT director.

Perhaps, sometimes, instead of just going ahead and making that small change that could have a significant positive impact on performance or reliability, raise it as a recommended change request, push it through the bureaucracy to have it “signed off” etc. – the painful process might slow things down, but it raises awareness of what you’re doing.

Mind you – unfortunately, in some orgs their “change approval” process is dysfunctional and this tactic won’t work.

6. Noons - August 28, 2016

And then you get “the database must be down, I can’t login” complaints.
Which invariably mean the moron has forgotten his/her password and the fault is – obviously – with the database! I don’t reply to them anymore…
10 years in this place, 30 odd Oracle dbs to take care of, one single incident of lost data on a very old and illegal db installed by some idiot on an illegal Windows box. Long since fixed with no data loss after all.
Outages? One (1) unscheduled. In 10 years. OS bug.
And of course what is the very first suspect when there is a problem anywhere?
“It’s the database”, of course!
I’m sorry, but companies get the IT management they deserve…
I still refuse to install or consider installing inherently unstable features of Oracle such as RAC. The last IT damager who tried to force me into “stretch RAC” got the boot he richly deserved.
The dbs I take care of are patched up, quite a few on 12c and stable as rocks.
Some do over 10 TB of I/O per day, with no failures ever.
Same with MSSQL – 360 last count, some on 2008, others on 2012.
Yeah, I’m an old fart. See if I care? 🙂

mwidlake - August 28, 2016

Nice Rant Noons 🙂

Neil Chandler - August 29, 2016

No No No, Noons! You’ve got it all wrong! Add RAC. Run on unusual filesystems and rare O/S’s. No referential integrity. Store everything in BLOBs. Gather thousands of histograms and revel in having SQL Plan Directives on every column and join. Implement multi-master Goldengate and take no backups!

Without all the moronity out there doing that stuff, I’d be unemployed… 🙂

Noons - August 30, 2016

Tsk-tsk!… You forgot the essential:
“create all indexes as bitmap indexes for the DW – they make all queries a lot faster!”.
Little small details like: “how does all that data get into the DW in the first place?” and the “ET” portion of “ETL” are just to be ignored or glossed over!
Ah yes: and don’t even think of using join predicates in your SQL joins: any duplicates are exclusively eliminated on the spot via a DISTINCT on every SELECT, including subqueries!
Better yet: do a UNION ALL and THEN add the DISTINCT to the result!
There: a job for life!

7. jgarry - August 29, 2016

Just pet your red stapler, rock back and forth, and nod knowingly that you’ve got things well in hand despite damagement.

mwidlake - August 29, 2016

My stapler is yellow. And it’s my friend.

Mary Elizabeth McNeely - August 31, 2016

Maybe you’ll find some large denomination traveler’s checks under the office door someday, as a reward for your unsung heroism. Just don’t burn the office down. It doesn’t have to go EXACTLY like the movie.

8. Mary Elizabeth McNeely - August 31, 2016

Excellent. That’s exactly how it works. No one notices when the database is UP.

9. exagriddba - September 5, 2016

Mary Poppins was not a hero, she was simply the best. This is undisputed. A wind storm carried her to 17 Cherry Tree Lane, where she was needed, but not forever. When her job was done, she opened her umbrella and let the wind carry her away to the next place.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: