Friday Philosophy: Be A Hero – OR Be The Best

There is a crisis! The database is not responding, the apps can’t work and the business is suffering. Management are doing what management are there for – panicking and demanding “Someone Do Something!!!”.

Step forward a DBA who logs into the server, checks the alert logs, spots what is wrong and fixes it. The database starts processing requests, the applications are all working fine and the business is back on track. What a hero!

The Mantra of the DBA Hero

Such situations are not just the preserve of the database and the DBA of course. You get the hero System Administrators who step in and sort out the lack of storage space at 3am. Or the programmers who look at the code that has been running slow for weeks, that others have not been able to fix, and make it run in 5 minutes rather than 5 hours. All heroes who then bask in the gratitude of management and colleagues. Thank goodness for the Hero Developer/DBA/Sys Admin or whatever. You even get articles and advice on how to be The Hero in some quarters. I’ve even seen job ads like “Are YOU our next Developer Hero?!?”.

Only, 9 times out of 10, whatever was wrong should never have occurred. Yes, there are always going to be hard-to-predict failures or unavoidable catastrophes. But the majority of situations I have seen when the database goes seriously wrong, a critical program messes up badly, or a server goes offline, it is down to something that could and should have been spotted before hand – or never set up in the poor manner that it has been. These are things like Archive Redo log areas filling up, an “innocuous” network tweak taking out a major connection or a data processing program that goes wrong if it is run with no data to process. Just a little bit of thought or testing will avoid these sorts of issues.

As you get better at your role, and I mean really, truly better and not just older, you learn about better ways to do things. Either you make mistakes yourself and have to fix them (the best way to learn, even though it does not often feel like it), correcting something someone else did poorly or you read about how to set up systems to be more fault tolerant. You become more experienced with the tools and you grab hold of any new features that are going to make the systems run better. I’d hope we also all learn skills and working practices that help avoid disasters, such as proper testing methodologies (something that we seem to get less and less time & resource for) and proactive rather than reactive monitoring of our systems. If I am owning a database and it unexpectedly runs out of space for the data files or archive redo – I failed. The database did not, I did – as I know how to set up checks for those things.

The best technicians (in my opinion) that I have worked with are all like this. They don’t monitor for things that have gone wrong so much as monitoring for things that are going wrong. Every week or month they will change something that was OK – but it could be better, more resilient. The end result is a much quieter life and a substantially better service provided to the business.

But that’s where the rub is. That’s where things become unfair. When you are being the Best DBA or the Best Developer, things just work without a fuss. There are no disasters that impact the business and thus no need for The Hero. The systems run smooth & fast and management figure you are probably not doing that much. Heck, you seem to be spending all your time tinkering rather than fixing stuff! They often don’t get that the “tinkering” is what stops the disasters and the need for Heroes. That can lead to a lack of appreciation for what you are doing and it is extremely hard to see someone get praise for fixing an issue that they should never have let happen and even getting a pay rise and you get just a “yeah, thanks for, like, keeping the lights on I guess”.

I had this in spades in one role. I turned up and the critical databases would all be going down once or twice a week. People just accepted it. I worked on the problems, got my team together (and trained them!) and improved the service. For a couple of years I was a card-carrying member of the cape and spandex pants club. I was a Hero. We provided more services and incidents became very rare. And then they decided I was not doing enough. No problems were occurring so what did they need me for? After I calmed down from that (it took a few months) I decided I agreed with them and left. But I left behind a fantastic team and rock-solid systems. {It actually took me years to stop resenting the way they handled it, to be fair, but I never stopped being proud of what I did and that team}.

blowing you own trumpet can help – a little

So what do you do when you are being the best you can and not the hero and, as a result, you are fading into the woodwork? Well, I advise people to do several things, some of which you can see from a slide (taken from my “disasters” presentation) shown to the left. Record the number of incidents and how they go down as you improve things. Document improved up-time and better performance (which might be the same response time under higher workloads). Generally blow your own trumpet. However, it never seems to be enough to counteract the prestige people get from being the hero. It’s not “Right” but it just seems to be the way it is. I know some people take the other approach, which is to actually let (or even create?) disasters in which they can be heroes. After all, this is your career.

One fix is to just move on. After all, in the situation I described above I had actually completed my job – I had been hired to put in place a professional service and I did. So it would have been best if we had all been grown up and decided it was job done and time for me to move on. As a contractor/consultant this is a lot easier to do. Turn up at a client, be a hero for a while and then do your real job of making the systems solid. And then move on.

But not everyone has that luxury to move on. There may be few opportunities where you live or you would lose other aspects to your job that are very important (good child care is one example I have seen). Or moving roles might be something that gives you a lot of stress. So if you are “stuck” in your role and you are doing the best that you can, it is massively demoralising to fade into the woodwork as a result. What is the reward for all your work – pride and less interrupted nights are good but not getting the credit you deserve is hard.

But in the end I think you have a choice. Be a Hero or be The Best You Can Be. I have to aim for the latter, I can’t knowingly allow disasters without trying to at least warn management it could happen. And if you decide to be the best you can be perhaps you have to accept that, unless your management is very unusual, it may well mean less respect than you deserve. But *you* will know. I suppose it is a pride thing.

Are you a Hero? Or are you Simply The Best!

Advertisement