jump to navigation

Friday Philosophy – The Secret to Being a Good IT Manager June 3, 2011

Posted by mwidlake in Friday Philosophy, humour, Management.
Tags: , ,
10 comments

If you go into a book shop there will probably be a section on business and, if there is, there will almost certainly be a load of books on how to be a manager. Shelves and shelves of them. There is also a large and vibrant market in selling courses on management and aspects of management. I’ve been on a couple of such course and, if you can manage to be open minded whilst keeping a cynical edge, I think they can be useful.

However, I think I most of them are missing the key points and that if you can but hold on to the following extensive list of guiding principles you will be a good IT manager. Maybe even an excellent one :-):

  1. Your top priority, at all times, is to see to the best interests of your people.
  2. Whatever you develop, be it code, databases, network, a team of support staff – User Acceptance is paramount.
  3. You must find ways to deal with other teams and your own management hierarchy in such a way as to be allowed to do (1) and (2).
  4. That’s it.
  5. OK, if pushed, I’d say Never Lie. Maybe that’s just personal though, it’s because I don’t have the memory, audacity or swiftness of mind to pull it off. By not lying I don’t have to try and construct what I said to who and why.

I’m sure people could cite some other hard rules like “you must be within budget” or “you need to get buy-in to your vision” but I don’t agree. Budgets can be negotiated and the difference between those deemed visionaries and those deemed fantasists seems to be to me down to success and luck. Luck is luck and for success I refer you to points 1 through 5.

OK, maybe a final rule is:

  • Never ask for or aim for something that is not realistic.

So, I am now able to develop my team and my application and not expect to be able to spend half the company profit on the fastest box out there, as it is not realistic.

There are a shed load of other things that I think are important to helping you be a good manager, you know, techniques and methods for improving things, but nothing else that is key.

And it’s such a simple, small list even I can aim for it.

The shame of it is that I don’t think it’s enough to be developed into a book or a course so I can’t sell the idea. That and I’ve gone and given it away in this blog. Also, though I feel I can give points 1,2 and 5 a good shot, point 3 is way beyond me…possibly because of point 5… So I am not a great manager.

I’m going to hide behind this stout wall now, with my hard hat on, and wait to be told how naive I am…

Update – A couple of weeks later, Kellyn on her DBA Kevlar blog put similar sentiments to looking after your guys, more from the employee’s perspective and far better covered

Why given so many of us feel this way and want things to be this way…are they not?

If you Really Can’t Solve a “Simple” Problem.. March 11, 2011

Posted by mwidlake in Friday Philosophy, Perceptions, Testing.
Tags: , ,
12 comments

Sometimes it can be very hard to solve what looks like a simple problem. Here I am going to cover a method that I almost guarantee will help you in such situations.

I recently had a performance issue with an Oracle database that had just gone live. This database is designed to scale to a few billion rows in two key tables, plus some “small” lookup tables of a few dozen to a couple of million rows. Designing a system of this scale with theory only is very dangerous, you need to test at something like the expected volumes. I was lucky, I was on a project where they were willing to put the effort and resource in and we did indeed create a test system with a few billion rows. Data structure and patterns were created to match the expected system, code was tested and we found issues. Root causes were identified, the code was altered and tested, fine work was done. Pleasingly soon the test system worked to SLAs and confidence was high. We had done this all the right way.

We went live. We ramped up the system to a million records. Performance was awful. Eyes swung my way… This was going to be easy, it would be the statistics, the database was 2 days old and I’d warned the client we would need to manage the object statistics. Stats were gathered.
The problem remained. Ohhh dear, that was not expected. Eyes stayed fixed upon me.

I looked at the plan and I quickly spotted what I knew was the problem. The below code is from the test system and line 15 is the key one, it is an index range scan on the primary key, within a nested loop:

   9 |          NESTED LOOPS                       |                           |     1 |   139 |    37   (3)| 00:00:01 |       
* 10 |           HASH JOIN SEMI                    |                           |     1 |    50 |    11  (10)| 00:00:01 |       
* 11 |            TABLE ACCESS BY INDEX ROWID      | PARTY_ABCDEFGHIJ          |     3 |   144 |     4   (0)| 00:00
* 12 |             INDEX RANGE SCAN                | PA_PK                     |     3 |       |     3   (0)| 00:00:01 |       
  13 |            COLLECTION ITERATOR PICKLER FETCH|                           |       |       |            |          |       
  14 |           PARTITION RANGE ITERATOR          |                           |    77 |  6853 |    26   (0)| 00:00:01 | 
* 15 |            INDEX RANGE SCAN                 | EVEN_PK                   |    77 |  6853 |    26   (0)| 00:00:01 | 

On the live system we had an index fast full scan (To be clear, the below is from when I had tried a few things already to fix the problem, but that index_fast_full_scan was the thing I was trying to avoid. Oh and, yes, the index has a different name).

|   9 |          NESTED LOOPS                 |                           |     1 |   125 |  1828   (3)| 00:00:16 |
|  10 |           NESTED LOOPS                |                           |     1 |    63 |     2   (0)| 00:00:01 |
|* 11 |            TABLE ACCESS BY INDEX ROWID| PARTY_ABCDEFGHIJ          |     1 |    45 |     2   (0)| 00:00:01 |
|* 12 |             INDEX UNIQUE SCAN         | PA_PK                     |     1 |       |     1   (0)| 00:00:01 |
|* 13 |            INDEX UNIQUE SCAN          | AGR_PK                    |     1 |    18 |     0   (0)| 00:00:01 |
|  14 |           PARTITION RANGE ITERATOR    |                           |     1 |    62 |  1826   (3)| 00:00:16 |
|* 15 |            INDEX FAST FULL SCAN       | EVE_PK                    |     1 |    62 |  1826   (3)| 00:00:16 |

Now I knew that Oracle would maybe pick that plan if it could get the data it wanted from the index and it felt that the cost was lower than doing multiple range scans. Many reasons could lead to that and I could fix them. This would not take long.

But I could not force the step I wanted. I could not get a set of hints that would force it. I could not get the stats gathered in a way that forced the nested loop range scan. I managed to alter the plan in many ways, fix the order of tables, the types of joins, but kept getting to the point where the access was via the index fast full scan but not by range scan. I thought I had it cracked when I came across a hint I had not known about before, namely the INDEX_RS_ASC {and INDEX_RS_DESC} hint to state do an ascending range scan. Nope, no joy.

By now, 8 hours had passed trying several things and we had a few other people looking at the problem, including Oracle Corp. Oracle Corp came up with a good idea – if the code on test runs fine, copy the stats over. Not as simple as it should be as the test system was not quite as-live but we did that. You guessed, it did not work.

So what now? I knew it was a simple problem but I could not fix it. So I tried a technique I knew had worked before. I’d long passed the point where I was concerned about my pride – I emailed friends and contacts and begged help.

Now, that is not the method of solving problems I am writing about – but it is a damned fine method and I have used it several times. I highly recommend it but only after you have put a lot of effort into doing your own work, if you are willing to give proper details of what you are trying to do – and, utterly crucially, if you are willing to put yourself out and help those you just asked for help on another day.

So, what is the silver bullet method? Well, it is what the person who mailed me back did and which I try to do myself – but struggle with.

Ask yourself, what are the most basic and fundamental things that could be wrong. What is so obvious you completely missed it? You’ve tried complex, you’ve been at this for ages, you are missing something. Sometimes it is that you are on the wrong system or you are changing code that is not the code being executed {I’ve done that a few times over the last 20 years}.

In this case, it was this:

Here is my primary key:

EVEN_PK EABCDE 1 AGR_EXT_SYS_ID
EVEN_PK EABCDE 2 EXT_PRD_HLD_ID_TX
EVEN_PK EABCDE 3 AAAMND_DT
EVEN_PK EABCDE 4 EVT_EXT_SYS_ID
EVEN_PK EABCDE 5 EABCDE_ID

Except, here is what it is on Live

EVE_PK EABCDE 1 EVT_EXT_SYS_ID
EVE_PK EABCDE 2 EABCDE
EVE_PK EABCDE 3 AGR_EXT_SYS_ID
EVE_PK EABCDE 4 EXT_PRD_HLD_ID_TX
EVE_PK EABCDE 5 AAAMND_DT

Ignore the difference in name, that was an artifact of the test environment creation, the key thing is the primary key has a different column order. The DBAs had implemented the table wrong {I’m not blaming them, sometimes stuff just happens OK?}.
Now, it did not alter logical functionality as the Primary Key is on the same columns, but as the access to the table is on only the “leading” three columns of the primary key, if the columns are indexed in the wrong order then Oracle cannot access the index via range scans on those values! Unit testing on the obligatory 6 records had worked fine, but any volume of data revealed the issue.

I could not force my access plan as it was not possible – I had missed the screaming obvious.

So, next time you just “know” you should be able to get your database (or code, or whatever) to do something and it won’t do it, go have a cup of tea, think about your last holiday for 5 minutes and then go back to the desk and ask yourself – did I check that the most fundamental and obvious things are correct.

That is what I think is the key to solving what look like simple problems where you just can’t work it out. Try and think even simpler.

Friday Philosophy – Run Over by a Bus December 3, 2010

Posted by mwidlake in Friday Philosophy, Perceptions.
Tags: , ,
7 comments

I chaired a session at the UKOUG this week by Daniel Fink, titled “Stop Chasing your tail: Using a Disciplined Approach to Problem Diagnosis”. It was a very good talk, about having a process, an approach to solving your IT problems and that it should be a process that suits you and your system. All good stuff and I utterly agree with what he said.

But it was a passing comment Daniel made that really set me thinking. It was something like:

You should be considering how people will look after the system after you have gone, the classic ‘what will we do if you are hit by a bus’….. No, I don’t like thinking like that, that phrase… I prefer ‘after you win the lottery and retire to a great life’.

It just struck a chord with me. Mr Fink’s {and I do go all formal when I intend respect} take on this is a far more positive way of looking at the situation of leaving the system in a state that others can look after once you are no longer able to help. The “Bus” phrase is very, very common, at least in the UK and I suspect in the US, and it is a very negative connotation. “Make sure it all works as something nasty is going to happen to you, something sudden, like being smeared across the tarmac by 25 tons of Greyhound doing 50mph, something basically fatal so you can’t prepare and you can’t help any more”. So, not just moved on, but dead.

Daniel made me realise that we should be looking at this from totally the other perspective and that doing so is much, much, much better. “Make it work so that they love you, even when you have gone away to a happier situation – one involving no road-based unpleasantness at all”.

Everyone leaves their job eventually and I like to think it is often for more positive reasons. Like retiring, or a better job {better for you, but a real shame for your old company as they like you so much}, moving to a new area, attempting a dream. Yes, sometimes (depressingly often at present) it is because you get made redundant or things go bad with your managers, or HR take over the organisation. But even so, better to leave knowing you did so with your professional duty intact I think. It’s one way of winning in a losing situation.

If turning the “bus” metaphor into a “lottery” metaphor results in the response in your brain of “well, when I do leave rich and happy, I still want to leave a painful mess behind me” – then it may indicate that you better leave where you are working as soon as possible in any case? As it is not a good situation and you are deeply very unhappy about it.

Up until now I have sometimes used a far more gruesome but less fatal phrase for the concept of making sure things continue after you leave and can no longer help, which is “involved in a freak lawnmower accident”. As in, can’t type but not dead. I’m going to stop using it, I’ve decided that even with my macabre sense of humour, it really is not a good way to think about doing your job properly. Daniel, your attitude is better. Thank you.

Oh, if you went along to the conference you can get the latest version of Daniel’s talk slides from the UKOUG web site (try this link), otherwise, he has a copy here – pick “papers and presentations”. It has lots of notes on it explaining what the slides mean (ie, what he actually says), which I think is a very nice thing for him to have spent the time doing.

How NOT to present November 30, 2010

Posted by mwidlake in Meeting notes.
Tags: , ,
9 comments

I’m at the UKOUOG this week and, as ever, the presentations vary in quality. Most are excellent {or even better than that}, some are not. I was in one first thing this morning and, I have to say, it was rushed, garbled, unclear and there was a definite air of unease and panic. I’m not even sure the guy got to his big point and I could think of at least three major things he did not mention at all.

I think his main problem was just starting off in a rush and never settling down. You see, I was stuck on the top floor of my Hotel and had to run to the venue. Yes, the poor presentation was by me :-(.

I usually present well {modesty forbids me from saying I am a very good presenter – but modesty can take a hike, my ego knows I am capable of giving great presentations}. I am one of those lucky people for whom presenting has never been particularly frightening and, in fact, I find it easier to present to a group of people than talk with them.

But not today. I was already worried about the session, have been for weeks, as I was doing interactive demos. But last night I ran through it, wrote down the names of the scripts and the slide numbers so I could just bang through them and timed it all. 50 mins, I would skip one unneeded section. Calm. I got a reasonable night’s sleep, got up early and ran through it all one more time, making sure my Big Point demo worked. And it did. Yes.

Went down to breakfast, had breakfast and back to the room to pick up my stuff. And realised I was late. Less than 10 minutes to do the 5 minutes over to the venue. So I fled the room, stuffing my laptop in my bag. But not my notes. Or my conference pass. I did not think of this as I stood on the top floor of the hotel, I just thought “where are the lifts?”. They were all below me, ferrying hungry people to and from breakfast. After what seemed like an hour and was only 4 or 5 minutes I decided 16 flights of stairs was OK to go down and, to give me credit, I managed those stairs and the few hundred yards to the venue in pretty good time. I did pause for a few seconds at floor 7, I think, when I remembered my notes. Too late.

But I was now panicked and arrived as a dash. I had to mess about with the Audio Visual guy to get going and started 2 mins past my slot start – and then did the 5 minutes of non-relevant stuff I had decided to drop. It was game over from there, I was failing to find the correct scripts, I was skipping relevant sections and I was blathering instead of just taking a few seconds to calm down and concentrate.

Oh well, my first time in a large room at the UKOUG and I messed up. At least I had the key lesson drummed into me. TURN UP EARLY!!!!

Friday Philosophy – The Worst IT Person I Have Met October 15, 2010

Posted by mwidlake in Friday Philosophy.
Tags: , , ,
9 comments

A couple of weeks ago I extolled the virtues of someone I felt was a great person to work with. This week I’m going to do the opposite (and it will be interesting to see which posting gets more hits).

The worst person I have worked with in IT is Mick. I’ve only known a couple of Micks {and if you are one of them, but you don’t know Barry, you are not the Mick}. In an ironic twist of fate I met Mick at the same time I met the best person I have worked with, Barry. We were all in the same team you see, a UNIX sys admin team I got parachuted into. Maybe the vast difference between the two help make them so distinct in my mind.

Mick was very knowledgable and technically very capable. No, that is not fair, he was extremely good. He actually knew all this system admin stuff and several variations of shell programming, perl, C and a few other two-steps-from-assembler type languages. And he was an absolute and utter pain in the behind.

Barry and I did not know much (or in some cases, any) of this sys admin stuff. If we needed to do something and did not know how, Mick was supposed to show us. It worked something like this:

“Mick, I need to copy all the files that were changed last week from this directory on box X to box Y, keeping the directory structure – Can you help?”. Mick would not hear. He suffered from “intermittent deafness” – though he never missed any announcements about free food. You had to go and stand by Mick and wait for him to deem to notice you. If you actually interrupted him he would swear at you and utterly refuse to help, you had to wait quietly. If it was a good day he would deem this acceptable after a minute or two, but he would do his utmost to convey the impression he despised your lack of knowledge and your concerns were beneath his talents… but he would stoop to help.

You would repeat the task you were trying to do and, pausing only briefly to pour scorn on such a trivial thing, he would turn his back and start typing. He’d write a script to do it. “no, no, don’t write it, just tell me the basic commands and I’ll work it out!” No, he insisted on writing the script.

The script would be a thing to behold. Mick would write it in as few lines as possible and the least number of letters. For ages. Oh, he would have a working version in about the time it took Barry or I to explain the task, but he would not give you that version, oh no. He would ignore you until he had made all variables 1 character, took out all whitespace, replaced anything obvious with something obtuse, replaced a small chain of simple commands with one or two arcane commands. Every script was an attempt to win an “obfuscated code” competition. If we waited for the end result, it was impossible for Barry or I to decipher. The only benefit to the process was you would see the commands he was using and you could wander off and start with the unix Manuals yourself and get the job done.

He had other methods with which to demonstrate his greater worth.
Mick would agree to help (under duress of the boss telling him to do so) with an urgent task, but keep asking you to wait all day – then go home without doing his bit.
He seemed to love to intercept anyone coming to you for help, tell them he would sort out the problem for them – only to not. And then tell the user the next day that it was Barry or My problem to sort out. Correct, Mick would not have mentioned this to us.

Mick was fair though, he would treat everyone the same. With scorn. Any expertise in a field he did not know was unimportant and anyone with skills in his field was just competition to be shown who was best. Sadly, he usually was best, if best means biggest smartass.

Over time, as Barry and I learnt stuff (almost never from him), Mick became redundant. Not because we caught him up, not by a long way, but because no one else in the department would ask him anything. They would come to Barry and I. We might be slow and we sometimes screwed it up but we did not sneer and we fixed the problem in a way they could understand.

The reason Mick is the worst person I ever worked with is, unlike people who simply break stuff or lie about their skills or are stupid, he was actually very talented and capable – and yet took a perverse pleasure in not doing so. Mick would put effort into the art of maximizing his unhelpfulness. It was the difference between his potential to help and his drive to not do so that made it so hard for me to deal with him. I’d rather work with a talentless, idiot liar because at least you don’t need or expect much from them.

*sigh*

Friday Philosophy – The Best IT Person I Have Met September 24, 2010

Posted by mwidlake in Friday Philosophy, Perceptions.
Tags: , ,
3 comments

I’ve had the pleasure of working with and meeting a lot of talented and capable people in IT – some of them have even been nice people too :-) {In fact, most people I have met do not match that annoying myth about IT’s reputation for social awkwardness). However, for me one person sticks out in my mind as the best person I have worked with in IT.

It’s Barry. I’m pretty sure none of you have met Barry, and in fact as I knew Barry back in 1996 I’m not so sure I would recognise him (and I have no hope of remembering his last name) if I met him now.

Barry and I met when I got press-ganged into a Unix system administration team. I was just getting started at being an Oracle Performance person and knew very little about Unix Sys admin. But, for reasons I won’t go into now, I went home on the Friday as an “Oracle expert” and came in on the Monday to find my desk had been physically lifted and moved into the Unix sys admin corral and I was now a “Sys Admin not-expert”. My protestations were listened to – and then ignored, with the information that if I did “not knuckle down and get on with it”, the money would stop flowing. So, rather dazed and just a tad unhappy with the situation, I sat – and sulked – at my desk. And there, sat next to me, was Barry.

You are probably expecting me to now tell you that Barry knew Unix sys admin inside out and how he took me under his wing and showed me the ropes. Well, he didn’t. I have no idea where they got Barry from, I think he was a pro-C developer, but he had just been similarly abused by management and deposited into a team he had not signed up for. And he knew even less about being a sys admin than I did. I at least knew my way around a few monitoring commands like top, w, ps, “glance” etc.

Barry was also not very quick with IT. Don’t get me wrong, he was not stupid, but he was not one of these people who just had an affinity for technology and spent all his spare time building their own media server when CD-ROMS for PCs were still quite new. In fact, he seemed to find the whole of IT to be something of a challenge.

What Barry had though was enthusiasm, commitment and curiosity. Not in an annoying, bouncing all over the place crying “this is great” way, but more a case of “OK, server Falcon has run out of disk space. What can I do about it? How do I find out where the storage has all gone, who is using it and can I get it back off them?”. And he would set to. He’d start with what he knew (which was little more than the “Man” {Online Manual} command in the first week) and the bits he could suck out of my head and work through it. Every few minutes he’d be tapping me on the shoulder and saying things like “Look, you can get information about disk usage here, and map it to the real physical disks by greping for this”.

It was Barry’s attitude that made him stand out, and also his ability to infect you with the same attitude. I started off in that team furious and demoralized, determined to find a new position and resign ASAP. But Barry got over his annoyance and started working. He asked me for advice and discussed the issues over with me, even though I was as clueless as him. When he found something he showed me it. When I found something, he was keen to learn it.

Between us, we got by. We knew very little and it was hard work, but because Barry was not daunted and would keep working on the problem until he had it sorted, he dragged me along with him. Often I would still be there with him into the evening, sorting something out when everyone else had gone home. He did not just take on every problem people came to us with though, he would stick with what he felt was the biggest issue until it was sorted, and he would keep with it, and ask for help, and try what you suggested.

{oddly enough , the worst person I ever worked with was already in this team. Maybe that is why the others left and Barry and I were pulled in!}

It only lasted a few months as we both escaped to jobs more suited to our skills, but I learnt a few things. One was that a crummy job could be made a lot better just by your attitude and another was that some people (Barry, not me) had a real talent for enthusing people and thus getting things done. And also, that you did not have to be highly intelligent or knowledgeable to do a very good job. That’s lucky for me, then :-)

Team Work & The Science of Slacking July 23, 2010

Posted by mwidlake in Friday Philosophy, Management, Perceptions.
Tags: , , ,
add a comment

We all know that working in a team is more efficient than working on your own (and I did say a week or two back how I was enjoying the rare privilege of working in a team of performance guys). Many of us also know about team dynamics and creating a balanced team of ideas people, completer-finishers, implementers, strategists and so forth. Those of use who have been exposed to training courses or books on team management know all these good things about teams and how we are supposed to get the most out of them.

How many of us, though, have been introduced to the work of the French Agronomist Max Ringelmann and the aspect of teams named after him, the Ringelmann Effect? In summary the Ringelmann Effect proposses that people in teams try less hard than they do when working alone. Especially if they think no one is watching them.

Back at the start of the 20th century Ringelmann tested out his ideas using a tug-of-war experiment. He would get people to pull on a rope as hard as they could and record their efforts using a strain gauge. Then he would get them to pull on the rope as part of a team, from 2 to 8 people. As soon as people were part of a team, they pulled less hard. With two people in the team, each pulled 93% as hard as on their own, with three people this dropped down to 85% and with 4 it was just 77%. By the time there were 8 people in the team, effort was down to 50%.

This idea of shirking work more and more as the team increased in size became established in modern psychology and was given Mr Ringelmann’s name. Psychologists explain that when someone is part of a group effort then the outcome is not solely down to the individual and, as such, is not totally in their control. This acts as a demotivating factor and the person tries that little bit less hard. The larger the team, the greater the demotivation and the more significant the drop in effort. Ringelmann found that effort was down to 50% in a team of 8 so how bad can the impact of the team be? I think most of us have at least witnessed, and quite possibly been in, the position of feeling like just a cog in a massive corporate team machine. Thoroughly demotivating (though, of course, we all of us still tried as hard as we could, didn’t we?).

The effect is also know under the far more entertaining title of Social Loafing.

Monsieur Ringelmann was far kinder at the time and pointed out that these chaps pulling on the rope could well have been suffering from a lack of synergy. They had not been trained together to pull as a team so that could account for the drop in effort, they were not synchronising their effort.

However, in the 1970’s Alan Ingham in Washington University revisited Ringelmanns work and he was far sneekier. Sorry, he was a more rigorous scientist. He used stooges in his team of rope-pullers, blindfolds and putting the one poor person pulling for real at the front of the team pulling the rope. Thus he could record the effort of the individual. Ingham found that there was indeed a drop in efficiency due to the team not pulling as one. But sadly, this was not the main factor. It remained that the drop in effort was mostly down to the perceived size of the rest of the team. The bottom line was proven to be the human capacity to try less hard when part of a team and that the drop in effort was directly proportional to the size of the team.

We are of course not immune to this effect in the IT world and someone has even gone to the effort of checking that out, James Suleiman and Richard T Watson.

It seems the ways to reduce this problem are:-

  • Don’t give people boring jobs.
  • Don’t give the same job to several people and let them know they all have the same job.
  • Ask people how they are getting on and give them mini-goals along the way.
  • Atually reward them for success. Like saying “thank you” and NOT giving them yet another boring, hard job to do as they did the last one so well.

I think it is also a good argument for keeping teams small {I personally think 5 or 6 people is ideal} and split up large projects such that a single team can cope. Then give tasks to individuals or pairs of people.

If you like this sort of thing you might want to check out one of my first blog post (though it is more an angry rant than a true discussion ofthe topic) which was on the Dunning-Kruger effect, where some people are unaware of their own limitations – though I did not know it was called the Dunning-Kruger effect until others told me, which only goes to show that maybe I am not aware of my own limits… Read the comments or click through to the links from there to get a better description of some people’s inability to guage their own inabilities.

Friday Philosophy – The power of cooperation June 27, 2010

Posted by mwidlake in Friday Philosophy, Perceptions, performance.
Tags: , ,
3 comments

Being the person responsible for the performance of an Oracle-based system can be an oddly lonely working experience. It can also be a very gregarious position as you get to meet a lot of people and discuss lots of different aspects of many systems – all systems seem to have performance issues and so people come along to you, hoping you can identify a “work faster” database setting for them.

But you are often the only person who specialises in Oracle performance. You generally need to be in an organisation that is very large or where performance is key to success for there to be justification for dedicating a person to the topic. To have more than one person dedicated to performance your organisations has to have a very strong focus on getting the best performance out of the Oracle and related systems {or really, really atrocious performance issues :-) }. So usually there is no one else around who is as experienced (or more so) as yourself to discuss such things over with or ask for a second opinion.

Which is why I am very lucky at the moment. I’m working in a team of oracle performance people. There are 2.5 of us (one is a manager with other responsibilities, so he only counts as half). Being able to say “Hey, Dave, What do you think of the wait times on scattered reads?” or “how in heck do I force this nested subquery on a view to use a hash join?” and get some other ideas is very valuable.

What is also interesting is how opinions and preferred techniques on tuning can be different and just as valid. As an example, last week I was working on a poorly performing statement. I was at home and it was the evening, so I was not communicating with the rest of the team. I managed to get the code down from 40 minutes to just under 20 by using a combination of a LEADING and USE_HASH hint. I sent the code back to the user. Only to see that within thirty seconds of each other my colleague Graeme had also sent the user a response, again getting the code down to around 20 minutes. Graeme had pulled a chunk of the code into a subquery factoring “WITH” clause and added cardinality hints. Totally different changes.

So Graeme and I then had a “philosophical” discussion about the different changes {“Mine is best” – “No! Mine is, yours is all bloated and complex”- “Your hint is less future-flexible!!!”}. Only joking, we actually discussed the changes and why we each chose what we did. Graeme commented that is showed that tuning was an art and not a science and I countered that it was a science, as we had both identified where we felt the execution plan could be improved but used different techniques to get there. The thing is, Oracle is so complex and has so many options to influence the optimiser that you have flexibility to chose different tools and techniques.

We had both identified the same main improvement but had each come up with different tweaks for later in the plan.

The end result was that we went with Graeme’s main plan {he is bigger than me} but we pulled in my tweak. That bought the execution time down to around 10 minutes, so about four times faster over all and twice as fast of either of us alone. That is one of the advantages of not working alone.

We also then discussed how we could get this code down to seconds with the use of either Materialized views or changing the process that generated the report to do so incrementally and store the daily results. Until one of us realised we had reached the boundary of compulsive tuning disorder. The report running in 10 minutes was enough improvement to satisfy the business, the report was only going to be run over the next couple of months, so spending a day or two re-working it further was going to be fun – but of no advantage to the business. We told each other to finish for the day. So another advantage of not working alone is that not only do you get more technical input but your help prevent each other losing sight of the overall aim.

It really does help to have two people working on the same area.

{There is a sneaky way of getting beyond being a lone performance specialist. If you are in an organisation long enough you can usually find some other idiot who is silly enough to want to learn more about performance and you can train them up. It does not take long before they know enough to start coming up with things you never thought of. Oracle is, after all, full of many ways to do the same thing and you can’t know it all}.

Friday Philosophy – Madness demands Attention May 28, 2010

Posted by mwidlake in Architecture, Friday Philosophy.
Tags: , ,
2 comments

Many years ago I had a good friend who was a psychiatric nurse. We were talking about his job once and he was saying how some patients just took up much more time than others. These were generally the ones who would be deemed “the most mad” in a non-clinical manner {and is pretty much how my friend the psychiatric nurse put it}. These patient’s actions or need for intervention would put demands on the staff far more than other patients. As a result, all the nurses tended to get to know (or know of) such patients better than others.

I thought of this the other day when a few of us were talking about some awful bit of application we were concerned about. This thing inserts rows from a MSSQL database into a table in an Oracle database. Triggers on the initial table fire and populate another table, in a 1-to-many relationship. This second table also has a trigger on it that further inserts into another set of tables. A regular process then aggregates this data – and sticks it back into the MSSQL database it came from.

Said process is done as a single transaction for all rows inserted for the day. Irrespective of the growth in rows. Or the fact that one source “application” has grown to 10 and soon will grow to 50. All rows in one transaction. The intermediate tables are never cleared out and get bigger and bigger. No one else needs any of this data in the Oracle side of the system.

There are several “madnesses” to this process – why put it into Oracle only to put it back in the source system, why use a busy production system to hold and process transient data, why no clean up, why are the records not processed in batches, the cascading triggers magnifies transaction data volumes…

This process is well-known in our group. I’ve been involved. Both the guys I was talking to have been involved. I can see from my desk 4 or 5 other people who have been roped into bullying this process thought before now. In fact, I reckon half the department have had to work on this damned thing at some point.

Can you see why I was reminded of my conversation with my old nurse friend?

The application is simply mad. And as a result it is demanding not only on our database but on all of the team, as so many of us have had to get involved working on it. We have all got to know it so well.

I’m glad to say that treatment for the application is planned and, hopefully, it will soon be a lot happier.

So will we.

Enrichment of the Code Monkey Environment April 15, 2010

Posted by mwidlake in Friday Philosophy.
Tags: ,
1 comment so far

In animal welfare there is a concept called “Environment Enrichment”. This is where you do things to make the life of the animal in captivity more enjoyable. Often it is centred on how you feed the animal – either hiding their food so they need to forage or presenting it in a way that better matches their natural behaviour . So, for example, you might take your gerbils out of their cage and then hide seeds and bits of fruit around the place {or get Ben Fogle dragging chunks of meat behind a Land Rover through the lion enclosure at Longleat Safari Park so the lions can chase after it}. Or you might buy some of that expensive, transparent, coloured plastic tubing that plugs together in little mazes – as Gerbils of course run around pink, blue and orange transparent tubes in the wild…

Where I am working at the moment, Environment Enrichment for the captives is achieved by relocating the coffee cups in the kitchen overnight. We all turn up in the morning and go up in little groups to play the “open the cupboards in random” game in order to track down the mugs. Never the same cupboard two days in a row – except occasionally just to keep us on our toes.

We also have several fruit bowls and someone occasionally fills them with a selection of fresh fruit. Of course, as we are Code Monkeys, the bananas are most popular. What most of my fellow inhabitants have not yet worked out is the occasional bonus cache in the pot that looks like a flower vase. It’s usually apples.

The best bit of the environment enrichment is at the end of the week though. 5pm Friday, with the gentle tick of cooling laptops and the gentle lowering drone of hard disks winding down, and the glow of the LCD sceens dimming, the oasis is opened. A bottle of beer or can of coke can be obtained and you all stand around and discuss anything but computers. Well, OK, maybe computers a bit, but they have to be non-work computers.

Follow

Get every new post delivered to your Inbox.

Join 171 other followers