jump to navigation

Friday Philosophy – Identifying and Nullifying Fake Urgency April 20, 2012

Posted by mwidlake in Friday Philosophy, Management.
Tags: , ,

You know how it goes. You get a call/mail/text with something along the lines of “I need to know all the details of customer orders placed on Tuesday 7th by customers based in Botswana – and I need it ASAP, by end of play today at the latest”. So you skip lunch, drop that task you have been trying to get around to doing all week and work out how to resolve the issue that has just been dropped on you. It takes a lot of effort and you finally get it sorted out around an hour after you told your girlfriend/boyfriend/cat you would be leaving the office that day – and mail it off to the requestor. You might even call them to let them know it is done, but oddly they don’t answer.

Next day, you see the guy who wanted this urgent request and ask if it was what they wanted “Oh, I have not looked at it yet – but thanks for doing it.”

NO! “Thanks” does not work in this situation. I’d have more respect for this guy if he laughed at me and said “got you again, sucker”. Many of you know what I mean don’t you – if you are in a support-type-role, this can be a big part of your life.

I had a job years back that seemed to consist 90% of such tasks. I was the development DBA team leader responsible for testing, validating and promoting code to production. Everyone’s changes were Urgency Level 1, to be done as an emergency release and many could not be put in place until after 5pm. I’d be sat there at 18:30 in a massive but virtually empty office, applying changes along with one or two of my guys. Everyone else had gone home. This was not once or twice a month, it was 4 or 5 times a week. What are you to do?

Well, I came up with one tactic that seemed to work pretty well.

Anyone who asked for an emergency change had to be there, on site, available when the change was done.
There were of course cries of protest and people stated it was ridiculous that they had to be there, they were not needed, the change had been tested thoroughly {oh how I laughed at that – a thoroughly tested “emergency” change huh?}. No, I replied, you had to be there in case it went wrong as it’s your system, your data and, frankly, your emergency. If it is not urgent enough for you – the guy wanting it to be done – to be inconvenienced, well it sure as hell is not urgent enough to inconvenience me. “You can call if there are problems” – What, after you have escaped the locality? Maybe turned off your phone? And if I get you , I have to wait for you to come back in? No no no. Urgent emergency now equates to presence in office. After all, I’ll be there.

I stuck to my rule. If the requester could not be bothered to stay, I downgraded the request to “Planned” and put it through the CAB process. If the requester dumped on one of their team and made them stay, I mentally marked them half a point down and factored it in next emergency.

The change was remarkable. I was no longer in the office on my own every evening. I was not there with someone else either. I was simply not there as, when you made the emergency a little bit inconvenient to the requester, it magically stopped being an emergency.

There was another change. Less cock-ups. Seeing as these changes now went through the CAB process and slightly more testing {like, some testing} the duff changes were more likely to be detected before they caused damage. My bosses went from regarding me as “not a team player” to “Not a team player – but we kind of get your point now”.

So my advice is, if someone wants to try and make something your emergency, find some way of making sure it remains inconvenient to them. If they are willing to put up with the inconvenience, then it is a real emergency and you need to crack on with it.

OUGN 2012 Second Day – Out on the Open Seas March 23, 2012

Posted by mwidlake in Meeting notes.
Tags: , , ,
1 comment so far

As I said yesterday, I am not one for blogging about conferences. So what the heck, I’ll do another post on this one🙂.

You might have picked up on the fact that I am not very good on the sea and have a lot of nervousness about this idea of spending 2 days on an ocean-going liner. So today is the day we move over to being on the water, the OUGN conference proper. I’m delighted to say I am not the only one nervous about this boat lark, Marie Colgan {or, as I should be calling her, “The Optimizer Lady” – since Doug Burns christened her back at the UKOUG 2011 conference} feels the same. There is nothing better to reduce one’s apprehension about something than finding someone else who is just as if not more anxious about it. I suspect this is an evil way to think but I can’t help it.

The day went well, my overview of VLDBs could have maybe been a little more polished – I think I was trying to cover too many topics and not spending enough time on anything, apart from “why a small number of spindles is bad” which I spent too long on. But I did make two points that I think are major considerations with VLDBs and are actually not very technical. (1) If you are going to have to look after a massive database, try very, very, very hard to keep everything as simple, standardised and controlled as you can. (2) Find other people to talk to about VLDBs. Most companies that have them are reluctant to talk, either because of the industry type (Defense, Banks, Pharma) or a perceived giving away of corporate advantage (Banks, Pharma, Utilities and pretty much all companies really).

Anyhow, having done my talk I was now free for the next 2 days to enjoy other talks, socialise, relax – and keep internally checking if I felt nauseous or not. The sea has turned out to be very calm and the boat very stable. But I keep checking and, of course, that makes me feel a little anxious – and thus ill.

However, I have to say that the travel sickness pills I have been taking do seem to be very effective. They are effectively in making me feel woozy. But, and this is important, not ill. I’m having to rely on beer for the latter.

One thing I really, really like about this conference. Everyone is stuck on a boat, they can’t escape. Which means you get to see lots of people all day and it makes the whole thing have a nice sense of community.

Right, Maria is about to present again. I’m going to go and sit in the front row and sway lightly from side to side. Apparently it makes her feel even worse…

OUGN 2012 First Day – First Panic March 21, 2012

Posted by mwidlake in Meeting notes, Perceptions, Private Life.
Tags: , ,
add a comment

I’m not really one for blogging about conferences – I mean who cares what someone else saw being presented? But this is the first time I have stopped moving long enough (and been in contact with the internet world) to blog and my brain is too fried to do a technical one.

The journey here was very smooth and the train from the airport to central Oslo makes the ratty, tatty, confined coaches of the UK look as awful as they really are. So I arrive in the central station and decide I will need some local Kroner to pay the taxi to the hotel. So I find an ATM, put in the card and ask for money. Card refused. Huh? But I rang my bank at the weekend and let them know there would be transactions from a foreign country (after some issues a couple of years back my wife always lets the bank know when she will be away and she made me do the same). I tried another ATM from a different company. Refused. OK, damned bank, I’ll use my other card. Refused. Errrr…… In a foreign country, no idea really where I am, not got any local currency. Not looking good.

I wander into Oslo looking for a real, proper bank. Most Norwegians speak perfect English, maybe if I still have problems I can go in and ask. Find bank, go to ATM, about to ask for money….Notice the figures being suggested on this ATM are a lot smaller than the last place. Yes, I had my mental decimal place in the wrong location and I had been asking for almost £1,000 rather than £100. No wonder the request got refused (I keep my daily limit low, it stops be buying too much rubbish on any given day).

So, I head off to the event, get there for Lunch and have a great afternoon. My presentation on IOTS went well and, as I said yesterday, I think I’ll put it to sleep for a while now.

I saw Harald Van Breederode talk about Oracle Database Smart Flash Cache before I did my slot and I always like to hear Harald talk. It was good as ever, but I found myself not so much interested in the idea of using SSD-type storage as an extra “slow SGA” extension (as opposed to a “fast storage” extension) but more that in 3 or 4 years, memory-based storage will be the default and a whole swathe of my knowledge will once more become redundant. I mean, how important will it be to keep physical reads down via things like IOTs once physical reads are relatively cheap? You won’t really care much about expanding your SGA with a secondary cache when you have 4TB of main memory and 100,000 IPS (inputs per second, we will have to see how much faster Output can be made with memory-based storage).

This highlights one of the things I really like about conferences, meetings and chatting to fellow techies in the pub. The actual main topic or point might well be interesting but the secondary thoughts and ideas can be just as striking. I was talking to Uwe Hesse after the talks had finished and part of that was about learning new stuff and training courses. It made me realise that it is way too long since I ran any training courses. I love running training courses.

OK, I’ve had a rest, back to socialising with fellow OUGN 2012 attendees.

Friday Philosophy – The Inappropriate Use of Smart Phones February 24, 2012

Posted by mwidlake in Friday Philosophy, off-topic, Private Life, rant.
Tags: , , ,

I’m kind of expecting to get a bit of a comment-kicking over this one…

I never much liked mobile phones – Yes they are incredibly useful, yes they allow countries that lack a ground-based telephony network to create a nationwide system, yes they allow communication all the time from almost anywhere. That last point is partly why I dislike them. {Actually, I don’t like normal phones much, or how some people {like my wife} will interrupt a conversation to dash across the room to answer it. It’s just a person on the phone, it will take a message if someone wants to say something significant. If someone calls your name out in a crowd, do you abandon the people you are talking to, dash across the room and listen to them exclusively? No, so what act that way over a phone?}.

However, I hold a special level of cynical dislike for “smart” phones. Why? Because people seem to be slaves to them and they seem to use them in a very antisocial way in social and even business situations. It is no longer just speaking or texting that people do, it’s checking and sending email, it’s twittering and blogging, it’s surfing the net and looking things up. I have no problem with any of this, I do all of these things on my desktop, laptop, netbook. But I don’t do them to the detriment of people who are there in the flesh – whilst supposedly in a conversation with mates at the pub or carrying out a transaction in a shop or using the coffee machine at work or, basically, standing in the bloody way staring at a little screen or rudely ignoring people who I am supposed to be interacting with.

The below is my phone. It makes calls, it sends texts, it might even be able to work as an alarm clock (I am not sure). It does not do anything else much and it was ten quid {actually the below might be the version up from the really cheap thing I have}:

I was pondering this rude (ab)use of Smart Phones in a meeting this week. It was a meeting to discuss a program of work, what needed doing and by whom. It was a meeting where everyone in the room was involved, each person’s opinion was important and we all had a vested interest in the outcome of the meeting. So why did over half of the people not only have their Smart Phone out but were tapping away, scrolling through stuff, looking at some asinine rubbish on Facebook {yes, I saw you}? One or two people in the room might have been able to argue that they needed to keep an eye out for important emails or calls – but really? Are things so incredibly important and only you can deal with them that you can’t just play your full part in a meeting for an hour? I was so annoyed by this that I missed half the meeting internally moaning about it…

I just see it as rude. It’s saying “while you people are talking, I can’t be bothered listening and I certainly don’t need to give you my full attention. And I don’t even care that I’m making it so obvious”. Or “I am buying this item from you and we need to deal with the transaction but you are so inconsequential I don’t even have to pause this conversation about which cafe to meet in next week. You do not deserve more than 15% of my attention”.

I supposed that is what really gets my blood slowly heating up, it’s that it has become accepted to be so rude. Just walk down the street, head down and eyes fixed on your glowing little screen, making no attempt to navigate with your fellow city dwellers. I made a decision 2 {correction, 3} years ago that, if you are walking along staring at your phone and you are going to collide with me, you ARE going to collide with me if you do not become aware of me and make allowances – and I am lower down than you, I braced my shoulder and I am going to win this one. If they are so fixated on that bl00dy screen that they do not heed any attention to others, people ping off me like they’ve been thumped by a tree stump. It now happens a lot and I always “win”. I’m surprised no one has punched me yet.

If I was a manager again I would introduce a simply rule. No Smart Phone in your hand unless you have a stated reason for doing so. There are many valid reasons, which will all be related to the meeting. Otherwise you are just being disrespectful. If you feel the meeting does not apply to you or this section is not relevant, fine. Sit still and listen anyway. You might actually find it useful to know what everyone else is doing. Stop playing bl00dy mental chickens or whatever or updating your status to “bored”.

I will hold strongly to these opinions. Right up until the minute I finally buy that iphone I’ve been considering getting. I really want to be able to check my twitter account during meetings, you see.

The Most Brilliant Science Graphic I Have Ever Seen January 5, 2012

Posted by mwidlake in biology, Perceptions.
Tags: ,

The below link takes you to an absolutely fantastic interactive demonstration of the relative size of everything. Everything. Stop reading this and go look at it, when it finishes loading, move the blue blob at the bottom of the screen left and right.

The Relative_scale_of_everything

The raw web link is:


The web page says scale_of_the_universe but it should be relative_scale_of_everything_in_the_universe. Did you go look at it? NO!?! If it’s because you have seen it before then fair enough – otherwise stop reading this stupid blog and Look At It! NOW! GO ON!!!

Yes, I do think it is good.

I have to thank Neil Chandler for his tweet about this web page which led me to look at it. Neil and I talked about relative sizes of things in the pub towards the end of last year, in one of the Oracle London Beers sessions. I think it was Neil himself who suggested we should convert MB, GB and TB into time to get a real feel for the size of data we are talking about, you know, when we chuck the phrases GB and TB around with abandon. Think of 1KB as a second. A small amount of time for what is now regarded as a small amount of data – This blog so far is around 1.2kb of letters. Given this scale:

1KB = 1 second. About the time it takes to blink 5, possibly 6 times, as fast as you can.
1MB = Just under 17 minutes. Time enough to cook fish fingers and chips from scratch.
1GB = 11 and a half days. 1KB->1GB is 1 second -> 1.5 weeks.
1TB = Just under 32 years. Yes, from birth to old enough to see your first returning computer fad.
1PB = pretty much all of known human history, cave paintings and Egyptian pyramids excepting, as the Phoenicians invented writing about 1150BC ago.

The wonderful thing about the web page this blog is about is that you can scan in and out and see the relative sizes of things, step by step, nice and slowly. Like how small our sun is compared to proper big ones and how the Earth is maybe not quite as small compared to Saturn as you thought. At the other end of the scale, how small a HIV virus is and how it compares to the pits in a CD and the tiniest of transistors on a silicon chip. I’m particularly struck by the size of DNA compared to a human red blood cell, as in how relatively large DNA is. Red blood cells are pretty big cells and yet all human cells (except, ahem, red blood cells) have 3.2 billion letters of DNA in each and every one of them. That’s some packaging, as cells have a lot of other stuff in there too.
{NB, do remember that the zooming in and out is logarithmic and not linear, so things that are close to each other in the graphic are more different than first appears, especially when the image becomes large and in effect covers a wide part of the screen}

Down at the sub-atomic scale there are a fair number of gaps, where one graphic is pretty much off the scale before the next one resolves from a dot to anything discernable, but that is what it’s like down that end of things. Besides. It’s so small it’s hard to “look around” as there is nothing small enough (like, lightwaves went by several orders of magnitude ago) to look around with.

My one criticism? It’s a shame Blue Whale did not make it into the show🙂

I actually had flashbacks looking at this web page. I remember, back in the mid-70’s I think, going to the cinema. Back then, you still had ‘B’ shows, a short film, cartoon or something before the main event. I no longer have a clue what the main event was, but the ‘B’ movie fascinated me. I think it started with a boy fishing next to a pond and it zoomed in to a mosquito on his arm, then into the skin and through the layers of tissue to blood vessels, to a blood cell… you get the idea, eventually to an atom. Some of the “zooming in” where it swapped between real footage was poor but it was 1970 or so and we knew no better. It then quickly zoomed back out to the boy, then to an aerial view of the field, out to birds-eye… satellite-like…the earth… solar system… I think it stopped at milky way. I wish I knew what that documentary was called or how to find it on the web…

{Update, see comments. Someone links to the film. I know I looked for this film a few years back and I did have a quick look again before I posted this message. I did not immediately find it but someone else did, in 10 seconds via Google. Shows how rubbish I am at using web searches…}

Friday Philosophy – When Things Feel Wrong October 28, 2011

Posted by mwidlake in Friday Philosophy, Perceptions.
Tags: , ,

I got pinged by someone else missing the Friday Philosophy today {BTW, Good news, the technical blogs start again on Monday}, so…

Take a look at the below. It is a rather pleasant spot of countryside on Sao Migel in the Azores, where the area in the foreground has been converted into a bit of a garden to take advantage of the natural beauty.

Nice, isn’t it? Sorry the sun is not shining, but there you go. This waterfall just across the road from a set of water mills and waterfalls at Achada, which is one of the tourist spots that features often in brochures for Sao Miguel. But look at the scene again. Does anything strike you as odd about that waterfall? I could not put my finger on what it was, I just knew it looked odd. (Graham, if you don’t spot this immediately you owe me a pint).

There was a path heading up the valley to one side of the waterfall, one of a network meandering through the gardens, and I went up it. After a short while there was a smaller path heading up the hill more directly. It looked maintained but too steep to be a “wandering around enjoying the scenary” path. So I went up that. At the top of this path was a structure, a concrete “block house” It hummed and it gurgled. There was another path heading back the way I came, towards the waterfall. I followed along it and I found the top of the waterfall…

Yes, the waterfall was a fake. It was coming out of this huge concrete trough fed by a large pipe which went back to the humming, gurgling concrete block house. Returning back down to the bottom of the waterfall I could put my finger on what was odd about that waterfall. There is a valley to the left. OK, that is not so odd, the water could be coming from high land to the right of the valley and draining into the valley at this point. Except there is another valley to the right of the waterfall as well. Both had small streams running through them. This waterfall could only be natural if there was a perfectly formed, shallow middle valley heading up to the hills between the other two valleys and the only point where the water could escape was at the confluence of the lower two valleys. There was also a lot more water coming down this waterfall than was coming down the two valleys.

What has this got to do with Oracle and databases? Well, have you ever been in the position where you look at the output from a system and it just does not “feel right”? I sometimes refer to something I call DBA Intuition. There is also Developer Intuition and there is certainly Tester Intuition. All are where you are not sure why but it just looks or feels wrong (or, you just get a feeling for what a problem is or what the solution might be, all I class as DBA intuition, but I digress).

As an example, you are tasked to look at one of those terrible BI-type reports that consist of two pages of SQL and they want to know why it takes so long to run. Before you dive into the code, you look at the result of the report and you just think “That seems like an awfully large number of people responding to that advertising campaign” or “I can’t believe 10 percent of our income comes from selling baby diapers”.

Usually when I have dug into the actual report or part of the system that feels wrong I find out one of two things. That I had no idea that part of that business really worked that way, or, that the report is utter garbage. Somewhere in that report there a missing table or a logic flaw {nested AND and OR statements are a good place to look}. This of course has the advantage that there is no need for me to tune the report until someone can tell me WHAT the report is supposed to be identifying.

DBA Intuition is, I think, basically a gut feeling you develop in response to experience. I suppose I have more “tuning intuition” these days, I look at how fast some SQL is coming back and the volume of data and I think “seems reasonable actually” or “something is very inefficient here”. I’ve noticed that good system testers have this intuitive knack of just asking the new system to do things in a way or order that does not match the original intention.

So, I encourage you to trust your intuition. If some part of the system feels wrong, go and root around in the system undergrowth; climb up to the top of the data waterfall {OK, I’ll stop with the bad IT metaphors) and see what you find.

Incidentally, after I found the pump house we walked the other way up the valley, following the pipe and the pleasant gardens. It took maybe 20 minutes but we found the "real source" of the fake waterfall, which was a very nice, natural waterfall sitting in the very bottom of a pleasant valley – just where a waterfall should be. It just took a little more effort to get to it. I'm sure there is some moral story in there but I'm damned if I can work it out🙂

Friday Philosophy – Should I Be a Twit? October 21, 2011

Posted by mwidlake in Blogging, Friday Philosophy, Perceptions.
Tags: , , ,

Something I have been pondering for a while now is should I join in with the “happening crowd” and sign up to Twitter? I know, I’m two or three years behind the times on this, but more and more people who I like have signed up – even Doug Burns now uses twitter and he used to be negative about it in the same way as I. I’ve asked a few of these friends what they think.

I’ve always resisted the whole micro-blogging world, probably due to the comments made by people about how much dross is tweeted and what a time sink it can be, something the people I asked all mentioned. And partly, being candid, because I know one of my faults is to shoot my mouth off before engaging my brain, especially if annoyed. If it takes 20 seconds to do a tweet, I shudder to think some of the things I might have put out there in the heat of the moment or when in the pub. Pub talk is fine, so long as it is kept in the pub. With a blog, it takes me so long to write them I tend to calm down before sending, if I am angry.

Also, tweets are not my style. I don’t know if you have noticed but I can be a little verbose {this means I spout several sentences where 5 words would do}. Can I be succinct enough to say anything anyone else would want to read?

That’s the writing of tweets, what about the following? I could just be a passive follower. But how many? This is part of the advice given to me by Neil Chandler and also Doug, to only follow a few people so that you do not just drown in an endless feed of stuff. I guess that anything good gets re-tweeted and so you see stuff not just by those you follow but the best bits of what they follow? Neil also suggested that part of what makes twitter work is joining in, don’t just be passive.

Something about Twitter that does bug me a lot, and this is just part of the whole texting/smartphone/constant communication thing of modern life, is when people you are spending social time with just sit staring at their bloody phone. I just find that really rude and I also think it’s a bit depressing when you see three or four people in the pub or restaurant, all staring at their smartphones and not communicating with each other. You might as well stay in bed. Alone. {Oh good grief what a terrible thought! Do couples who do social media now just sit in bed with their phones in front of them, ignoring their supposed love of their life?}

But of course there are advantages to Twitter. There is a lot of interesting stuff that goes on only in the twitter world and some of the tweets I have seen have been hilarious. It’s far more lightweight than blogging, something you can do in a quiet moment. Though Doug suggests this is why you get so much dross about travelling, people catch up on twitter when bored and also tweet then. Niall Litchfield made a very interesting point to me in that it allows you to keep up on a large range of topics and see new ideas and thoughts more. I like that. In fact, Niall sent me an excellent list of reasons for and against, which I’ll {almost} finish on.

From Niall:


Reasons for :

– I get vicarious updates from a large number of sources on a large number of subjects. (Oracle,SQL,Science,Politics)
– It can suit my short and acerbic style from time to time.
– More people are doing it
– It indirectly promotes me (albeit with a somewhat misleading image)
– I find stuff I wouldn’t otherwise have found
– I have engaged with product managers I probably wouldn’t have done

Reasons not:
– time wasting
– addictive
– it promotes me in a misleading way
– it can annoy others

I like it because I’m an information junkie and like to keep abreast of stuff in several areas. It can be a colossal waste of time and irritate immensely.


I think I might well give it a go and that leads to my last thought. If I am going to enter the world of twitter, I am going to have to get a new phone. My current one is so basic that it does little more than just make and receive calls. But a single charge lasts a week.

Friday Philosophy – The One Absolute Requirement for System Success October 14, 2011

Posted by mwidlake in development, Friday Philosophy, Perceptions.
Tags: , , , ,

What must you always achieve for an IT system to be a success?

  • Bug free? Never happens.
  • Within budget/time frame? That would be nice.
  • Includes critical business functionality? Please define critical.
  • Secure? Well, it’s important for many systems but then it is often lacking (even when it is important).
  • That it is to specification? Well we all know that’s wrong.

There is only one thing that an IT system must always achieve to be a success.

User Acceptance.

For an individual system other considerations may well be very important, but the user acceptance is, I think, non-negotiable.

{Slight update – This blog was written in 2011 with corporate IT systems in mind, where the use of the system is almost if not explicitly mandated. With the explosion of Apps and web-based systems over the last 5 years, for a lot of IT systems the users, the customers, have to be attracted to the system. We now call User Acceptance User Experience and it is, if anything, even more vital. If the user does not find the system easy to use and of some benefit, they will click off somewhere else.}

The user must get enough out of using the system for it to be worth their while, otherwise at best they will resent using it and at worst… Well, at worst they will use it but will put in any old rubbish to fulfill the dictate that it be used. In this worst case you would be better off if they did not use the system at all. Here are a couple of examples from my working past.

In the first one, I was involved in extending a hospital management system so that it kept track of the expected arrival and departure times for ward patients, allowing a predication of when beds would become available and calculation of expected occupancy rates. Yes, this was a while ago (maybe 1990) and on an a system that was old then. The information was needed by someone with the title “bed sister” {or something similar – and would now be titled “head of occupancy management!…} so that they could better manage the flow of patients and keep a higher bed usage ratio. Was this to make the hospital more efficient? No, it was to satisfy a politically demanded report to the NHS executive. Oh, the overall intention was to increase efficiency – but the report soon became more important than any true efficiency. So, we added columns in tables and field on screens and prompts for the ward staff to fill in the required new information.

And they didn’t.

The nurses were busy, they were pretty demoralized {due to having recently been used by the government as a way to control public sector pay} and they had more nursing duties to do than they could manage. They were not going to waste a couple of minutes trying to check when Mrs Jenkins was going to be sent home when Mrs Leonard needed a bed pan. The nursing staff were given a hospital-wide telling off, this information had to be entered. So they put in the data – but guessed wildly, just to get management off their backs. Thus the design of the system enhancement was fine, the report was logically accurate, only the correct staff could run it, but we failed to achieve User Acceptance and thus the system was a failure.

So I added something else. It was a very crude screen (this was 1990) that showed a “diagram” of a ward – Down the left and right side of a VT220 screen you saw little oblong boxes labelled with a bed number and in it the patient name, the consultant’s initials, a medical specialty code and the arrival and departure date-time. This was based on some information we already had plus the new information we wanted and the screen was quite basic, limited and slow to draw. But it was useful to the ward staff. They could find any patient, they knew who to call if there was an emergency {not the actual consultant of course, but their secretary :-)}, they could check when they expected a patient to leave, they could even see when a new patient was expected. They used it. They put in the expected departure times {sobering thought, this might not be expected leaving alive}, patients would be booked into empty beds when they became available (often before they actually arrived) and the bed nurse could plan new arrivals and the oh-so-important report could be run. We had achieved User Acceptance and the system became a success.

The second example is also from healthcare but from a different system in a different hospital. We were putting together a system to schedule outpatient clinics. We knew what we were doing, it’s pretty simple. You have some medical staff holding the clinic (a consultant and probably a senior house officer), a period for the clinic (3 or 4 hours) and a set of people to be seen, say 40. We gave some flexibility in slot lengths (some people need 5 minutes, some 15), checked for and stopped double booking and verified against other data such as correct speciality for the outpatient’s clinical data. We did not go and ask the patient admin staff about the spec, we knocked up the design and the screens and asked them to test. After all, I was very experienced now, I’d been doing these systems for 3, 4 years…

They very quickly came back to us and said it was rubbish. Oh dear.

We went and saw them. “What’s the problem?” There were a few – but the main one was that you could not double book a slot. Why would you want to do that? Do two patients really want to be consulted at the same time with the same doctor? {NB this can happen in some special cases – such as fertility treatment, psychology and a couple of others, but not Urology or Orthopedics!}
“Err, maybe, it might happen, can we just be able to double book?” OK, we could maybe alter things to allow two patients to be seen at the same time… The patient admin staff are not looking happy. The hospital liaison is looking confused & concerned and interrupts – “You can’t do that! Patient confidentiality can’t be broken!” he says. It got worse. “We need to be able to book… several people in one slot”. How many? “We need to book all the patients into the first slot, with the consultant, so the letters go out to them saying come to see Mr Winders at 1pm”. The admin staff are now looking very shifty.

If any of you have worked in the health service you are probably way ahead of me. The admin staff needed to book all the patients in at this first slot so that the ill and suffering would all turn up at the start of the session. The consultant would then run through his notes and see the two or three he was interested in – and then go and play golf. The Senior House Officer (or whoever) would then work through the rest of the patients for the following three or four hours. If you have ever had to turn up at the start of a consultancy session and sat there for three hours, now you know why. You see, back then, the consultant was only a very small step away from deity level (and I leave it to you to decide if it was a step up or down). What they said went and if they wanted to go and play golf or store 200 medical records in the boot of their car or refuse to speak to “that stupid idiot in renal medicine” then you worked around it. {I’m assured that things are a lot better now, but I’d love to know how it really is}.

We had designed a sensible solution. The users of the system needed a non-sensible {to our mind} solution. Even the NHS liaison chap had never appreciated exactly how much the consultants abused the system, he thought they just booked the people s(he) wanted at the start of the session, but no. The consultant decided that day who was interesting and as a result every patient had to be there at the start.

I count myself lucky that I learnt from direct experience so soon in my working life that (a) you have to deliver what the user will accept and (b) the only way to know what they want is to show them the system and talk with them.

Friday Philosophy – Human Tuning Issues September 23, 2011

Posted by mwidlake in Friday Philosophy, humour, Perceptions, performance.
Tags: , ,

Oracle Tuning is all about technical stuff. It’s perhaps the most detail-focused and technical aspect of Oracle Administration there is. Explain Plans, Statistics, the CBO, database design, Physical implementation, the impact of initialisation variables, subquery factoring, sql profiles, pipeline functions,… To really get to grips with things you need to do some work with 10046 and 10053 traces, block dumps, looking at latching and queueing…

But I realised a good few years ago that there is another, very important aspect and one that is very often overlooked. People and their perception. The longer I am on an individual site, the more significant the People side of my role is likely to become.

Here is a little story for you. You’ll probably recognise it, it’s one that has been told (in many guises) before, by several people – it’s almost an IT Urban Myth.

When I was but a youth, not long out of college, I got a job with Oracle UK (who had a nice, blue logo back then) as a developer on a complex and large hospital system. We used Pyramid hardware if I remember correctly. When the servers were put in place, only half the memory boards and half the CPU boards were initiated. We went live with the system like that. Six months later, the users had seen the system was running quite a bit slower than before and started complaining. An engineer came in and initiated those other CPU boards and Memory boards. Things went faster and all the users were happy. OK, they did not throw a party but they stopped complaining. Some even smiled.

I told you that you would recognise the story. Of course, I’m now going to go on about the dishonest vendor and what was paid for this outrageous “tuning work”. But I’m not. This hobbling of the new system was done on purpose and it was done at the request of “us”, the application developers. Not the hardware supplier. It was done because some smart chap knew that as more people used the system and more parts of it were rolled out, things would slow down and people would complain. So some hardware was held in reserve so that the whole system could have a performance boost once workload had ramped up and people would be happy. Of course, the system was now only as fast as if it had been using all the hardware from day one – but the key difference was that rather than having unhappy users as things “were slower than 6 months ago”, everything was performing faster than it had done just a week or two ago, and users were happy due to the recent improvement in response time. Same end point from a performance perspective, much happy end point for the users.

Another aspect of this Human side of Tuning is unstable performance. People get really unhappy about varying response times. You get this sometimes with Parallel Query when you allow Oracle to reduce the number of parallel threads used depending on the workload on the server {there are other causes of the phenomena such as clashes with when stats are gathered or just random variation in data volumes}. So sometimes a report comes back in 30 minutes, sometimes it comes back in 2 hours. If you go from many parallel threads to single threaded execution it might be 4 hours. That really upsets people. In this situation you probably need to look at if you can fix the degree of parallelism that gives a response time that is good enough for business reasons and can always be achieved. OK, you might be able to get that report out quicker 2 days out of 5, but you won’t have a user who is happy on 3 days and ecstatic with joy on the 2 days the report is early. You will have a user who is really annoyed 3 days and grumbling about “what about yesterday!” on the other 2 days.

Of course this applies to screens as well. If humans are going to be using what I am tuning and would be aware of changes in performance (ie the total run time is above about 0.2 seconds) I try to aim for stable and good performance, not “outright fastest but might vary” performance. Because we are all basically grumpy creatures. We accept what we think cannot be changed but if we see something could be better, we want it!

People are happiest with consistency. So long as performance is good enough to satisfy the business requirements, generally speaking you just want to strive to maintain that level of performance. {There is one strong counter-argument in that ALL work on the system takes resource, so reducing a very common query or update by 75% frees up general resource to aid the whole system}.

One other aspect of Human Tuning I’ll mention is one that UI developers tend to be very attuned to. Users want to see something happening. Like a little icon or a message saying “processing” followed soon by another saying “verifying” or something like that. It does not matter what the messages are {though spinning hour glasses are no longer acceptable}, they just like to see that stuff is happening. So, if a screen can’t be made to come back in less than a small number of seconds, stick up a message or two as it progresses. Better still, give them some information up front whilst the system scrapes the rest together. It won’t be faster, it might even be slower over all, but if the users are happier, that is fine. Of course, Oracle CBO implements this sort of idea when you specify “first_n_rows” as the optimizer goal as opposed to “all_rows”. You want to get some data onto an interactive screen as soon as possible, for the users to look at, rather than aim for the fastest overall response time.

After all, the defining criteria of IT system success is that the users “are happy” -ie accept the system.

This has an interesting impact on my technical work as a tuning “expert”. I might not tune up a troublesome report or SQL statement as much as I possibly can. I had a recent example of this where I had to make some batch work run faster. I identified 3 or 4 things I could try and using 2 of them I got it to comfortably run in the window it had to run in {I’m being slightly inaccurate, it was now not the slowest step and upper management focused elsewhere}. There was a third step I was pretty sure would also help. It would have taken a little more testing and implementing and it was not needed right now. I documented it and let the client know about it, that there was more that could be got. But hold it in reserve because you have other things to do and, heck, it’s fast enough. {I should make it clear that the system as a whole was not stressed at all, so we did not need to reduce system load to aid all other things running}. In six months the step in the batch might not be fast enough or, more significantly, might once more be the slowest step and the target for a random management demand for improvement – in which case take the time to test and implement item 3. (For those curious people, it was to replace a single merge statement with an insert and an update, both of which could use different indexes).

I said it earlier. Often you do not want absolute performance. You want good-enough, stable performance. That makes people happy.

In Defense of Agile Development (and Their Ilk) September 21, 2011

Posted by mwidlake in development, Management.
Tags: , , , ,

In my previous post I asked the question “why doesn’t Agile work?”. I’m not sure the nuance of the question came over correctly.

I’d just like to highlight that the question I asked was “Why does agile not work”. It was not “Why is Agile rubbish“. I’ve said a few times in the past couple of weeks that I like the ideology of Agile and I am (and have been for years and years) a strong proponent of prototyping, cyclic development, test driven design and many other things that are part of the Agile or XP methodologies.

That distinction in the title is a really important distinction and one I’d hoped I’d made clear in my post. Looking back at my post though, I think it is clear I failed😦. I highlighted reasons why I think Agile does not work and in my head I was thinking “if we avoid these, Agile could work” – but when you write something down it does not matter what is in your head if it does not reach the paper.

I’m actually frustrated that in the last few years I have not seen Agile really succeed and also that this must be the normal situation, going on the response you get when the topic of Agile comes up with fellow technicians and comments on my own blog.

However, on that post about Agile two people who’s opinion I deeply respect came back at me to say “Agile does work!”. Cary Millsap, who many of you will have heard of as the “Method R” guy and the person behind Oracle Flexible Architecture. And Mike Cox, who most of you won’t have heard of but Mike taught me a lot about sensible development back in the 90’s. He’s one of the best developers I have ever had the pleasure of working with and I know he has had great success with Agile and RED. I’m not sure if they read my post as “Agile is Rubbish” or they are, like me, simply frustrated that it can work but so often does not.

So I’ve been thinking about this a lot this weekend and I was helped by Cary’s paper on the topic that he mentioned in his comment. I’d highly recommend downloading it as it is an excellent description of not only why Agile can help but describes how and some of the pitfalls {I’d started my own post on that, but go read Cary’s}. I should add, you can see Cary present his case for Agile at the UKOUG conference this year.

So where does this bring me to? Well, I think “Is Agile good or bad” has become almost an “IT religion” topic, people love it or loath it and it is based on what they have seen of the methodology in real life. No, that’s wrong, it is based on what they have seen that has been labelled with that methodology in real life. Or worse, it is based on anecdotal opinion of those around them. The thing is, if you look at what XP is supposed to consist of or what Agile Programming is supposed to consist of, most of us would agree that a great deal of it makes sense in many situations. I’d disagree with some of the details in Cary’s paper but overall I’m in strong agreement. Sadly, What Agile and XP is supposed to be is not well matched by what you see on the ground in most cases. So even if these methodologies are right for the situation, what has been implemented is not the methodology but probably more a slap-dash process that simply jettisons documentation, design and proper testing. This whole thread sprung from my lamenting the demise of database design and several of the comments highlighted that the introduction of Agile seemed to equate, at least in part, with the demise of design. As MIke and Cary say, and as I think anyone who has successfully utilized Agile would say, Design is an integral part of Agile and XP methodology.

Agile can and does work. But many things can and do work, such as taking regular exercise to keep healthy or regularly maintaining your house to keep it weathertight. Like Agile, both take effort but the overall benefit is greater than the cost. And like Agile, do it wrong and you can make things worse. If your window frames are starting to rot and you just slap a new layer of top-coat on them all you will do is seal in the damp and rot and hide the problem – until the glass falls out. Going for a regular 5 mile run is good for you – but not if you are 10 stone (60KG) overweight and have not run in years. A 5 mile run is also not a good idea if you want to be a long-jumper. Right training (methodology) for the right aim. Also, just like keeping healthy, house maintenance or anything that takes effort but works, proponents tend towards extremism – probably as a reaction to the constant {perceived} pig-headedness of critics or the failure of people to just do what now seems so sensible to them {think reformed smokers}. I’ll have to buy Cary and Mike pints to make up for that jibe now, and promise them it was not aimed at them personally…

Sadly, the reality is, Agile does not work 90% of the time it is tried. So, does that mean Agile is actually rubbish? Or at least, not fit for purpose, because many companies are not able to use it? Companies are there to achieve something and the IT systems are part of achieving that something. If Agile cannot aid that IT department then Agile is the wrong way for that department and company.

*sigh* I’ve gone on and on about this and still not got to my own main point, which is this.

– Can we identify reasons for Agile and XP Failing.
– Having identified the Reasons, can we fix them in simple ways?
– Can we create some simple guidelines as to when a project should be more Agile and when it should be more Up-Front design.

I’d love to know people’s opinions on those three points above.