jump to navigation

Friday Philosophy – Is the CBO becoming way too complex? October 19, 2012

Posted by mwidlake in Friday Philosophy, performance.
Tags: , ,
20 comments

I was at the SIOUG annual conference in Slovenia this week (and a very good conference it was too) and I was watching a presentation by Christian Antognini about how the CBO learns by it’s mistakes. This was of course mostly about adaptive cursor sharing and cardinality feedback. Chris was also able to share a few tid-bits about 12c enhancements in this area. I can’t go into details, but basically it looks like the CBO is going to not only enhance those two features but there is a new one where the CBO can change the plan on the fly, as the same query progresses.

As I watched the presentation I found I was in two minds about this. Part of me was thinking “Yes, these features do help in the CBO realising it has chosen a poor plan and improving things for the next execution – and it will get better at it”. The other part was thinking “Oh dear, yet another source of confusion and of performance of queries changing when ‘nothing has changed’“.

It also solidified an idea I have for a new presentation, about how what you see in the execution plan may well not be what is actually executed. ie there are optional bits in there that do not get run at execution time, depending on the results of prior steps.

But I digress. What I ended up wondering at the end of Chris’s talk was this. Is the CBO getting just too complex? It is all very good that extra abilities are being added so that there is a better chance of a good plan being chosen in the first place and the various ways we can control the choice of the plan grows and grows. But we are long past the point when you could write even a simple sql statement and know what the plan will be before you explain it – let alone run it, check the plan and then run it again a few minutes later and be sure the plan will be the same.

Why does this bother me? For three reasons:

1) Performance will not be stable, so the user experience will not be consistent – and so they will be unhappy. Users will generally be happy if they run a report and it takes one minute, if it takes one minute each time. If it takes one minute and then it takes 5 seconds, they want it to always run in 5 seconds, else they fell cheated or that the system is broken. In fact, I am pretty sure a user will be happier if a report always takes 2 minutes rather than it take 5 seconds or 40 seconds but they never know which! (So long, that is, that they never, ever see it run in less than 2 minutes).

2) If performance of a particular SQL statement is absolutely crucial, I need to know how the CBO can and probably will satisfy it so that I can work that into my solution. Otherwise I am reduced to trying out informed options (or maybe random changes :-) ) in order to find the optimal solution – and I cannot be sure that the plan won’t suddenly swap later unless I fix the plan. OK, it might swap to be faster, but I only notice when it swaps to be a lot slower.

3) I’ve said this before, but I am starting to really wonder how anyone new to this starts understanding oracle performance tuning any more. Reading blogs and books about how Oracle works and what impacts the CBO teaches you a lot about what is possible but it only makes sense when you do little tests and proofs of concepts. ie you actually run simple examples and see what happens. But with all these options, considerations and now on-the-fly alterations to plans by the CBO, it is very confusing. Even I, after 20 years of doing performance tuning on and off, am constantly finding myself looking at relatively simple SQL and having to work out why the plan is as it is and satisfying myself that it has not changed. I started with the CBO when it was a lot simpler and I’ve been introduced to the complexities gradually, as they have been introduced, so learning about it has been easier for me.

Perhaps I should not worry about this. I mean, the CBO is getting better at doing what it does, that is good. It is getting hard for those younger and smarter than me to learn about it, so my career is protected. And there is more stuff to talk about so I can keep going to conferences and talking about them.

And with that happy idea, I’m off to start the weekend. Where was that cork screw?

Friday Philosophy – Work Inside Life August 24, 2012

Posted by mwidlake in Friday Philosophy, off-topic, Private Life.
Tags:
add a comment

I know, the usual phrase is “Life Outside Work” but I like to think that, no matter how much we may like our jobs, our overall life is the key thing.

I was prompted to write today’s Friday Philosophy when I was reading Neil Chandler’s blog a few days ago and saw that he had posted about his up-coming {and now in progress} Banger Rally. He and some mates are doing something very cool in their spare time, taking part in a rally from, basically, Calais in France to Naples in Italy, in a “banger”. This is a UK (and wider?) term for an old, tired, worn out car. The most they could spend on the car was £250 and it has to go over some pretty extreme mountain passes. I presume it also has to carry the set of 4 people involved too, so not a lot of weight there then, Neil :-). You can follow their progress at this blog. I was in a pub in Leeds with Neil when he was considering one car for this rally.

I can’t claim that I do anything as striking as Neil’s current jaunt when not working, but I do like to spend some time doing things that are nothing to do with IT at all. For me, this tends to be physical tasks like cutting down trees with my chainsaw. Or building structures in the garden with wood (OK, half-building them – I constructed the below platform and another out into our pond maybe 18 months ago, both still have no balustrade around them and there is a 45cm gap between the bank and the pond platform – NEXT weekend I’ll maybe finish one. Maybe.)


Another garden task recently was building our wood-fired, mud-constructed pizza oven. My wife and I went on a 1-day course to learn how to make a mud-based pizza oven and then spent, ohhhh, about 10 days over 8 weeks building one! We dug a big hole in the garden to get some clean clay (we live in an area that is on top of clay), stole some straw from the horse that lives at the bottom of our garden (long story), bought some sand (because nicking it from beaches is both illegal and bad form) and mixed up our first batch of clay-sand-straw. Add in some old bricks and we got started. It took about 30 lots of mixture, a few alterations to the ratios as we went, but we ended up with this monster.

If you are wondering what the white stuff on the oven is, some ants decided to build their nest in the oven, between the layers I guess (you build the inner oven first and then add several extra mud layers to give insulation and a larger thermal mass, so that the oven stays warm longer).

Like anything, if we did it a second time we would probably make a much better job of it as we learnt so much from the first attempt. One of them was to invite friends with young children over to see us during the build. Kids love the idea of treading the mud mix and you can get a suprising amount of work out of a single ten-year-old (and very insistent I get the age right) German girl if they are allowed to get very muddy.

I found the whole process very relaxing (but very tiring) as you do not need to use too much brain power, slapping the mud mixture on the outside is deeply cathartic and you end up with something useful. Well, slightly useful.


Having built it we then had to learn how to fire it. As an ex-boy-scout I thought this would be easy. Getting the fire going is not too hard, you need to keep it burning well for about an hour to heat up the oven and, one big tip, don’t use any damp wood or wood that “spits” (like willow). Having a burning log explode and shower your almost-ready pizza with charcoal embers can lead to angry garden scenes and considerable bad language.

We can just about get a 9″ pizza in there. The pizza in this shot is actually about 6″. The oven was designed to take a 12″ pizza but, errm, I forgot to allow enough space to one side to push the fire into as well.

Another major thing to keep in mind? That arch! You can’t put anything in the oven that is larger than that hole. More importantly, you can’t get anything out that won’t easily come back through that hole. Sticking your arm inside to jiggle things about is also tricky as it gets up to around 300C! {for US readers – about a millon F}

As you can see, the results are anything but professional! And, compared to nipping down to the shops to buy a pizza and just slamming it into the kitchen oven, it takes a lot, lot longer. But you can drink beer or enjoy a bottle of win and sit in the garden whilst getting the mud oven fired and hot. Last weekend we had friends over and the ladies made the pizzas, I cooked them and the other gentlemen helped us work through the beer collection and ate. As for taste? Fantastic.

My wife is far more artistic than I and she makes decorative cakes, sews and now even makes hats when not auditing IT systems (Have you ever been involved in an IT audit and the person knows how to make lists and check them – but can’t plug in a PC? Well, Sue was once a DBA and also a Unix Sys Admin, so when she audits you, she finds what you are hiding!). As such, maybe as an industry we should encourage her hat-making

So what, if anything is the point of this Friday Philosophy, other than to show off my pizza oven and my wife’s hats?

Well, I see this outside work activity as important in the workplace as well, especially if you are having to manage people. Firstly, it is important, I feel, for us all to do something we really enjoy to relax and re-energize (and this can be “computers” for IT people, but as I get older I notice more of my friends who were utterly technology-focused in their 20’s are now moving more towards non-IT hobbies).

Secondly, what people do in their spare time can tell you a lot about the person and what keeps them happy. If they have hobbies that are artistic and their IT job is very regimented, it could indicate they would be happier with some more less rigid task in the mix, say some design work. If they have NO outside hobbies, you are probably working the poor dears a little too hard!

Thirdly, and I have seen this for real, you may have a person or a team that is not performing well in the office – but it turns out a lot of them do interesting and challenging things at home. Someone who can motivate themselves to be a part-time fireman or has the dedication to train each day for a sport has skills and energy. Why, as their manager, am I not seeing much of these traits in the office? Because they are not happy and/or motivated. So I have an issue to sort out. I don’t manage people at present, but I still find it interesting what people do and achieve outside the office as it can indicate that they have talents and skills not being seen in the workplace.

I wonder how I can introduce my chain-saw skills into performance tuning?

Friday Philosophy – New Game: Phone Zombies! (You Too can Play) August 17, 2012

Posted by mwidlake in Friday Philosophy, humour, off-topic, Perceptions.
Tags: , ,
8 comments

I’m spending a lot more time in Central London at the moment due to current work commitments. A few weeks ago I was having a quiet stroll through the streets and had what I can only describe as an odd moment:

I looked around and found I was being converged upon by 5 or 6 people walking slowly and aimlessly towards me – all from different angles, all only vaguely aware of their surroundings, all looking like they were making straight for me. I instantly thought of one of the scenes from “Shaun of the Dead” {A cracking film, go hire it tonight}.

They were all on their smart phones of course, intent on the little glowing screens of whatever it was they could not drag their eyes from – despite them also trying to navigate a busy city landscape. As some of you know, I have a bit of a “hate-mild dislike-grudging acceptance” relationships with Smart Phones.

I was so struck by this scene that I nearly did not move in time, but finally I did step to one side as I watched them do this quite wonderful, little, shuffling-dance around each other. I think only one of them actually looked up properly, the others all did that micro-glance; frown; direction shift; re-engage-with-screen procedure that is becoming so common. As a species we must be somehow pre-designed to cope with this as none of them actually bumped into each other – but it took several micro-glance manoeuvres for some of them to make it through.

This has resulted in a new game I can’t stop playing as I make my way through London:

Phone Zombies – How many people can I see at any time who are effectively lobotomised by their personal electronic device?

I have a few rules:

  • I has to be an electronic device – phones, smart phones, crackberries, tablets, electronic books etc.
  • Real books and papers do not count.
  • They must be upright (so no sitting).
  • If they are moving they count.
  • If they are stopped in the middle of the path they count.
  • If they have put themselves in a doorway or some other sensible place they do not count.
  • Unless, even though they have done that, they are still e.g. blocking ingress and egress from the doorway.
  • A bonus point if they micro-glance manoeuvre.
  • 5 bonus points if contact is made with another person in the time I am watching.
  • 10 bonus points for contact with something inanimate {only once to date}.
  • 20 points if they go “uuurrrgggghhh” and have blood on them. {no one has got 20 bonus points yet, but I live in hope}

I think my best so far is about 14, but that is because two phone-zombies both walked into each other. Classic.

Friday Philosophy – I Am An Exadata Expert August 10, 2012

Posted by mwidlake in Exadata, Friday Philosophy, Perceptions.
Tags: , , ,
5 comments

(Can I feel the angry fuming and dagger looks coming from certain quarters now?)

I am an Exadata Expert.

I must be! – I have logged onto an Exadata quarter rack and selected sysdate from Dual.

The pity is that, from some of the email threads and conversations I have had with people over the last 12 months, this is more real-world experience than some people I have heard of who are offering consultancy services. It’s also more experience than some people I have actually met, who have extolled their knowledge of Exadata – which is based solely on the presentations by Oracle sales people looking at the data sheets from 10,000 feet up and claiming it will solve world hunger.

Heck, hang the modesty – I am actually an Exadata Guru!

This must be true as I have presented on Exadata and it was a damned fine, technical presentation based on real-world experience and I have even debated, in public, the pros and cons of point releases of exadata. Touching base with reality once more, I did an intro talk “the first 5 things you need to know about Exadata” and the “debate” was asking Julian Dyke if he had considered the impact of serial direct IO on a performane issue he had seen and he had not only done so but looked into the issue far more than I – so he was able to correct me.

But joking aside –  I really am a true consulting demi-god when it comes to Exadata

I have years of experience across a wide range of Exadata platforms. That would be 0.5 years and I’ve worked intensively on just one system and am in a team now with some people who are proper experts. So a range of two. Yes, tongue is still firmly in cheek.

This situation always happens with the latest-greatest from Oracle (and obviously all other popular computing technologies). People feel the need to claim knowledge they do not have. Sometimes it is to try and get consultancy sales or employment, sometimes it is because they don’t want to be seen to be behind the times and sometimes it is because they are just deluded. The deluded have seen some presentations, a few blog posts and maybe even got the book and read the first few chapters and are honeslty convinced in their own minds that they now know enough to make effective use of the technology, teach {or, more usually, preach} others and so proclaim on it. {See Dunning Kruger effect, the certainty of idiots}. I’m certainly not arguing against going to presentations, reading blogs and books and learning, just don’t make the mistake of thinking theoretical, second-hand knowledge equates to expert.

With Exadata this situation is made worse as the kit is expensive and much of what makes it unusual cannot be replicated on a laptop, so you cannot as an individual set up a test system and play with it. Real world experince is required. This is growing but is still limited. So the bullshit to real skills quotient remains very, very high.

If you are looking for help or expertise with Exadata, how do you spot the people with real knowledge from the vocal but uninformed? Who do you turn to? {NB don’t call me – I’m busy for 6 months and I really am not an expert – as yet}. If your knowledge to date is based on sales presentations and tidbits from the net which may or may not be based on a depth of experience, it is going to be hard to spot. When I was still without real world experience I had an unfair advantage in that I saw email threads between my fellow OakTable members and of course some of those guys and gals really are experts. But I think I was still hoodwinked by the odd individual on the web or presenting and, I can tell you, though this background knowledge really helped – when I DID work on my first exadata system, I soon realised I did not understand a lot about the subtulties and not-so-subtulties of using a system where massively improved IO was available under key conditions. I had to put a lot of time and effort and testing to move from informed idoit to informed, partially experienced semi-idiot.

I know this issue of the non-expert proclaiming their skills really frustrates some people who do know their stuff for real and it is of course very annoying if you take someone’s advice (or even hire them) only to find their advice to be poor. Let’s face it, is is simple lying at best and potentially criminal mis-selling.

I guess the only way is for peopel needing help to seek the help of someone who has already proven themselves to be honest about their skills or can demonstrate a real-world level experience and success. I would suggest the real experts should do that most difficult task of pointing out the mistakes of the false prophets, but it is very tricky to do without looking like a smartarse or coming over as a big head or jealous.

I’ll finish on one thing. Last year I said how I thought maybe I should do more blog posts about things I did not know much about, and be honest about it and explore the process of learning. I did actualy draft out about 3 posts on such a topic but never pushed them out as I was way too busy to complete them… That and, being candid, I really did not want to look like an idiot. After all, this Oracle lark is what puts beer in my hand, hat fabric on my wife’s millinary worktop and food in my cat’s bowl. The topic was….? Correct, Exadata. Maybe I should dust them off and put them out for you all to laugh at.

Friday Philosophy – Whatever Happened to Run Books? July 27, 2012

Posted by mwidlake in Friday Philosophy.
Tags: ,
3 comments

I realised recently that it is many years since I saw what used to be called a Run Book or System Log Book. This was a file – as in a plastic binder – with sheets of paper or printouts in it about a given system. Yes, this was a while back. It would often also have diagrams {occasionally drawn by hand on scraps of paper – so that would be the database ERD then}, hand-written notes and often the printed stuff would have scribbles against it.

{BTW I asked a colleague if he remembered these and when he said he did, what he used to call them – “err, documentation???”. Lol}

There was one book per key system and you could tell if a system was key (that is, Production, or a development system where a large development manager would punch you in the eye for losing anything, or any system the DBAs wanted) as it had a run book. It held information that was important about the system and, although you could look up most of it when logged onto the system itself, was useful to grab and just check something. However, it was vital if you had to recover the system.

Being a DBA-type, the run books I used to see and use were database focused. The front page would have the SID, name, host name (and even the spec of the host), version, tnsnames info, block size, backup strategy and schedule and, very importantly, the system owner. Yes, the big guy who would be upset if you lost the system. In there you would have printouts of the tablespaces, datafiles and sizes, the backup script, users (and passwords, very often), reference data tables, filesystem layout, OS user details and anything else
needed to recover the system.

This was an evolving and historical set of data. I mentioned above that you would have maybe scraps of paper from when a design session had come up with an alteration to the system. Corrections would often be done by hand. When you printed off the tablespace sizes on Monday, you did not throw the old one away but just added the new one, so you had information about the growth of the DB going back in time. Once in a while you might thin out the set but you kept say one a month.

It was actually that which got me to thinking about runbooks. At a site recently one of the DBAs was asking me if I knew of a screen in OEM that showed the growth of space used over time and my immediate thought was “well look in the run book” {I was very tired that day and losing my grip on reality}. Not being able to find a screen for what he wanted and knowing the data in OEM/AWR was only going back a month anyway, I suggested a simple spreadsheet that he could maintain. With the run book you could flip to the printouts of tablespace sizes, grab a piece of paper and do something lo-tech like this:

This would take less time than firing up Excel, typing the figures in, getting the graph wrong 3 times and then printing it out. Though if you had to go show Managers how the data was growing, you invested that time in making it pretty {why do high level managers insist on “pretty” when what they really want is “informative”?}

So why have Run Books gone {and does anyone out there still use them, in physical or electronic format}? It certainly seemed standard practice across IT in the 80’s and 90’s. I suspect that the reason is that most of the information that used to go into them is now available via online GUI admin tools and looking at them is actually faster than going and grabbing a physical book. Besides, if your DBA or Sys Admin team is split between UK, India and Australia, where do you keep a physical book and allow everyone to check it? I have vague memories of electronic Run Book applications appearing but they never seemed to get traction.

That is one of the drawbacks of using GUI admin tools. No, this is not just some tirad by a bitter old lag against GUI tools – they are generally a massive improvement on the old ways – but they are not perfect. Most of them only hold a short history and printing out the data is often tricky or impossible. All you can really do is screen dumps. No one has those little scripts for listing out basic information anymore {except us bitter old lags} as they have GUIs to do all that and, heck, I can’t go printing off a load of stuff on paper and sticking it in a binder – that is so 20th century!

Maybe I’m being unfair and OEM has a “run book” section I have simply never seen – but I’ve never seen it. If it is/was there, how many people would use it?
I do miss the Run Book though. Especially the ease with which I could look up all those passwords…

Friday Philosophy – Presenting Leads to Drinking, Discuss June 15, 2012

Posted by mwidlake in Friday Philosophy, humour, Meeting notes, UKOUG.
Tags: , ,
9 comments

Just a quick Friday Philosophy {the day job is very demanding at the moment, thus the silence on the Blog front}

I’m presenting in Leeds at the start of July on UKOUG AIM SIG on “The First Few Things You Need To Know About Exadata”. As part of the final preparation of the agenda it’s been raised that we should have a beer after the event and put it on the agenda.

Now, when I ran the Management and Infrastructure SIG, there was always a last item of “retire to a pub for a drink or two”. It is a common feature of technical UKOUG SIGs and a great opportunity to chat to the speakers more.

Chat to the speakers. Now I think about it, most of the speakers always make it to the pub after a SIG. If the attendance on the day is 10% speakers, 10% committee and 80% delegates, the make-up in the pub will be 30% speakers, 20% committee and 50% delegates, or similar.

At conferences, the bars in the evening are covered in speakers (all still speaking – loudly and {usually} drunkenly).

So, is it that:

  • Speaking leads to elevated drinking
  • Drinking makes you more of a sucker for speaking
  • Speaking and Drinking have a shared genetic basis
  • It’s just me.

My excuse is that all that hot air coming out my mouth makes it dry and it needs a little wetting afterwards…

You Will Be Our Slave – Err, no, I Won’t May 27, 2012

Posted by mwidlake in contracting, Friday Philosophy, rant.
Tags: , , ,
10 comments

For the sake of current clients, this posting has been time-shifted.

I’m looking at the paperwork for a possible new job in front of me. Document seven out of 13 is the Working Time Directive Waiver. It’s the one where you sign on then dotted line saying your proposed new client can demand more than 48 hours of work a week out of you. {This may be UK or European Union specific but, frankly, I don’t care}.

I’m not signing it. For one thing, I doubt the legality of the document under EU law – especially in light of the issues the UK government had with this and junior doctors {who often, and still do, end up making life-deciding decisions on patients when they are too tired to play Noughts and Crosses, having worked 80 hours that week}. For another, well, I don’t give a damn. I ain’t signing it.

Now, I’ve just completed about 60 hours this week for my client. Not a problem at all, something needed doing, I could do it and so I have. I have done, am doing and will continue to do long weeks for clients when there is a business need and it fits in with the rest of my life and it is not a chronic situation {chronic is a medical term that means “long lasting and on-going”}.

If I am doing 60 hours plus every week, that means I am trying to do 2 people’s job at the same time and doing both of them badly. I don’t care how great I am at doing what I do, if it is 60 hours each and every week, I’m doing it badly because I am too stressed and tired to be doing it well. Also, where is the rest of my life? I have no “rest of my life”.

If my client is asking me to do 60 hours this week and I say “no” and they sack me under the Working Time Directive waiver – that means it is not a request, it is an enforcible demand. I am their slave. Nope. Not happening. It is best all round if it is acknowledged up front before I arrive on site that the client may ask and I may well say yes – but I can say no.

I know, some of you will be reading this and saying “but I need my job and if that is what it takes, I do it”. Well, I’ve worked for 20+ years and I’ve realised that (a) there are organisations that don’t abuse you and (b) you actually get little real payback for those ridiculous hours. But it can ruin your non-work life, even your family life. I don’t need any individual job and I am bloody well not playing those games any more. Employment in a modern, democratic society is supposed to be a mutual agreement and, if is it not, I ain’t playing. That is my small win for all those years of grind and I’m insisting on it.

I know, some of you will say “look, it never comes to anything, just sign it and ignore it like the rest of us”. No. If you are right, it is a corporate lie and is not required. And, to my detriment, I know you are wrong and sometimes there is an attempt to enforce it. If you cannot get me to do the 60 hours by asking and explaining, either you do not have a valid reason {and history proves I am an utter push-over to a half-reasonable request} or there is a reason very important to me why I can’t comply. If you try and insist, you really are treating me like a slave. That empty space? That’s me having gone for a looong walk.

I am not signing a document saying “you can demand I work over 48 hours any and all weeks you like”. Your are not signing a form saying “I can demand any time off I like week in and week out”. All contracts have a clause saying “this is not working between us, we will curtail the agreement”. We will use that if need be, not a bullying document that says I am your slave.

I am not signing.

Friday Philosophy – It’s not “Why Won’t It Work!” it’s “What Don’t I Understand?” April 27, 2012

Posted by mwidlake in Friday Philosophy.
Tags: , ,
add a comment

I had a tricky performance problem to solve this week. Some SQL was running too slow to support the business need. I made the changes and additions I could see were needed to solve the problem and got the code running much faster – but it would not run faster consistently. It would run like a dream, then run slow, then run like a dream again 2 or 3 times and then run like a wounded donkey 3 or 4 times. It was very frustrating.

For many this would provoke the cry of “Why won’t it work!!!”. But I didn’t, I was crying “What don’t I understand???”. {I think I even did a bit of fist-pounding, but only quietly as my boss was sitting on the desk opposite me.}

I think I’ve always been a bit like that in respect of How Things Work”, but it has been enhanced within me by being blessed to work with or meet people for whom it is more important for them to understand why something is not working than fixing it.

I was reminded of this by a thoughtful comment in an email that one of the oaktable sent to the list. They made the comment that what they felt was common between members of the oaktable is “that we’re not interested (really) in what the solution is of most of our problems, but actually, what is the underlying issue that really causes the problem?”

It struck a real chord with me. Quite a few people I’ve come across seem to be fixated on wanting to know solutions – so that they can look knowledgeable and be “one of the best”. But that’s just stamp collecting really. It’s like one of those ‘games card’ fads that each generation of children has, where you want to collect the best cards so you can win. I never got it as a kid as there are few rules, tactics, ‘how it works’ to elucidate. What success is there in winning when it’s just down to the cards you have? {And being candid, I didn’t like them as partly as I never had the money to buy many cards and partly I was rubbish at trading them. No sales skills.}

I know the solve-it-don’t-just-fix-it position is a topic I have touched on before, but I think the attitude of trying to fix problems by understanding how it works is far more satisfying than doing so by knowing a set of solutions. You develop a deeper understanding to help solve new problems than any amount of solution-stamp-collecting ever will. However, another wise voice on the Oaktable discussion pointed out that you can be in a work environment where there is no time to investigate and you simply have to try your set of fixes and move on if you hit one that works. Your work environment can strongly influence how you work and, it some ways, the ways you think.

I bet some people are wondering what my problem at the start of this post actually was? Well, a nice technical blog about it may appear over the weekend, but the core reason for the toggling of working/not-working was partition swap. We have data coming into the system very fast. We build a new summary of the key data in one table and then swap it into active play via partition swap. On the live system, stats had not been gathered on the “swap” table we had introduced but had on the active table. So, each time the partition swapped, we went from good stats to “empty” stats or the other way around. The empty stats gave a quite, quite dreadful execution plan.

Friday Philosophy – Identifying and Nullifying Fake Urgency April 20, 2012

Posted by mwidlake in Friday Philosophy, Management.
Tags: , ,
6 comments

You know how it goes. You get a call/mail/text with something along the lines of “I need to know all the details of customer orders placed on Tuesday 7th by customers based in Botswana – and I need it ASAP, by end of play today at the latest”. So you skip lunch, drop that task you have been trying to get around to doing all week and work out how to resolve the issue that has just been dropped on you. It takes a lot of effort and you finally get it sorted out around an hour after you told your girlfriend/boyfriend/cat you would be leaving the office that day – and mail it off to the requestor. You might even call them to let them know it is done, but oddly they don’t answer.

Next day, you see the guy who wanted this urgent request and ask if it was what they wanted “Oh, I have not looked at it yet – but thanks for doing it.”

NO! “Thanks” does not work in this situation. I’d have more respect for this guy if he laughed at me and said “got you again, sucker”. Many of you know what I mean don’t you – if you are in a support-type-role, this can be a big part of your life.

I had a job years back that seemed to consist 90% of such tasks. I was the development DBA team leader responsible for testing, validating and promoting code to production. Everyone’s changes were Urgency Level 1, to be done as an emergency release and many could not be put in place until after 5pm. I’d be sat there at 18:30 in a massive but virtually empty office, applying changes along with one or two of my guys. Everyone else had gone home. This was not once or twice a month, it was 4 or 5 times a week. What are you to do?

Well, I came up with one tactic that seemed to work pretty well.

Anyone who asked for an emergency change had to be there, on site, available when the change was done.
There were of course cries of protest and people stated it was ridiculous that they had to be there, they were not needed, the change had been tested thoroughly {oh how I laughed at that – a thoroughly tested “emergency” change huh?}. No, I replied, you had to be there in case it went wrong as it’s your system, your data and, frankly, your emergency. If it is not urgent enough for you – the guy wanting it to be done – to be inconvenienced, well it sure as hell is not urgent enough to inconvenience me. “You can call if there are problems” – What, after you have escaped the locality? Maybe turned off your phone? And if I get you , I have to wait for you to come back in? No no no. Urgent emergency now equates to presence in office. After all, I’ll be there.

I stuck to my rule. If the requester could not be bothered to stay, I downgraded the request to “Planned” and put it through the CAB process. If the requester dumped on one of their team and made them stay, I mentally marked them half a point down and factored it in next emergency.

The change was remarkable. I was no longer in the office on my own every evening. I was not there with someone else either. I was simply not there as, when you made the emergency a little bit inconvenient to the requester, it magically stopped being an emergency.

There was another change. Less cock-ups. Seeing as these changes now went through the CAB process and slightly more testing {like, some testing} the duff changes were more likely to be detected before they caused damage. My bosses went from regarding me as “not a team player” to “Not a team player – but we kind of get your point now”.

So my advice is, if someone wants to try and make something your emergency, find some way of making sure it remains inconvenient to them. If they are willing to put up with the inconvenience, then it is a real emergency and you need to crack on with it.

Friday Philosophy – The Abuse of Favours March 30, 2012

Posted by mwidlake in Friday Philosophy, humour, Perceptions, rant.
Tags: , , ,
8 comments

You probably all recognise this situation:

Dave needs something doing that he can’t do himself – let’s say it is creating an API for the file management package. It isn’t your job to do but it is something you can do. Dave is blocked until the API is created.

So, being a nice person, you tell Dave you will see what you can do for him over the next couple of days.

So why is it that what Dave hears is “Dave, I love you more than life itself, I am dedicated to this task and I WILL complete it before the end of tomorrow. My other tasks, emergency production issues and the untimely demise of my cat are all secondary to this endeavour.”.

You see, 24 hours later, Dave is at your desk “When will this be done?! I’m blocked until I get this!!!”. If he’s the guy I had recently his next step is to slap his fist into his palm as he utters, almost shouts “I NEED this!”.

No. No you don’t need it. What you need is for that slap to be in your face, followed by “wake up! You don’t go shouting at the guy digging you out the hole!”.

I find this particularly unacceptable when the favour is to be fixing some mess that Dave created, or doing something Dave told his boss he had finished last week. Of course, those are the exactly situations where Dave is most likely to get upset, as he is in real trouble and most likely to commit that ultimate Favour sin:-

Dave to Boss “I Didn’t get my task done as Martin promised to create the API and he hasn’t. I’d be there now if I only did it myself”.

If you are thinking “Hmmm, I think I might have been ‘Dave’ recently” then Shame On You and go beg forgiveness. Of course, if you were ‘Dave’ you may well be the sort of sod who will twist the situation around in your head so it was not your fault anyway. Grrr, bad Dave.

For a while I gave up doing work favours as I got sick of the situation above playing out. Then I started doing favours again but being a bore about saying repeatedly, up front, that this was a favour, it was only if I had time, not to rely on me and, if it is that important, go ask someone else. Yeah, sounds really grumpy doesn’t it? That gave me a reputation for not being a Team Player (which is code for “mug”).

Now I have a rule system. As soon as someone starts getting demanding about the favour, I immediately challenge it. If they get shouty they lose their favour rights. No more favours for you until the requisite number of beers have been bought. It’s three.

Of course, you see this scene played out on help forums all the time. Initial message is nearly always in upper case text speak “PLS HLP ME, IS URGNT! CN U TELL ME HOW 2 DO MY JOB – THNX!!!” and soon degrades into helfull person asking for details of the exact person and Mr Shouty demanding more and more help. I don’t help. After all, this guy is never going to buy me a beer.

Follow

Get every new post delivered to your Inbox.

Join 156 other followers