jump to navigation

Friday Philosophy – Human Tuning Issues September 23, 2011

Posted by mwidlake in Friday Philosophy, humour, Perceptions, performance.
Tags: , ,
6 comments

Oracle Tuning is all about technical stuff. It’s perhaps the most detail-focused and technical aspect of Oracle Administration there is. Explain Plans, Statistics, the CBO, database design, Physical implementation, the impact of initialisation variables, subquery factoring, sql profiles, pipeline functions,… To really get to grips with things you need to do some work with 10046 and 10053 traces, block dumps, looking at latching and queueing…

But I realised a good few years ago that there is another, very important aspect and one that is very often overlooked. People and their perception. The longer I am on an individual site, the more significant the People side of my role is likely to become.

Here is a little story for you. You’ll probably recognise it, it’s one that has been told (in many guises) before, by several people – it’s almost an IT Urban Myth.

When I was but a youth, not long out of college, I got a job with Oracle UK (who had a nice, blue logo back then) as a developer on a complex and large hospital system. We used Pyramid hardware if I remember correctly. When the servers were put in place, only half the memory boards and half the CPU boards were initiated. We went live with the system like that. Six months later, the users had seen the system was running quite a bit slower than before and started complaining. An engineer came in and initiated those other CPU boards and Memory boards. Things went faster and all the users were happy. OK, they did not throw a party but they stopped complaining. Some even smiled.

I told you that you would recognise the story. Of course, I’m now going to go on about the dishonest vendor and what was paid for this outrageous “tuning work”. But I’m not. This hobbling of the new system was done on purpose and it was done at the request of “us”, the application developers. Not the hardware supplier. It was done because some smart chap knew that as more people used the system and more parts of it were rolled out, things would slow down and people would complain. So some hardware was held in reserve so that the whole system could have a performance boost once workload had ramped up and people would be happy. Of course, the system was now only as fast as if it had been using all the hardware from day one – but the key difference was that rather than having unhappy users as things “were slower than 6 months ago”, everything was performing faster than it had done just a week or two ago, and users were happy due to the recent improvement in response time. Same end point from a performance perspective, much happy end point for the users.

Another aspect of this Human side of Tuning is unstable performance. People get really unhappy about varying response times. You get this sometimes with Parallel Query when you allow Oracle to reduce the number of parallel threads used depending on the workload on the server {there are other causes of the phenomena such as clashes with when stats are gathered or just random variation in data volumes}. So sometimes a report comes back in 30 minutes, sometimes it comes back in 2 hours. If you go from many parallel threads to single threaded execution it might be 4 hours. That really upsets people. In this situation you probably need to look at if you can fix the degree of parallelism that gives a response time that is good enough for business reasons and can always be achieved. OK, you might be able to get that report out quicker 2 days out of 5, but you won’t have a user who is happy on 3 days and ecstatic with joy on the 2 days the report is early. You will have a user who is really annoyed 3 days and grumbling about “what about yesterday!” on the other 2 days.

Of course this applies to screens as well. If humans are going to be using what I am tuning and would be aware of changes in performance (ie the total run time is above about 0.2 seconds) I try to aim for stable and good performance, not “outright fastest but might vary” performance. Because we are all basically grumpy creatures. We accept what we think cannot be changed but if we see something could be better, we want it!

People are happiest with consistency. So long as performance is good enough to satisfy the business requirements, generally speaking you just want to strive to maintain that level of performance. {There is one strong counter-argument in that ALL work on the system takes resource, so reducing a very common query or update by 75% frees up general resource to aid the whole system}.

One other aspect of Human Tuning I’ll mention is one that UI developers tend to be very attuned to. Users want to see something happening. Like a little icon or a message saying “processing” followed soon by another saying “verifying” or something like that. It does not matter what the messages are {though spinning hour glasses are no longer acceptable}, they just like to see that stuff is happening. So, if a screen can’t be made to come back in less than a small number of seconds, stick up a message or two as it progresses. Better still, give them some information up front whilst the system scrapes the rest together. It won’t be faster, it might even be slower over all, but if the users are happier, that is fine. Of course, Oracle CBO implements this sort of idea when you specify “first_n_rows” as the optimizer goal as opposed to “all_rows”. You want to get some data onto an interactive screen as soon as possible, for the users to look at, rather than aim for the fastest overall response time.

After all, the defining criteria of IT system success is that the users “are happy” -ie accept the system.

This has an interesting impact on my technical work as a tuning “expert”. I might not tune up a troublesome report or SQL statement as much as I possibly can. I had a recent example of this where I had to make some batch work run faster. I identified 3 or 4 things I could try and using 2 of them I got it to comfortably run in the window it had to run in {I’m being slightly inaccurate, it was now not the slowest step and upper management focused elsewhere}. There was a third step I was pretty sure would also help. It would have taken a little more testing and implementing and it was not needed right now. I documented it and let the client know about it, that there was more that could be got. But hold it in reserve because you have other things to do and, heck, it’s fast enough. {I should make it clear that the system as a whole was not stressed at all, so we did not need to reduce system load to aid all other things running}. In six months the step in the batch might not be fast enough or, more significantly, might once more be the slowest step and the target for a random management demand for improvement – in which case take the time to test and implement item 3. (For those curious people, it was to replace a single merge statement with an insert and an update, both of which could use different indexes).

I said it earlier. Often you do not want absolute performance. You want good-enough, stable performance. That makes people happy.

Friday Philosophy – Dyslexia Defence League August 19, 2011

Posted by mwidlake in Friday Philosophy, Perceptions, Private Life.
Tags: , ,
2 comments

NB This post has nothing to do with Oracle or even technology really. It’s just some thoughts about one aspect of my life.

I know I’ve mentioned this once before, though it was in an early blog post when I had a readership of about 8, but I am mildly dyslexic.┬áIf you want to know how I found out I was dyslexic then check out the original post. I’m quite fond of that post, as a non-technical one, though almost no one read it.

The thing is, I now cringe slightly when I say I am Dyslexic. I’ve sat on this post for weeks, wondering if I should post it. You see, it seems to me that dyslexia, along with some other oddities of perception, have over the last few years almost become a thing to be proud of. A banner to wave to show how great you are. “Hey, look at me, I am this good even though I have Dyslexia” or even “I am great because I have dyslexia”. Maybe I am just a little sensitive about it but it seems to me that more and more people make a thing about it. If I am being candid, I feel a little proud that I did OK academically despite it {I should point out there is no proven link between dyslexia and IQ but in exams you get marked down for spelling and slow reading speed means it takes longer to, well, read stuff!} and in the past I have been very open about mentioning it. Hey, this is my second blog on dyslexia!

However, I’ve had it suggested to me in the past that I use it as a defense for being lazy – Can I prove I am dyslexic? Does it really impact me that much? Well, actually no I cannot prove it and has it impacted me? Not a great deal I guess as I can read pretty much anything {I did say it was mild. Scientific papers and anything with very long words can be a challenge, but isn’t that true of everyone?}. My reading speed is about 120,150 words a minute. Average is about 250wpm. My wife seems to read at about 500wpm :-)

Also, don’t get me wrong, I fully appreciate that looking at a challenge you have and taking the benefits from it that you can is a very healthy attitude. If I remember right it was Oliver Sacks in one of his books (“the man who mistook his wife for a hat” maybe) who describes a man with sever Tourette’s syndrome {which is more often all about physical ticks and uncontrolled motions rather than the famous “swearing” aspect of it} who could somehow take advantage of his physical manifestations in his jazz drumming. He could just make it flow for him. But when he took treatment to control the physical issues, his jazz drumming suffered. He really wanted the benefit of the drugs for day-to-day life but keep the Tourettes for jazz. So he took the drugs during the week and came off just before the weekends when he played. Neat.

Does Dyslexia help me? I think I am more of a diagrams and pictures person than a text person because of my dyslexia and I think I maybe look at things a little differently to most people at times – because of the differences in how I perceive. That can help me see things that maybe others have missed? Maybe an advantage. I’ll take that.

Also, in my case at least, dyslexia is not an issue for me comprehending or constructing written prose. I think I write some good stuff at times.

But I don’t want to be dyslexic. Frankly, it p122es me off.

I’ll give you an example. I did a blog post a few weeks back and it had some script examples in it. I had nearly finished it when I realised I had constantly spelt one word utterly wrong. The spell checker picked it up. But just before I posted it, I realised I had also got my column aliases utterly wrong. I have a little set of rules for generating table and column aliases, it is not complex, but in my head the leading letters of a word are not always, well, the leading letters. I had to alter my scripts and then re-run them all as I knew if I tried to unpick the spelling mistakes manually I would mess it up, I’ve been there before. It took me hours. I can really do without wasting that time. {Update, since originally drafting this post the same situation with another technical post has occurred}. Then there is the embarrassment of doing something like spelling the name of a column wrong when you design and build a database. I did that in a V8 database when renaming columns was still not a simple task {was it really Oracle 9 release 2 before column rename was introduced?}. The database went live and accrued a lot of data before anyone made an issue of it. It then kept getting mentioned and I had to keep explaining.

I don’t see Dyslexia as a badge of honour and every time I see someone being proud of it (or to my odd mind it seems they are proud of it) or suggesting they are better than average for overcoming it (again, maybe it is just my perception), I just feel uncomfortable. I think all and everyone of us has something we have had to overcome to be “normal”.

Yet, on reading that above paragraph back, it is simply insulting to people who have fought and striven to overcome severe dyslexia or other issues with perception or communication. I certainly do not mean that (and I apologise unreservedly to anyone who is now fuming at me because of my callousness).

Maybe that is my issue with the whole topic – I am not uncomfortable with the notion of being proud to have overcome something like dyslexia and I admire people who cope with other conditions which make it harder for them to get by in our culture, but I just can’t see why you would be proud of the condition or want to use it as a bragging right.

I guess I want to be able to just acknowledge my dyslexia, point out it is no big deal in my case but it is why I spell like a 10 year old. It is as significant as the fact I’m scared of heights. I guess I cringe a little when I say it as I don’t want to be seen to be making excuses and I certainly do not feel, that in my case at least. I have won through against the odds. Maybe I’ve been a little hard-done-by occasionally but haven’t we all?

Friday Philosophy – Picture Theft!!! July 28, 2011

Posted by mwidlake in Blogging, Friday Philosophy, Perceptions.
Tags: , , ,
7 comments

Last week’s Friday Philosophy was a bit of a moan about how hard I find it to make nice graphics, how long it takes and no one seems to care that much about the results.

Well, after those two days effort on the pictures and the afore mentioned moan, irony of irony, someone has stolen one of my graphics!. So someone likes my efforts ;-). It is the one that represents how you scan down the levels of an index and then link across to the table via the rowid.

Before I go any further I better make it clear that I am not really upset about it at all :-). In fact, since the scoundrel included a link back to my web page and they are considerably better known than I, my little blog has had a big up-swing in traffic as a result, which is nice. Mind you, as the person who borrowed my diagram is SQL Server expert Steve Jones, of SQLSeverCentral/Redgate fame, most of my new audience are probably pretty focused on the SQL Server RDBMS and not Oracle, so unlikely to make many return visits unless they are work across the RDBMS boundaries.

What also gives me a little smile is that I have stumbled over the fact that I myself, back in November 2009, was looking for such a diagram {of the way Oracle steps down the index to the leaf blocks, gets the rowid and then straight to the table row} to ‘borrow’ for a post of my own on BLevel and heights of indexes. I even confessed at the time to looking for and failing to find one to use…

Humour aside, it set me to thinking though. Borrowing content is a perennial and thorny issue.

Occasionally someone will start putting content out on their blog or web site and it turns out that much of that content is directly obtained from other peoples’ blogs and websites – copy&pasted straight in or with little changes. That is generally seen by the original author as unacceptable and once they find out they object. In such cases it sometimes seems the culprit is unaware of this being a transgression and, once it is explained that they have effectively stolen many hours or days of someone’s efforts, they remove the material. Others seem aware this is theft but do not care until caught. Occasionally the culprit sees no error in their ways at all, even when challenged, as the material had been put “out there” so they now consider it free to all. I certainly do not agree. Perhaps the worst thing you see though is people including parts of published books, or even putting the whole book out there for download. Such people should of course have their hands stapled to their backsides in punishment, that is simple theft. Writing blogs takes a long time and effort, writing technical books takes forever and monumental effort. I know from friends that the financial return for such efforts is pitiful enough as it is.

On the other side of the coin, many of us put our stuff out there on the web to be read and used and are very happy for it to spread, to be borrowed from and disseminated. Like nearly all DBAs and developers, over the years I have written lots of little SQL scripts to pull information out of the data dictionary or do little database management tasks. I happily give away copies of these to anyone who wants them (and you can get them off my web site if you like, but just pretend it is not my website, as it is truly awful). All I ever ask is that whoever takes them leaves my name in them.

I think that is core to the issue. I suspect many of us bloggers are happy for small parts of our output to be borrowed so long as credit is given. I certainly am {but please note, this is my personal opinion – other bloggers may object very strongly and any repercussions on you in respect of taking material from other blogs and web sites is your concern}. However, Volume is also part of it. The larger the chunk you borrow, the more acknowledgement I would need to be happy about it. Borrowing a single diagram or a paragraph out of a page of text is OK, given I am cited for it. Taking most of a post would probably not, unless you asked first, were really nice about it and about me. Nicking a set of course notes I wrote is certainly unacceptable, no matter how much you put “originally written by that wonderful Martin Widlake” on it.

So, I think you need to cite the source as “payment” for using it. Perhaps the best way to do it is by simply linking to the material rather than putting it on your blog/website, but that does not work if you need the content within yours to make sense. In which case, I think Steve Jones’ approach of putting the content in his and including a link is reasonable. It might have been nice if there was a comment saying where the image came from but I can live without it. Despite my joking about it giving me more hits to my blog, it does not matter that his is a popular web site and gives me more hits. Even if a site gets no traffic, if someone has borrowed a small part of my output but cited me as the source, I’m cool with that.

The problem though is judging what is a “small” part to borrow and what is acceptable to the original author. We all perceive such things differently. So the safest thing is to ask the original author. If I want to use an idea that came from someone else in one of my blogs or a solution they came up with, I always ask and I ask if they want to be cited. This includes discussions in email or in the pub. I ask. If when preparing my blogs I learn a lot from someone else’s blog, I stick in a link and a comment, even though I will have written my own text. I hope that so far I have not upset anyone when I borrow a little.

Photos are a different issue though. I am not going to even attempt to cover that one!

Snowdon viewed from Yr Aran

If you Really Can’t Solve a “Simple” Problem.. March 11, 2011

Posted by mwidlake in Friday Philosophy, Perceptions, Testing.
Tags: , ,
12 comments

Sometimes it can be very hard to solve what looks like a simple problem. Here I am going to cover a method that I almost guarantee will help you in such situations.

I recently had a performance issue with an Oracle database that had just gone live. This database is designed to scale to a few billion rows in two key tables, plus some “small” lookup tables of a few dozen to a couple of million rows. Designing a system of this scale with theory only is very dangerous, you need to test at something like the expected volumes. I was lucky, I was on a project where they were willing to put the effort and resource in and we did indeed create a test system with a few billion rows. Data structure and patterns were created to match the expected system, code was tested and we found issues. Root causes were identified, the code was altered and tested, fine work was done. Pleasingly soon the test system worked to SLAs and confidence was high. We had done this all the right way.

We went live. We ramped up the system to a million records. Performance was awful. Eyes swung my way… This was going to be easy, it would be the statistics, the database was 2 days old and I’d warned the client we would need to manage the object statistics. Stats were gathered.
The problem remained. Ohhh dear, that was not expected. Eyes stayed fixed upon me.

I looked at the plan and I quickly spotted what I knew was the problem. The below code is from the test system and line 15 is the key one, it is an index range scan on the primary key, within a nested loop:

   9 |          NESTED LOOPS                       |                           |     1 |   139 |    37   (3)| 00:00:01 |       
* 10 |           HASH JOIN SEMI                    |                           |     1 |    50 |    11  (10)| 00:00:01 |       
* 11 |            TABLE ACCESS BY INDEX ROWID      | PARTY_ABCDEFGHIJ          |     3 |   144 |     4   (0)| 00:00
* 12 |             INDEX RANGE SCAN                | PA_PK                     |     3 |       |     3   (0)| 00:00:01 |       
  13 |            COLLECTION ITERATOR PICKLER FETCH|                           |       |       |            |          |       
  14 |           PARTITION RANGE ITERATOR          |                           |    77 |  6853 |    26   (0)| 00:00:01 | 
* 15 |            INDEX RANGE SCAN                 | EVEN_PK                   |    77 |  6853 |    26   (0)| 00:00:01 | 

On the live system we had an index fast full scan (To be clear, the below is from when I had tried a few things already to fix the problem, but that index_fast_full_scan was the thing I was trying to avoid. Oh and, yes, the index has a different name).

|   9 |          NESTED LOOPS                 |                           |     1 |   125 |  1828   (3)| 00:00:16 |
|  10 |           NESTED LOOPS                |                           |     1 |    63 |     2   (0)| 00:00:01 |
|* 11 |            TABLE ACCESS BY INDEX ROWID| PARTY_ABCDEFGHIJ          |     1 |    45 |     2   (0)| 00:00:01 |
|* 12 |             INDEX UNIQUE SCAN         | PA_PK                     |     1 |       |     1   (0)| 00:00:01 |
|* 13 |            INDEX UNIQUE SCAN          | AGR_PK                    |     1 |    18 |     0   (0)| 00:00:01 |
|  14 |           PARTITION RANGE ITERATOR    |                           |     1 |    62 |  1826   (3)| 00:00:16 |
|* 15 |            INDEX FAST FULL SCAN       | EVE_PK                    |     1 |    62 |  1826   (3)| 00:00:16 |

Now I knew that Oracle would maybe pick that plan if it could get the data it wanted from the index and it felt that the cost was lower than doing multiple range scans. Many reasons could lead to that and I could fix them. This would not take long.

But I could not force the step I wanted. I could not get a set of hints that would force it. I could not get the stats gathered in a way that forced the nested loop range scan. I managed to alter the plan in many ways, fix the order of tables, the types of joins, but kept getting to the point where the access was via the index fast full scan but not by range scan. I thought I had it cracked when I came across a hint I had not known about before, namely the INDEX_RS_ASC {and INDEX_RS_DESC} hint to state do an ascending range scan. Nope, no joy.

By now, 8 hours had passed trying several things and we had a few other people looking at the problem, including Oracle Corp. Oracle Corp came up with a good idea – if the code on test runs fine, copy the stats over. Not as simple as it should be as the test system was not quite as-live but we did that. You guessed, it did not work.

So what now? I knew it was a simple problem but I could not fix it. So I tried a technique I knew had worked before. I’d long passed the point where I was concerned about my pride – I emailed friends and contacts and begged help.

Now, that is not the method of solving problems I am writing about – but it is a damned fine method and I have used it several times. I highly recommend it but only after you have put a lot of effort into doing your own work, if you are willing to give proper details of what you are trying to do – and, utterly crucially, if you are willing to put yourself out and help those you just asked for help on another day.

So, what is the silver bullet method? Well, it is what the person who mailed me back did and which I try to do myself – but struggle with.

Ask yourself, what are the most basic and fundamental things that could be wrong. What is so obvious you completely missed it? You’ve tried complex, you’ve been at this for ages, you are missing something. Sometimes it is that you are on the wrong system or you are changing code that is not the code being executed {I’ve done that a few times over the last 20 years}.

In this case, it was this:

Here is my primary key:

EVEN_PK EABCDE 1 AGR_EXT_SYS_ID
EVEN_PK EABCDE 2 EXT_PRD_HLD_ID_TX
EVEN_PK EABCDE 3 AAAMND_DT
EVEN_PK EABCDE 4 EVT_EXT_SYS_ID
EVEN_PK EABCDE 5 EABCDE_ID

Except, here is what it is on Live

EVE_PK EABCDE 1 EVT_EXT_SYS_ID
EVE_PK EABCDE 2 EABCDE
EVE_PK EABCDE 3 AGR_EXT_SYS_ID
EVE_PK EABCDE 4 EXT_PRD_HLD_ID_TX
EVE_PK EABCDE 5 AAAMND_DT

Ignore the difference in name, that was an artifact of the test environment creation, the key thing is the primary key has a different column order. The DBAs had implemented the table wrong {I’m not blaming them, sometimes stuff just happens OK?}.
Now, it did not alter logical functionality as the Primary Key is on the same columns, but as the access to the table is on only the “leading” three columns of the primary key, if the columns are indexed in the wrong order then Oracle cannot access the index via range scans on those values! Unit testing on the obligatory 6 records had worked fine, but any volume of data revealed the issue.

I could not force my access plan as it was not possible – I had missed the screaming obvious.

So, next time you just “know” you should be able to get your database (or code, or whatever) to do something and it won’t do it, go have a cup of tea, think about your last holiday for 5 minutes and then go back to the desk and ask yourself – did I check that the most fundamental and obvious things are correct.

That is what I think is the key to solving what look like simple problems where you just can’t work it out. Try and think even simpler.

Where is Sun? January 3, 2011

Posted by mwidlake in Friday Philosophy, Perceptions.
Tags: ,
3 comments

First of all, may I wish everyone who comes by my blog a heartfelt Happy New Year.
Secondly, I promise I’ll blog more often and more on technical aspects this year than I have for most of 2010.
Thirdly, I’ll admit the title to this blog is nothing to do with the hardware company now owned by Mr Larry Ellison, but is about the huge glowing ball of fire in the sky (which we have not seen a lot of here in England and Wales for the last couple of weeks – not sure about Scotland but I suspect it has been the same). I apologise for the blatantly misleading (and syntactically poor) title.

A quick question for you – It is the depths of winter for most of us, and it has been unusually cold here in the UK and much of Europe. When are we, as a planet, furthest from the Sun during winter? January the 1st? The Shortest day (21st December)? The day the evening start drawing out (December 14th)?
I think many in the Northern Hemisphere will be surprised to learn that we are closest to the sun today (3rd Jan 2011). A mere 147.104 million kilometers from the centre of our solar system. I mentioned this to a few friends and they were all taken aback, thinking we would be furthest from the warmth of the sun at the depths of our winter.

Come the 4th July 2011 it is not only some strange celebration in the US about having made the terrible decision to go it alone in the world {Joke guys!}, but is the day in the year that the Earth is furthest from the sun – 152.097 million kilometers. That is about 3.39% further away and, as the energy we receive from the sun is equal to the square of the distance, does account for a bit of a drop in the energy received. {Surface of a sphere is 4*pi*(R{adius}squared), you can think of the energy from the sun as being spread over the sphere at any given distance}.

Some of you may be wondering why this furthest/closest to the sun does not match the longest/shortest day. As some of you may remember, I explained about the oddities of the shortest day not matching when the nights start drawing out about this time last year. It is because as we spin around our own pole and around the sun, things are complicated by the fact that the earth “leans over” in it’s orbit.

Check out this nice web site where you can state the location and month you want to see sunrise, sunset, day length and (of particular relevance here) the distance from the sun for each day.

I find it interesting that many of the things us most of us see as “common sense” are often not actually right (I always assumed that the shortest day coincided with both the evenings starting to draw out and mornings getting earlier until I stumbled across it when looking at sunset times – I had to go find a nice Astronomer friend to explain it all to me). I also like the fact that a very simple system – a regularly spinning ball circling a large big “fixed point” in a fixed way – throws up some oddities due to little extra considerations that often go overlooked. Isn’t that so like IT?

That lean in the Earth’s angle of spin compared to the plane we revolve around the sun is slowly rotating too, so in a few years (long, long, long after any of us will be around to care) then the furthest point in the orbit will indeed match the northern hemisphere winter. Again like IT, even the oddities keep shifting.

Friday Philosophy – Run Over by a Bus December 3, 2010

Posted by mwidlake in Friday Philosophy, Perceptions.
Tags: , ,
7 comments

I chaired a session at the UKOUG this week by Daniel Fink, titled “Stop Chasing your tail: Using a Disciplined Approach to Problem Diagnosis”. It was a very good talk, about having a process, an approach to solving your IT problems and that it should be a process that suits you and your system. All good stuff and I utterly agree with what he said.

But it was a passing comment Daniel made that really set me thinking. It was something like:

You should be considering how people will look after the system after you have gone, the classic ‘what will we do if you are hit by a bus’….. No, I don’t like thinking like that, that phrase… I prefer ‘after you win the lottery and retire to a great life’.

It just struck a chord with me. Mr Fink’s {and I do go all formal when I intend respect} take on this is a far more positive way of looking at the situation of leaving the system in a state that others can look after once you are no longer able to help. The “Bus” phrase is very, very common, at least in the UK and I suspect in the US, and it is a very negative connotation. “Make sure it all works as something nasty is going to happen to you, something sudden, like being smeared across the tarmac by 25 tons of Greyhound doing 50mph, something basically fatal so you can’t prepare and you can’t help any more”. So, not just moved on, but dead.

Daniel made me realise that we should be looking at this from totally the other perspective and that doing so is much, much, much better. “Make it work so that they love you, even when you have gone away to a happier situation – one involving no road-based unpleasantness at all”.

Everyone leaves their job eventually and I like to think it is often for more positive reasons. Like retiring, or a better job {better for you, but a real shame for your old company as they like you so much}, moving to a new area, attempting a dream. Yes, sometimes (depressingly often at present) it is because you get made redundant or things go bad with your managers, or HR take over the organisation. But even so, better to leave knowing you did so with your professional duty intact I think. It’s one way of winning in a losing situation.

If turning the “bus” metaphor into a “lottery” metaphor results in the response in your brain of “well, when I do leave rich and happy, I still want to leave a painful mess behind me” – then it may indicate that you better leave where you are working as soon as possible in any case? As it is not a good situation and you are deeply very unhappy about it.

Up until now I have sometimes used a far more gruesome but less fatal phrase for the concept of making sure things continue after you leave and can no longer help, which is “involved in a freak lawnmower accident”. As in, can’t type but not dead. I’m going to stop using it, I’ve decided that even with my macabre sense of humour, it really is not a good way to think about doing your job properly. Daniel, your attitude is better. Thank you.

Oh, if you went along to the conference you can get the latest version of Daniel’s talk slides from the UKOUG web site (try this link), otherwise, he has a copy here – pick “papers and presentations”. It has lots of notes on it explaining what the slides mean (ie, what he actually says), which I think is a very nice thing for him to have spent the time doing.

The Immoral Unfairness of Contracts October 11, 2010

Posted by mwidlake in Perceptions, Private Life.
Tags: ,
10 comments

You can tell from the title that this is going to be a rant…

Do you remember the last time you signed a contract for a job? Did you read all the terms, conditions and clauses? How angry did it make you? If you did not read it, dig it out and do so. It will ruin your whole day.

I do a mixture of contracting and consulting to provide bread on the table and catfood in the cat bowl and I get to sign a lot of contracts. And they send me mad as so many of them have such outrageously immoral, unfair and, I strongly suspect, illegal clauses in them. But if you don’t sign, you don’t get a job.

If the contract says they can get rid of me on a week’s notice, but I have to give them a month, I insist they pick one or the other and it applies to both parties. If there is a clause saying everything I think of belongs to them then I say no – if it is based on their intelectual property or code specific to their application, then it is theirs and I will comply utterly, but if it is the sort of generic data dictionary query that all these client rely on me to use to do my job, it is mine and I want the right to use it {and give it to other people, like I gave it to you, Mr Client}. Another clause that seems to be becoming rampant in the UK contracting arena is the 40-hour working week and signing away any right to complain. I absolutley object to that as it has been proven scientifically that continuous long hours are detrimental to health. If I choose to do 40, 50 hours in a week (and I often do) it is my choice but they damned well are not going to insist on it. I also know if I do the 50 hours for too long, my productivity and quality drops – and I think we all know this is the real case.

There is often a discussion with the actual people you work with, how the contract is just “admin” and they would never treat you in the way it says they can and “just sign it and forget it as we know you will do the job and we will never use clause 17.3.2 on you”. And they probably won’t, but it makes the contract a big, fat lie at best and a potential stick to beat you with at worst.

A few years ago I decided that I had had enough of this and I now challenge the worst of these clauses and I have had some succes. I also challenge them because, just once or twice, I have had someone try and take advantage of me due to these clauses. Usually recruitment agencies, I have to say.

With small organisations I usually can agree fair and equitable terms. With larger organisations it is a fight but I can usually get some sense into the agreement. But with international corporations, it is a blank refusal. They do not need me, they can buy in someone else and they damn well ain’t going to negotiate or treat you as an equal.

I’m facing this one right now. I’m looking at the contract and the blank refusal by the faceless (and probably deeply annoyed {and overworked}) minion in Admin to even consider a single letter change to a contract. And I am thinking “well sod you and your job and your immoral and bullying contract then”. This morning I really considered walking off site and sacrificing any chance of payment to “punish” such unbending unfairness.

But I probably won’t, I’ll probably roll over and sign the abusive, vile document because I have already been on-site for a week and I like the people I work with, I like the job and I want their project to succeed. And the potential unfair aspects of the contract will probably never be a real issue. So why can’t they just be fairer and why does it make me so absolutely incandescant with rage?

*sigh*

Friday Philosophy – The Best IT Person I Have Met September 24, 2010

Posted by mwidlake in Friday Philosophy, Perceptions.
Tags: , ,
3 comments

I’ve had the pleasure of working with and meeting a lot of talented and capable people in IT – some of them have even been nice people too :-) {In fact, most people I have met do not match that annoying myth about IT’s reputation for social awkwardness). However, for me one person sticks out in my mind as the best person I have worked with in IT.

It’s Barry. I’m pretty sure none of you have met Barry, and in fact as I knew Barry back in 1996 I’m not so sure I would recognise him (and I have no hope of remembering his last name) if I met him now.

Barry and I met when I got press-ganged into a Unix system administration team. I was just getting started at being an Oracle Performance person and knew very little about Unix Sys admin. But, for reasons I won’t go into now, I went home on the Friday as an “Oracle expert” and came in on the Monday to find my desk had been physically lifted and moved into the Unix sys admin corral and I was now a “Sys Admin not-expert”. My protestations were listened to – and then ignored, with the information that if I did “not knuckle down and get on with it”, the money would stop flowing. So, rather dazed and just a tad unhappy with the situation, I sat – and sulked – at my desk. And there, sat next to me, was Barry.

You are probably expecting me to now tell you that Barry knew Unix sys admin inside out and how he took me under his wing and showed me the ropes. Well, he didn’t. I have no idea where they got Barry from, I think he was a pro-C developer, but he had just been similarly abused by management and deposited into a team he had not signed up for. And he knew even less about being a sys admin than I did. I at least knew my way around a few monitoring commands like top, w, ps, “glance” etc.

Barry was also not very quick with IT. Don’t get me wrong, he was not stupid, but he was not one of these people who just had an affinity for technology and spent all his spare time building their own media server when CD-ROMS for PCs were still quite new. In fact, he seemed to find the whole of IT to be something of a challenge.

What Barry had though was enthusiasm, commitment and curiosity. Not in an annoying, bouncing all over the place crying “this is great” way, but more a case of “OK, server Falcon has run out of disk space. What can I do about it? How do I find out where the storage has all gone, who is using it and can I get it back off them?”. And he would set to. He’d start with what he knew (which was little more than the “Man” {Online Manual} command in the first week) and the bits he could suck out of my head and work through it. Every few minutes he’d be tapping me on the shoulder and saying things like “Look, you can get information about disk usage here, and map it to the real physical disks by greping for this”.

It was Barry’s attitude that made him stand out, and also his ability to infect you with the same attitude. I started off in that team furious and demoralized, determined to find a new position and resign ASAP. But Barry got over his annoyance and started working. He asked me for advice and discussed the issues over with me, even though I was as clueless as him. When he found something he showed me it. When I found something, he was keen to learn it.

Between us, we got by. We knew very little and it was hard work, but because Barry was not daunted and would keep working on the problem until he had it sorted, he dragged me along with him. Often I would still be there with him into the evening, sorting something out when everyone else had gone home. He did not just take on every problem people came to us with though, he would stick with what he felt was the biggest issue until it was sorted, and he would keep with it, and ask for help, and try what you suggested.

{oddly enough , the worst person I ever worked with was already in this team. Maybe that is why the others left and Barry and I were pulled in!}

It only lasted a few months as we both escaped to jobs more suited to our skills, but I learnt a few things. One was that a crummy job could be made a lot better just by your attitude and another was that some people (Barry, not me) had a real talent for enthusing people and thus getting things done. And also, that you did not have to be highly intelligent or knowledgeable to do a very good job. That’s lucky for me, then :-)

Team Work & The Science of Slacking July 23, 2010

Posted by mwidlake in Friday Philosophy, Management, Perceptions.
Tags: , , ,
add a comment

We all know that working in a team is more efficient than working on your own (and I did say a week or two back how I was enjoying the rare privilege of working in a team of performance guys). Many of us also know about team dynamics and creating a balanced team of ideas people, completer-finishers, implementers, strategists and so forth. Those of use who have been exposed to training courses or books on team management know all these good things about teams and how we are supposed to get the most out of them.

How many of us, though, have been introduced to the work of the French Agronomist Max Ringelmann and the aspect of teams named after him, the Ringelmann Effect? In summary the Ringelmann Effect proposses that people in teams try less hard than they do when working alone. Especially if they think no one is watching them.

Back at the start of the 20th century Ringelmann tested out his ideas using a tug-of-war experiment. He would get people to pull on a rope as hard as they could and record their efforts using a strain gauge. Then he would get them to pull on the rope as part of a team, from 2 to 8 people. As soon as people were part of a team, they pulled less hard. With two people in the team, each pulled 93% as hard as on their own, with three people this dropped down to 85% and with 4 it was just 77%. By the time there were 8 people in the team, effort was down to 50%.

This idea of shirking work more and more as the team increased in size became established in modern psychology and was given Mr Ringelmann’s name. Psychologists explain that when someone is part of a group effort then the outcome is not solely down to the individual and, as such, is not totally in their control. This acts as a demotivating factor and the person tries that little bit less hard. The larger the team, the greater the demotivation and the more significant the drop in effort. Ringelmann found that effort was down to 50% in a team of 8 so how bad can the impact of the team be? I think most of us have at least witnessed, and quite possibly been in, the position of feeling like just a cog in a massive corporate team machine. Thoroughly demotivating (though, of course, we all of us still tried as hard as we could, didn’t we?).

The effect is also know under the far more entertaining title of Social Loafing.

Monsieur Ringelmann was far kinder at the time and pointed out that these chaps pulling on the rope could well have been suffering from a lack of synergy. They had not been trained together to pull as a team so that could account for the drop in effort, they were not synchronising their effort.

However, in the 1970’s Alan Ingham in Washington University revisited Ringelmanns work and he was far sneekier. Sorry, he was a more rigorous scientist. He used stooges in his team of rope-pullers, blindfolds and putting the one poor person pulling for real at the front of the team pulling the rope. Thus he could record the effort of the individual. Ingham found that there was indeed a drop in efficiency due to the team not pulling as one. But sadly, this was not the main factor. It remained that the drop in effort was mostly down to the perceived size of the rest of the team. The bottom line was proven to be the human capacity to try less hard when part of a team and that the drop in effort was directly proportional to the size of the team.

We are of course not immune to this effect in the IT world and someone has even gone to the effort of checking that out, James Suleiman and Richard T Watson.

It seems the ways to reduce this problem are:-

  • Don’t give people boring jobs.
  • Don’t give the same job to several people and let them know they all have the same job.
  • Ask people how they are getting on and give them mini-goals along the way.
  • Atually reward them for success. Like saying “thank you” and NOT giving them yet another boring, hard job to do as they did the last one so well.

I think it is also a good argument for keeping teams small {I personally think 5 or 6 people is ideal} and split up large projects such that a single team can cope. Then give tasks to individuals or pairs of people.

If you like this sort of thing you might want to check out one of my first blog post (though it is more an angry rant than a true discussion ofthe topic) which was on the Dunning-Kruger effect, where some people are unaware of their own limitations – though I did not know it was called the Dunning-Kruger effect until others told me, which only goes to show that maybe I am not aware of my own limits… Read the comments or click through to the links from there to get a better description of some people’s inability to guage their own inabilities.

Friday Philosophy – The power of cooperation June 27, 2010

Posted by mwidlake in Friday Philosophy, Perceptions, performance.
Tags: , ,
3 comments

Being the person responsible for the performance of an Oracle-based system can be an oddly lonely working experience. It can also be a very gregarious position as you get to meet a lot of people and discuss lots of different aspects of many systems – all systems seem to have performance issues and so people come along to you, hoping you can identify a “work faster” database setting for them.

But you are often the only person who specialises in Oracle performance. You generally need to be in an organisation that is very large or where performance is key to success for there to be justification for dedicating a person to the topic. To have more than one person dedicated to performance your organisations has to have a very strong focus on getting the best performance out of the Oracle and related systems {or really, really atrocious performance issues :-) }. So usually there is no one else around who is as experienced (or more so) as yourself to discuss such things over with or ask for a second opinion.

Which is why I am very lucky at the moment. I’m working in a team of oracle performance people. There are 2.5 of us (one is a manager with other responsibilities, so he only counts as half). Being able to say “Hey, Dave, What do you think of the wait times on scattered reads?” or “how in heck do I force this nested subquery on a view to use a hash join?” and get some other ideas is very valuable.

What is also interesting is how opinions and preferred techniques on tuning can be different and just as valid. As an example, last week I was working on a poorly performing statement. I was at home and it was the evening, so I was not communicating with the rest of the team. I managed to get the code down from 40 minutes to just under 20 by using a combination of a LEADING and USE_HASH hint. I sent the code back to the user. Only to see that within thirty seconds of each other my colleague Graeme had also sent the user a response, again getting the code down to around 20 minutes. Graeme had pulled a chunk of the code into a subquery factoring “WITH” clause and added cardinality hints. Totally different changes.

So Graeme and I then had a “philosophical” discussion about the different changes {“Mine is best” – “No! Mine is, yours is all bloated and complex”- “Your hint is less future-flexible!!!”}. Only joking, we actually discussed the changes and why we each chose what we did. Graeme commented that is showed that tuning was an art and not a science and I countered that it was a science, as we had both identified where we felt the execution plan could be improved but used different techniques to get there. The thing is, Oracle is so complex and has so many options to influence the optimiser that you have flexibility to chose different tools and techniques.

We had both identified the same main improvement but had each come up with different tweaks for later in the plan.

The end result was that we went with Graeme’s main plan {he is bigger than me} but we pulled in my tweak. That bought the execution time down to around 10 minutes, so about four times faster over all and twice as fast of either of us alone. That is one of the advantages of not working alone.

We also then discussed how we could get this code down to seconds with the use of either Materialized views or changing the process that generated the report to do so incrementally and store the daily results. Until one of us realised we had reached the boundary of compulsive tuning disorder. The report running in 10 minutes was enough improvement to satisfy the business, the report was only going to be run over the next couple of months, so spending a day or two re-working it further was going to be fun – but of no advantage to the business. We told each other to finish for the day. So another advantage of not working alone is that not only do you get more technical input but your help prevent each other losing sight of the overall aim.

It really does help to have two people working on the same area.

{There is a sneaky way of getting beyond being a lone performance specialist. If you are in an organisation long enough you can usually find some other idiot who is silly enough to want to learn more about performance and you can train them up. It does not take long before they know enough to start coming up with things you never thought of. Oracle is, after all, full of many ways to do the same thing and you can’t know it all}.

Follow

Get every new post delivered to your Inbox.

Join 185 other followers