jump to navigation

Friday Philosophy – Should the Software or the User be the Stupid One? August 7, 2009

Posted by mwidlake in internals, performance.
Tags: , ,

Oracle’s performance engine is complex and copes with a lot of database situations automatically – or to be more precise, it tries to cope with lots of database situations automatically.

Over the last few versions, Oracle has added many, many things to allow the database to cope automatically with all sorts of different data volumes, spreads of data, relationships between tables, use of different oracle technologies (By this I mean bitmap tables, index tables, partitions, clusters, external tables). All of these things aim to allow the database to just “cope” with whatever you need it to do, with less and less effort by the users {by users, I mean technical users; DBAs and Developers}. Thus it allows for “stupid” users. {I mean no offence, maybe read “inexperienced” instead of stupid}.

As an example, you can now have some very large tables consisting of several partitions and some status look-ups. You query against them. Oracle’s CBO will automatically ignore partitions it can ignore, use indexes or full table scans to use the least amount of IO,use histograms to spot where clauses are on low-cardinality values, Hash joins rather then nested loops as appropriate depending on memory availability, use bitmap indexes when it thinks it can and merge the results from several bitmap indexes, use function based indexes to support functions in where clauses….
It even self-gathers the information to look after all this. Column usage and table modifications are tracked, statistics are gathered when needed and in ways to support data skew, PGA and SGA can be automonitoring and managing…

It all sounds great. In fact, most of the time, for most people, it is great. {I know, most people reading this post are probably people who have encountered the problem systems and so know it goes wrong and so you need more knowledge to cope – you are a biased set of people. In the nicest way, I should add :-) } The idea is, I believe, that you do not neet to be smart to look after Oracle.

If it is not great, if this highly complex system gets it wrong and tries to satisfy SQL statements in sub-optimal ways, then the User has to step in and fix things. ie You.

It is now horrifically complex for us technical users to understand what is going on. You have to not only be “not stupid”, but “not average” either. Sometimes you have to be “not great”, ie brilliant.

In my example, we need to look at if the SQL is constructed to allow the indexes to be used, are functions correctly laid out to use function indexes, are partitions being maintained correctly, when were stats last gathered, did it include histograms and do they help, has oracle missed the need for histograms, are the indexes analyzed at a high enough sample size, are the bitmaps greatly slowing down inserts, have hints been used on the code, are initialisation parameters set to override default fucntionality…

You get the idea, I won’t drone on further. I didn’t even mention memory considerations though {OK, I’ll shut up}.

My point is, the more complex the software, the more “intelligent” it is, the more it is designed to allow for “stupid” users, then the more super-intelligent the user has to be to cope when it breaks.

How about an alternative?

How would it be if we went back to the Rule Based Optimizer and no automatic management of complex situations?

Oracle would maybe need to add a few rules to the RBO for it to cope with later developments, so it would be slightly more complex than V6 but not a lot.
Everything else, the User decides. You only gather stats you decide to gather, on objects you decide need them. No you don’t, it’s a Rule Based Optimizer – no stats gathering! {But see below}.

No automatic memory management. No automatic anything.

The User {the technical user, the DBA and Developer} would have to be smart. Not brilliant, just smart. You would probably have to do more, but most of it would be easier as the levels of complexity and interdependence are reduced. All those tweaks and tricks in the CBO and all the monitoring to cope with “complex” would not exist to go wrong.

Plus it might solve another concern I have. I think there is a chasm growing as there is no need to solve simple problems as Oracle copes but then having to solve complex problems when Orcle does not cope. If you don’t develop skills and experience solving the simple problems, how do you solve the complex ones? I think this is why most Oracle performance and architecture experts are old {Sorry, pleasantly middle-aged}. Young people new to the arena have a massive learning mountain to climb.

So, if we have stupid software, maybe we can get away with more stupid “smart” expert users. ie ALL of us can cope. You cut your teeth on smaller, simpler systems and learn how to cope with the stupid software beast. As you learn more, you learn to cope with more complex situations and they never get that complex as the database is not so “clever”

I’d actually still argue that all the intelligence gathering the Oracle database does should still continue – stats gathered on objects, the ability to gather information on memory usage and thus advice on changes, tracking column usages and table changes. But We, the Stupid Users get to look at it and use it as we see fit for our systems.

I’m sure many systems would not work quite so fast in my senario, but I’d rather have a system working at 75% it’s theoretical fastest all the time rather than one working at 95% and breaking regularly, and in ways so complex it needs weeks to work out and fix.

I now await all the comments to tell me how stupid I am {I can be blindlingly stupid, especially on Fridays}.

Unhelpful “helpful” people June 9, 2009

Posted by mwidlake in Management, Perceptions.
Tags: ,

I keep meaning to getting back to more technical blogs but I need to spend some time sorting it out first, and something has just bugged the hell out of me, so another “wordy” one today.

Richard Foote has gone back to explaining the basics of indexes and the CBO, which if you are new to CBO or indexes, hot-foote {sorry} it over there immediately and check it out. He is a brilliant teacher.

Near the start, he has a link to someone on one of the OTN forums who is stating the Cost Based Optimiser sucks. I won’t repeat the link. No, sod it, I WILL repeat the link. Here it is. It might elevate the page in google’s scoring but what the heck.

This person’s rather poor outlook on the CBO did not bother me, nor the fact that he is, in my opinion, wrong {he suggests using the rule hint on oracle 10, which is an option but is not, in my opinion, a mighty good idea as (a) who now knows the rules of the Rule Based Optimizer and (b) the hint will be ignored if you are using most new features added since 8, such as bitmap indexes or IOTs or partitioning. It might {and I have no proof for this} cause the feature not understood by the RBO to not be used, so maybe ignoring a nice bitmap index or function based index. Oh, and (c), if you have good stats the CBO usually wins.}.

What irritated me was his/her high-handed and abusive posting. That really annoys me. Then I thought “no, we all lose our temper sometimes and the person they are having a pop at did kind of ask for it”. But because I think that being abusive or condescending on forums is such bad behaviour, I dug a little into the other postings by this person.

Some were helpful. Many were simply links back to other pages or to some front end to Google the person’s question. And several, many, were abusive. Along the lines of “Why are you so stupid”; “If you can’t be bothered reading the manual you don’t deserve help”; “I would not employ you as you are a moron”. You get the idea?

It is a big problem with forums, and actually also in the work place (and occasionally, sadly, at meetings and conference). People being condescending, antagonistic and demeaning to others who do not know what they, the Mighty Brain, knows, who seem to Mighty Brain to not be trying quite as hard as they could or are seemingly asking something obvious.

OK, if it is obvious, give the answer. It might be that you, oh Mighty Brain, did not understand the question. OK, they maybe are not trying hard enough. Suggest to them where they could look, maybe this person has 3 managers breathing down their necks and they just really, really want an expert opinion now as they are not sure which of the seven opinions in google to trust. And Mighty Brain, unless you were born with your knowledge placed in your head by God, as a special force on this earth, you didn’t know what this “moron” does not know at one time. Someone told you or, vary rarely, you worked it out for yourself.

The truly, blood-boilingly, unjust thing about Mighty Brain? They are so sure of their own towering knowledge that they can’t see that they are often wrong.  I can’t think of any Oracle Expert who is widely accepted by their peers who is a “Mighty Brain”. In fact, a common trait of the very best practitioners and teachers (of any subject, not just IT) is that they are always willing to admit they do not know and to learn.

I did look for a way to ask for this particular Mighty Brain to be barred from the forum, but then I just decided to vent my spleen on my blog and have a glass of wine.

I hope that person trips over and really cracks their shins or something. Nothing permanent, just something incredibly painful. Grrrrrr.

Update – Jonanthan Lewis has commented to let me know that this “unskilled and unaware” {what a brilliant phrase} is the Dunning-Kruger effect. The link got filtered out by the comment mechanism, so I’ve posted it {well, maybe a similar one} here. Thanks Jonathan.

Fear of Databases May 29, 2009

Posted by mwidlake in Management, Perceptions.
Tags: , ,

“It’s all in the database!”

I’m sure most of you (if you are in the UK, I must remember that the web is a world spanning medium) have seen the adverts by the wonderful TV Licencing authority or the DVLA. If not they go something like:

“We keep records” {background music}
“We know if you have paid…or not.” {Music become more sinister}
We will find you, you cannot hide” {more affirmative music}
“It’s all in the database” {doom-laden musical flourish}

OK, maybe I lay it on a bit with the music.

Now, as a database professional, I see “it’s all in the database” as a good thing. With luck it will be a well designed database with referrential integrity and all nicely validated.

Nearly all news media stories about actual or perceived threats to electronic privacy also site “The Database” as the core.
“They {who?} will hold all your web searches in a vast Database”.
” A laptop holding a Database of 1 million double glazing customers has been stolen”. I bet it was actually 10 thousand and in a spreadsheet.

It’s getting to the point where I don’t feel comfortable telling people I meet outside of the IT world that I am a database expert. Databases are hardly ever now seen in a good light, they seem to be linked only to things bad and Orwellian.

The Database is also often cited when companies get things wrong for their customers. You ring up to complain about some aspect of non-service and are often told “Oh, it doesn’t agree with you in the Database” or “the Database has got it wrong”. No it hasn’t, the person putting the information in the database got it wrong. I’ve been in the unusual situation of being told a lie where the database was given as the cause but I had access to that database. So I checked and the database was fine. It was being used as a convenient and much maligned excuse.

Very little is mentioned of the beneficial uses of databases.
For most of us our salary is processed via databases and it is a lot cheaper and more reliable than having half a hundred pay clerks doing it manually in pen and ink.
Databases are used to hold or index much of that vast quantity of stuff that you can search for on the net. Even the useful stuff on Klingons.
I for one would welcome a UK-wide database holding my basic medical details so that when I go to my GP or hospital, they do not need my memory (and in fact my consciousness) to tell them my medical past. If I have an allergy to a common drug I damned well want all medical people to know that before they put 10cc of the stuff into my veins.

And to wrap up my bad-tempered tirad, I now find it particularly tricky to talk about what I still feel is my most significant achievement in IT, namely an 80TB Database of genetic information. Without getting into the topic of Bioethics, which is beyond the scope of this blog, Genetics and a lot of biological stuff is now painted grey, if not deep, murkey, scary Red by the media. I tell John down the pub that I created a huge genetics database, he is sure I am either working on a secret government project to know all about his inner workings or some evil company combining tomatoes and monkeys into some awful, new thing that {and he has seen the movies to prove this} in all likelihood will turn into a zombie killer, escape and do for mankind.

Maybe I’ll just tell people I shoplift for a living, it might be more socially acceptable than being involved in Databases or Genetics.


Get every new post delivered to your Inbox.

Join 176 other followers