jump to navigation

Testing is Not Just for Code. September 16, 2009

Posted by mwidlake in Architecture, VLDB.
Tags: , , ,

Someone I am currently working with has a wonderful tag line in her emails:

Next time we want to release untested, why don’t we just release undeveloped?

Testing is not limited to testing code of course. I have recently posted about how a backup is not a backup until you have tested it with a practice recovery.  How you think the database will work by looking at the data dictionary is just a nice theory until you run some actual tests to see how the database responds, as I have been doing with Histograms lately. Sadly, you could even say an Oracle feature is not an Oracle feature until you have tested it.

In my experience, this is particularly true when you test the edges of Oracle, when you are working on VLDBs {Very Large DataBases}.

Last month Jonathan Lewis posted about a 2TB ASM disc size bug, where if you allocated a disc over 2TB to ASM, it would fill it up, wrap around and write over the begining of the file. This week I heard from some past colleagues of mine that they hit this very same bug.
With these very same colleagues we hit a big in 10.1 where you could not back up a tablespace over 8TB in size with RMAN {I can’t give you a bug number for it as we were working with HP/Oracle direct at the time and they “handled it internally”, But when I mentioned it to him, Jonathan found a similar one, bug 5448714 , which stated a 4TB limit on backups. It could be the same bug}.

Yet another VLDB issue was we wanted to move just under one thousand tablespaces from one database to another {again, 10.1}, using transportable tablespaces. We tried to use the utility for checking you are working on a consistent set of tablespaces, but it could not cope with that many. But to plug them into the new tablespace you have to export the metadata and we found a 4000 character limit on the variable stating the tablespaces to transport. That’s 2.3 characters per tablespace, as you need comas to delimit them…Yes, you could manage if you renamed all tablespaces to AA, AB, AC…BA.,BB, BC etc.  If memory servers, the problem was with data pump export and we reverted to old style export which did not have the problem.

Another limit I’ve blogged on is that the automated stats job chokes on very large objects.

Some Data dictionary views can become very slow if you have several tens of thousands of tables/extents/tablespace/indexes

I can appreciate the issues and problems Oracle has with testing their code base, it is vast and people use the software in odd ways and it has to run on many platforms. You might also feel I am being picky by saying Oracle breaks a little when you have 8TB tablespaces or a thousand tablespaces. But

  • Oracle will say in big, glossy presentations, you can build Petabyte and Exabyte databases with Oracle {and have a product called Exadata, don’t forget}.
  • More and more customers are reaching these sizes as data continues to grow, for many site, faster than mores law.
  • Some of these limits appear with databases well below a Petabyte (say a tiddly small 50TB one 🙂 ).

I’ve been running into these issues with VLDBs since Oracle 7 and they are often with pretty fundamental parts of the system, like creating and backing up tablespaces! I think it is poor show that it is so obvious that Oracle has been weak in testing with VLDB-sized database before release. 

I wonder whether, with 11gR2, Oracle actually tested some petabyte data sizes to see if it all works? After all, as is often said, disk is cheap now, I’m sure they could knock one up quite quickly…


1. coskan - September 17, 2009

Isn’t it better to claim that it works and patch it when it start not to work 🙂 After you sell the price all you need to do is create the reason to buy maintenance and I think this is a good reason to buy maintenance. As I comment before till someone do better DB we will probably discuss this kind of issues for a long time.

And what is VLDB ? What makes a DB a VLDB?

Maybe it was 100GB, 5 years ago now it is terabyte and looks like it is going to be petabyte. but the term stays same.

Question for you is who is VLDB DBA and who is not? under the above condition?

Will I be VLDB DBA if our DB hits to 1TB size or I will still be tiny DB DBA ?

Regular readers future post request: You already covered a bit in this post but can you also write a post about what is the difference in daily life of a VLDB DBA, normal DBA, DW DBA. What would you ask me different if you make an interview to me for a VLDB DBA position.

2. PdV - September 17, 2009

It is not the size, it is how you use it…

An Oracle employee cut that disucssion short once:
“A VLDB is any database that is too large for its hardware.”

In my opinion, that makes a VLDB-DBA:
“any DBA that runs into trouble because of the size of his database.”
But like Coskan says, the VL limit tends will move upwards. Fast.

A VLDB generally can not have a representative live-size test environment to experiment on.
And a VLDB DBA will be busy with plotting “sizes” of various components, will be weary of any resource consumers, and will scrupulously analyze run-times of components. Prediction of “size” and “time”, and deriving those predictions from too-small-test-sets are a major issue for a VL DBA.

On the testing and quality angle: Be careful what you write.
I once made a career limiting move by criticising an Oracle VP (sales for some provincial region). I told him, jokeingly(?),
“Your promise becomes my problem, and you call me certified-advantage-gold-partner for that”.

NB: It is no different with other large vendors, but Oracle can be particularly enthusiastic on the sales-side.

So.. Testing, testing, testing. Prove something works.
If some “solution” doesnt work, Oracle (sales) will weasel out, Oracle (support) will stall, and my customer will become nasty… to me.
And, like the Sales-VP would say: Blaming a well-known, reputable global product vendor is hardly a professional attitude is it?

That same quality issue is why some of us roam the internet searching for knowlege on what is broken and what works. I tip my hat to Coskan for aggregating his links!

Keep up the blogstuff Martin.

coskan - September 17, 2009

In my opinion, that makes a VLDB-DBA:
“any DBA that runs into trouble because of the size of his database.”

Piet your opinion has just become my favourite quote about VLDB-DBA like.

Thanks for sharing

But I think the generalization about the hardware/data size relationship about VLDB is not exact truth. Lets wait what Martin will bring to the table.

3. mwidlake - September 17, 2009

Hi Piet, Coskan,

I’ll wade in later (up to my eyes in the day job just now) but of course I’ll answere questions for a regular like yourself Coskan – though I think Piet has stolen my thunder, he covers the question very well.

A non-technical definition of a VLDB that takes into account general database growth might be:

A database is a VLDB when 8 out of 10 DBAs say “that’s really big”


4. Martin Berger - September 17, 2009

I basically stick to “any DBA that runs into trouble because of the size of his database.”
Or to add 2 other sentences:
“any DBA thar runs a DB which faces troubles; as the DB does not act as the documentation suggests because of the size of the DB” (a very pesimistic aproach)
“any DBA than must change his daily (and not so daily) tasks and methods because of the size of the database” [backup, restore-tests, even monitoring]
just an idea to keep the discussion running,

mwidlake - September 18, 2009

Thanks for that Martin. I especially like your second point. I’ve added a new post today, specifically on what a VLDB is. I think we share views.

5. What is a VLDB? « Martin Widlake’s Yet Another Oracle Blog - September 18, 2009

[…] this will link to the comments sectionof that post as Piet deVisser added a wonderful comment answering the question for me. Go and read, […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: