Testing Methodology

This is my 101st posting and {almost} my start for 2010. I was going to kick off 2010 on something I knew well, so I could look all smart – specifically, handling stats for big databases. But I have decided to kick off on something I know very poorly. Security. Namely, use of SQL AUDIT. Why? Why blog about something I know little about?

Because I want to blog about the process of finding out about stuff and testing that stuff. It’s actually going to be a thread about testing – the SQL AUDIT is just an excuse {and matches my work life, which is jolly important when time is limited and you need to keep the employer happy}. I started out my whole blog (before almost anyone paid any attention to my meanderings) on what you could find out just from doing a “select count(*)” from a table. You can find out a lot from simple beginings.

How do you test something? Well, that question is actually a step or two down the line from “how do I understand something”. Sorry, I was trained as a scientist at college and you get sold this line as a trainee scientist about Baconian Science whereby you record all the data about something and then come to conclusions. It does not work really. You need an idea first and then you need to test that idea.

So, in the case of IT, you come across something (the initial idea) and you want to know how it works. It is usually a feature of the software new to you. With software (and the Oracle Database is really just a huge pile of software) there is a source of starter information and it is called the “documentation”. If I was a Baconian scientist I would just use the feature and record everything I could think of about it and record what I discovered. But this is software and some human (or humans) designed it and implemented it. And documented it.

So, First thing, don’t google it. Go and read the official manual on it. It is not going to tell you everything (it is, after all, written by people desperate to put it in the best light possible – and yet leave space of consultancy and training dollars to be earned) and remember that the manual can be wrong, but it is likely to be more accurate than 80% of what you come across on web searches. {If I was a tabloid newspaper I would go and ask a bunch of people who knew little about the subject what they thought and then publish an article on it, emphasising any shocking or outrahgeous parts. Remember that when you Google or Bing or Yahoo anything, Anything. }.

Having read (and I would suggest you start with) the Concepts Manual on the feature and then any specific manual covering the feature, now do your web search and…
(a) look for sources you have grown to trust. Not popular sources necessarily, sources where you have rarely found them wrong or they publish examples and worked proofs of their utterences.
(b) look for repeated claims about that feature and keep them in mind. They could be urban myths, they could be accurate. Just keep them in mind.

Now go and do some tests. Yourself. Otherwise, you just never know…

And this is the nub, I think, of testing. Get some basic information about how it should work, throw in there your experience to date (eg, this AUDIT stuff is putting data into an oracle table, so it is inserting and updating rows in a table, it should react like data going into a normal table. What do I know already about insert and updates to tables?) and ask yourself a few simple questions:-
– How do I think this should work?
– How could this go wrong?
– What scenarios can I think of for using this?
– What are the simplest examples I can think of for this process?
– How can I reduce tests down to trying only this feature or even one aspect of this feature?
– How could this go wrong?
Yep, I ask that last question twice and it is no oversight. I want to try and understand where this new feature might break. I need to ask the “ahh but” questions.