jump to navigation

Re-forming the Martin Cluster in Norway April 5, 2013

Posted by mwidlake in Friday Philosophy, Meeting notes, Presenting, Uncategorized.
Tags: , ,
7 comments

Later this month, on April 17-20th, I am presenting again at the Norwegian Oracle user group (OUGN) spring conference {modern browsers will allow you to translate any Norwegian if you need to} . I loved it last year, as you can see from my third-day post on it. I’ve been lucky to attend some very good conferences over the last few years but those three days at the OUGN conference last year were, I think, my favourite single event to date. If you are within striking distance of Oslo and can negotiate the time out of the office, I would strongly recommend the event. If you can’t negotiate the time, heck, take a holiday and go anyway :-)

Part of what I really enjoyed about the event was the fact that two of the days are spent on a ferry/cruise ship from Oslo to Kiel and back. And oddly enough, that is what initially put me off going to the conference – I am very susceptible to Sea Sickness. I had no problems though, partly due to the large quantities of travel calm pills I swallowed, partly due to the good weather, but mostly because the talks were excellent and the atmosphere was fantastic. I don’t mean “hey, it was a bit like a holiday” {though in some ways it was as it was such fun} but because the speakers and the attendees can’t escape, at least not without a long swim, everyone is around all day and all evening. It just adds to the whole event. I spoke to more “new” people during that conference than I ever have before.

At most conferences the presentations at the end of the day tend to be less well attended and there can be a feeling of wind-down, especially on the last day. A fair number of people feel the need to make an early exit to get home before the worst of the traffic or are under pressure to get back to the office and just sort out some issue that is pressing. The people around in the evening tend to be the presenters and the conference die-hards and so are usually the same sad old bunch of geezers and gals :-) . However, on the OUGN Boat this is not the case. All sessions tend to be well attended and in the evening nearly everyone is out having something to eat, a few drinks (those North Europeans sure do like the odd drink, but in a relaxed and affable way) and just being sociable.

Over the last few years the conference has developed a reputation for being technically strong too. This is of course partly due to the excellent atmosphere attracting good presenters and the good presenters in turn help make the conference better. popular and well attended – and that in turn attracts presenters. A nice positive feedback loop. I certainly learnt a lot of stuff last year and I cannot think of a poor presentation that I attended. Hmm, maybe one of mine was a little weak :-| . The organisers do an excellent job of helping the presenters feel relaxed and appreciated too. For example, I was nervous about the boat part of the trip to they gave me one slot on the mainland the day before we sailed and suggested I could bail out at Kiel if I was suffering. As a presenter, that sort of consideration counts for a lot. I don’t want or expect to be treated like some minor celebrity and I was not, but for the whole conference I just felt like the organisers appreciated my taking time out from work and flying out to come and present.

The final reason I am looking forward to the event (and thus the odd title) is the re-forming of the Martin Oracle Cluster :-) – this is myself, Martin Nash and Martin Bach. We all do several conferences a year, we all used to go along to the London Oracle Beers and we have become good friends. Other Oracle Martin’s are welcome to join us – At the OUGN last year there was also Martin Büchi, who I had not met before, but this year I think we are the only Martins presenting. We just don’t seem to have managed to re-from the cluster for many months now, partly as Mr Bach returned to Germany.

Martins_1

Martin Nash – Martin Büchi – Martin Bach – Martin Widlake
Thanks to Øyvind Isene for the picture.

I suppose I should mention what I am presenting on? Well, as I mentioned in my last Friday Philosophy, I am concentrating more on introductory presentations. You can see my official agenda here. I am doing:

  • an introductory presentation on Row Level Security, VPD and hiding rows or columns of data {it will be interesting to see how large the audience is for that one!}
  • an introduction to SQL tuning where I cover the absolute basics, but hopefully in a way that allows those new to it (or maybe even not so new) to treat tuning as a logical and sensible process, as opposed to Black Magic of favourite hints and arcane practices
  • my disasters talk. I love giving my disasters talk. I’ve “been in the vicinity” of a lot of disasters and I only ever talk about things I have seen first hand, so no urban myths.

Friday Philosophy – Level of Presentations March 8, 2013

Posted by mwidlake in Exadata, Friday Philosophy, Perceptions, Presenting.
Tags: , ,
7 comments

This FF is a bit of a follow-up to the one I posted last week on PL/SQL skills and a comment made by Noons on how much knowledge you need to be an OakTable member.

I have a question to answer and I would appreciate other people’s opinion. Should there be more intro talks at conferences? If so, should the experts be giving them?

I am an OakTable member (and really quite worryingly proud about that) and I guess that means I know a lot about some aspects of Oracle. But also, and this is a key part of being asked to be a member of the OakTable, I both try and base my “knowledge” on scientific testing of what I think I know (be it by myself or others willing to show their workings) and I try and pass that knowledge on. I don’t think there is a member of the OT that does not abide by those two concepts.

This is not false modesty on my part, but most other people on the OT know a shed load {UK colloquialism for “really quite a lot”} more than I do about the Oracle database and how it works. Some of them are so smart I can’t but secretly dislike them for it :-). But I have a reasonable stash of knowledge in my head and I am a stong proponent of those last two factors. In particular, I want to put what I have in my head about {ohh, let’s pick partition pruning} in other people’s heads. Which is why for the last 4 years most of my presentations have run over horribly. I must get every detail into the audiences’ heads, even if they don’t want to know it!

Of late I have started to feel that I present in the wrong way. I take a subject, I write down what I know about it and I try to present that knowledge. I’ve only picked topics I feel I know well and in which I can handle almost any question the audience asks. For me to be that confident I have to have used that aspect of Oracle technology a lot and had months, if not years of experience. You cannot pass that on in 1 hour. I’ve already said I am not that smart, but I learn over time. So I started to strip out the basics and present just the clever stuff, which shows how fabulous I am. {British self-deprecating sarcasm, please}. Just like so many other experts. Hell, if we are experts, we should be talking expert stuff!!!

To balance that I think there is a gap in what is talked about at conferences. I know this is felt by some conference organisers and attendees too, but there is just way too much “impressive smart stuff” and not enough “how X works in the real world, for people who have never done it”. Not 10,000 feet up sales pitch rubbish that gives no substance, but talks on a topic where you can go from level 1 for knowledge-free beginners to level 5 for the 25 people at the conference who know this stuff inside out – and the talk stops at level 2. I’ve made a decision to try and address that gap, which is why I now offer more talks on “an intro to Exadata” and “how to get going with performance tuning” than the smart stuff.

The problem is, how many people, probably mostly young people, go to conferences? Am I wasting my time trying to offer these talks if no one going to the conferences wants them? Do people going to conferences, generally speaking, want the technical nitty-gritty or do they want the intro stuff? Yes, I know there is a spread but where is the real need? I suppose I thought of this recently when I did a talk on Index Organized Tables and almost everyone in the room already used them. Then, a few hours later, I did an intro to database design. Almost everyone in the room was a seasoned database designer… I doubt I said much of real value to either audience.

That leaves my last point about should the experts do intro talks? A problem with experts doing intro talks is the expert knows it all. It can be hard to remember what you really needed to know at the start (and also, my own problem, knowing what to drop out the talk as, really, it is “being smart as an ego-trip” that the new people can do without). But if you are not an expert, can you handle the What If questions? I have played with this issue with my Intro to Exadata talk. I wrote the draft when I had very little real experience and I have only modified it as I gained more experience. I’m glad I did as when I revisited the presentation recently I started putting loads of stuff in that only makes sense when you are used to it’s peculiarities. Thankfully, I realised this and stripped most of it out again. So well, in fact, that one person who wrote about the talk said “it was nice to see someone talk about it who was an Oracle expert who obviously knew little about the topic”. Well, something like that :-)

Enough. I would appreciate other people’s opinions and experiences on this.

Friday Philosophy – Do good DBAs need PL/SQL Skills? March 1, 2013

Posted by mwidlake in Friday Philosophy, PL/SQL.
Tags: ,
15 comments

This Friday Philosophy was prompted by a discusion between some OakTable people about did we think “good” DBAs should know PL/SQL? Not all the tricks, bulk processing, using all the built-ins, but able to write PL/SQL with cursor loops and some exception handling that could eg cycle thorough tables and archive off data or implement some logon trigger functionality.

My response was “that depends on the age of the DBA”.

If you had asked me that question 15 years ago I would have said Yes, a good DAB would and should know PL/SQL.
If you had asked me 10 years ago I would have said I’d hope they would and most DBAs I respected has some PL/SQL skills.
If you had asked me 5 years ago I would have sighed and had a little rant about how they should but the younger ones don’t and that is wrong.

But now, I would say that no, a good DBA does not need PL/SQL skills as so often they have so many other things they have to do and the tools to manage the database are somewhat better than they were. But inside I would still be thinking any DBA beyond their first year or two in the job would benefit from knowing the basics of PL/SQL.

It seems to me that a DBA now is generally expected to look after a very large number of instances, application servers, agents etc and all their time is taken doing the bread-and-butter tasks of backups and recoveries, patching, duplicating data, raising SRs (and that seems to take more and more time each SR every year), unlocking accounts, sorting out permissions…

Not only that but there is more and more to Oracle that a good DBA needs to understand as the technolgy gets more complex. Oracle has tried to make Oracle look after itself more but the result seems to be that for larger systems there are more moving parts to go wrong – and when they do it is often the DBA who has to sort it out. As an example, you no longer need to set several instance parameters to allocate memory to the components of the PGA and SGA. Just set the Memory Target. But if the system starts to throw odd errors about components being out of memory, the DBA needs to sort that out. They need to know about the dynamic memory adjust ments and check them out. They need to understand that certain components are now calculated in a different way, like the log buffer size. And probably they will have to revert back to the old parameters so they still need to know all about that!

So unless a DBA is an old hand and “grew up” with all this, they have no time to develop PL/SQL skills. Thus my response was “that depends on the age of the DBA”.

Should a good DBA know SQL?

Yes. I still see that as a given. Buttons, widgets and assistants will only get you so far.

RMOUG – Here I Come February 9, 2013

Posted by mwidlake in Exadata, Meeting notes, Private Life.
Tags: , ,
add a comment

Well, I’ve just finished pushing the last few bits into my suitcase for my trip to the US for the Rocky Mountain User Group Training Days 2013.

It is a few years since I went to the US for pleasure (3 years?) and much longer since I went there on a combined work/pleasure trip – as I HATE going through US immigration.

I was tempted into this trip when I met up with a fellow OakTable member Tim Gorman at the Slovenia User Group last October. Tim was a true gentleman throughout that conference (and this is not to detract from the kind ministrations of Jože Senegačnik who was a wonderful host and also a fellow OakTable member) and he suggested I put forward a talk or two to the RMOUG training days conference. I was in two minds due to my huge dislike of being shouted at by the {I am sad to say, usually bl00dy unfriendly} members of the US immigration services. I mean, I went to Moscow in around 1994 and those chaps in the USSR were positively oozing cordial welcome compared to the hard-nosed and antagonistic attitude of the US chaps a year or two later!

Anyway, Tim swayed my opinion in a simple and direct manner. He mentioned Skiing. I went skiing in 1992. My one and only Skiing holiday. I loved it, I spent 2 weeks going from terrified starter to someone who could swish down Blue runs with the rest of them, the occasional Red run towards the end. Not Black, I saw enough sad-looking stretchers coming down from the Black runs as I sat in the bars at the bottom to think there was something to be scared of there – and when I chatted to the barman in my chosen watering hole he told me how so many of them were us Brits who had lost the brains to realise what our limits were.

So, I will be at the conference doing my bit about “first things to know about Exadata” and then I will be up in the mountains, scaring myself on hills that when I was 20 looked like a walk in the park…. What is the betting I do not get as far as Red runs this time?!?

Friday Philosophy: It’s Not What you Know – It’s What you are Getting To Know. December 7, 2012

Posted by mwidlake in Friday Philosophy, Meeting notes, Perceptions.
Tags: , , , ,
add a comment

This has been a good but tiring week. It started with the UKOUG TEBS conference where I saw lots of people I know, a few who I didn’t but now do and I had good times in pubs and restaurants. One evening culminated in my singing part of “Two out of Three ain’t bad” in the style of a munchkin with Nial Litchfield in a pub at 1am, which I am sure he woud rather forget about – so if you know him, ask him about it. For me that was the indicator to go lie down in the dark and sleep. Irrespectve of drunken singing, I must had talked about 20 topics with 40 people over the conference, exchanging ideas, tricks and war stories.

I also presented on the Sunday, teaching people. I went to many fine presentations through the week and learnt stuff, balancing the knowledge books. This included one talk on Wednesday where I found out I had been teaching people wrong on Sunday! *sigh* Oh dear. And it is something I have been getting wrong for years. I’ll check the fine detail before I blog about that though. I now hate SQL Maria for pointing out my error – which is unfair but that’s life. Sorry Maria, I’ll buy you a beer sometime to make up for it.

After all that, I came back to work to find we had some big issues with one system. I won’t go into the details but there were half a dozen of us working on it, coming up with ideas, knocking each other’s suggestions down or agreeing with them – and coming up with working solutions. None of us could have done it alone and I reckon most of us learnt a couple of things in the process (such as how bad some of the app design was :-) ).

So all in all, a week of learning that has worn me out but left me happy and that is what I like about my work life at present. I am working with people who know stuff and who want to know stuff. I am in a role that exposes me to a lot of issues. I am working for a boss (and team) who let me take time out from the official job to learn stuff too. I suspect it is going to end soon though as the company I am working for are taking large numbers of people to the carpark and shooting them (figuratively speaking only, I should point out). It will be a shame if it does end. But then, just as when I worked with the best IT person I ever worked with, I’ll be happy I had the experience.

Yes, I know I am lucky to have such a good working life at present {but, you know something? I did take efforts to make it like this – maybe a future FF on that}.

So having said I like my working life so much, who is willing to bet I’ll be complaining like a spoilt child all next week?!?

Friday Philosophy – The Tech to Do What You Need Probably Exists Already November 30, 2012

Posted by mwidlake in Friday Philosophy, Perceptions.
Tags: , , ,
4 comments

How many of you have read the Oracle Concepts manual for the main version you are working on?

This is a question I ask quite often when I present and over the last 10 years the percentage number of hands raised has dropped. It was always less than 50%, it’s been dropping to more like 1 in 10 and Last year (at the UKOUG 2011 conference) was the nadir when not a single hand was raised. {Interestingly I asked this at the Slovenian User Group 3 months ago and something like 40% raised their hand – impressive!}.

Why do I feel this is important? Well, do you know all the technology solutions available across just the core RDBMS with no cost options?. No, you don’t, you (and I) really don’t. If you read the concepts manual, even just skimming it, you will be reminded of a whole load of stuff you have only dim memories of and perhaps you will even see some features that passed you by when they were first introduced.

Of course, you would need to read a few more manuals to get the full picture, such as the PL/SQL Packages and Types References, as so much good stuff is introduced via built-in packages, and the SQL Language Reference, as SQL has been extended quite a lot over the last couple of versions. Dull reading indeed but I’d estimate that if you read those three you would be aware of 90%+ of the Oracle technologies that are available to you out-of-the-box and considerably more than all but a handful of Oracle Experts. You’d know more than I as (a) I have not skimmed the PL/SQL one for years and (b) I have a rotten memory.

My point it, you can’t consider using Oracle technologies you don’t know about or remember – and they could be just what you need to fix the problem you see in front of you.

I’ll give a couple of examples.

Problem, physical IO is too high, your storage system is bottle-necked.
Answers, Reducing Physical IO:
-First up, Index Organized tables. Some of you will be aware that I am very keen on IOTs and the reason I am is that I’ve used them to physically group data that the application needed to select over and over again. It can make a massive improvement to that sort of system. They are rarely used.
-Clustered Tables. Even less used, in fact has anyone reading this used them in anger in the last 10 years? Great for situations where you need parent-children or parent-children-type-1+parent-children-type-2 data. I confess, I have not used them in anger for years.
-Move table (and order them as you do it!) and re-build indexes to remove “dead” space. This one got a bad name, especially the index rebuilds, as people were doing it needlessly without appreciating what the intention was, but now I hardly ever see it done – even when it is of benefit.
-Compress your tables and indexes. With normal Oracle compress (no need for HCC). In tests I’ve done I pretty much always see a drop in physical IO and run time. Being candid, I can’t remember doing any tests and NOT seeing an improvement but I usually only test when I expect and improvement and I don’t want to give anyone the impression it will always help.

All the above were available in Oracle 7 or 8 and all have improved over the versions.

Problem, you want to carry out some long,complex data processing in PL/SQL and if a step fails, be able to handle it and carry on.
Answers:
– Savepoints. You can rollback to a savepoint, not just to the last commit. In this way you can break the task into chunks and, if something towards the end fails, you roll back one step (or several, your choice) and call an alternative routine to handle the exception.
– Autonomous Transactions. You want to record that an error occurred but not fail the original action or not save anything it has done to date. An autonomous transaction runs in it’s own sub-session and commits in it do not effect the calling session.
– Temporary Tables. You can put your working information in them as you progress and if you need to bomb out (or some evil DBA kills your session for running too long) the temp table contents just disappear. No clean up needed.

Maybe the above is not so fair, I have not been a proper PL/SQL developer for a while now, but I hardly see the above used. Especially Savepoints. I can’t remember not having savepoints available (hmm, maybe Oracle 6) and Autonomous Transactions and Temporary tables are Oracle 8 (I thought Temporary tables might be 9 but Tim Hall’s OracleBase says 8)

Another thing I have noticed over the years is that so often I will read up on some oracle feature I know little about, only for it to come up in the next few weeks! There is a psychological aspect to this, that we only remember these “coincidences” and not the more common situations where we read up on something which does not subsequently come up before we forget about it, but I think that it is also that we tend to use only a few solutions to solve the problems we see and adding another solution to our list means the chances are high it will be suitable for something soon.

OK, so it would help us all to read the manuals (or other Oracle technical books) more. Now the big problem is finding the time.

Why Present (UKOUG TEBS 2012) November 27, 2012

Posted by mwidlake in Friday Philosophy, Presenting.
Tags: , , ,
3 comments

The annual UK Oracle User Group Technical and E-Business Suite conference is fast approaching and rather than just say “hey everyone, I’m going to present at a conference” I thought I would say why I present at conferences (and SIGs (and any opportunity I get) ).

The UKOUG TEBS conference is the one that is, in my eyes, the UKOUG conference. It covers E-business suite and the core database stuff – DBA-type things and developer-type things. The things I am interested in. You can see the details and agenda via this link. If you would like to see someone else’s view of this year’s conference, from a delegate’s perspective, check out this post by my friend Big Dave Roberts. I’ve said it before and I’ll say it again, I’ve personally got more out of the UKOUG TEBS conference than when I have been lucky enough to go to Oracle Open World. For me, I want to see user stories and opinions as well as the Oracle company line – and user groups give far more of that. UKOUG is the biggest and most encompassing of the European conferences, but check out and attend your local user groups if you are not based in the UK (or are in Scotland) – I’ve been to a couple now and all have been good.
{Oh, I should add, I am not presenting at the main TEBS conference, for the first time that I’ve made it there in…10 years! I’m presenting on the OakTable Sunday. I get to fully enjoy the whole 3 days of TEBS this year, as a normal punter!}

So why do I present?

First up, I am lucky that presenting does not scare me. Oh, I get nervous before I start and I worry about my material and doing a decent job of it, but I never feel sick or faint before starting(*). I’ve got some good friends who know as much or more about database technology than I and, in a social situation, they will let fourth with eloquence and passion about some topic. Usually in a pub. But the idea of formally presenting fills them with the heebie-jeebies if not the screaming abdabs. Or, when they get up to talk, they morph from erudite and eloquent {ie they speak clearly, concisely and interestingly) into stuttering, uncomfortable rabbits caught in the lights. And they quickly drop back from it. It’s a real shame as they have a lot to say. Actually, it is a blessing as it leaves space open for me :-)

Secondly, I want to be noticed. There, I’ve said it. I’m pretty sure that 75%+ of the regular speakers at conference want to be noticed and that is a factor in why we do it. Oh, of course, we can wrap it up as “marketing your skills” or “maintaining a presences” but that’s fluff. We want you to see us talk and we want you to like what you see. Well, not the physical aspect of it, that would be weird, but we want you to either think we know what we are talking about or enjoy listening to us prattle on. We want to be appreciated. That’s not so noble, huh? But true.

As a sub-topic to “I want to be noticed” let’s deal with getting work. For me personally, speaking has never got me a job. Asking around friends and contacts most people are the same in that Presenting does not get most of us any work. Maybe it’s a factor when I get interviewed but no one has once come up to me after a talk and said “can you come to our site and be paid a vast sum of money to solve this problem”. I’d love you to be the first, though…

Third, and this may seem noble – or corny or self-worthy – but I talk because I want to teach. I want you to have in your head the knowledge I have in mine, preferably with less pain and anguish than it took me to get it in mine. I love teaching people stuff. It is a common trait to the OakTable network. We teach and we demonstrate why we think what we think. Usually. I can’t decide if I prefer making something that is broken work or teaching people more. Those two are why I do what I do. Frankly I gave up on making a “significant difference” years ago, I’ll stick to making lots of insignificant differences.

Fourth, I get to meet people. It’s taken a while as, except in small groups, I am not great at being sociable. That might seem odd given I said I like to be noticed, but I don’t like to be noticed as a fool and for reasons I will side-step, I’m not too great at judging how I am coming across. Plus I cannot remember names or faces, which is tricky sometimes. Actually, a lot of actors are not “social people”, for some they act because they can’t “do” people. Anyway, presenting has opened doors to me by meeting other presenters who know their stuff, non-presenters who know their stuff and, generally, people. Most of whom have been nice.

Fifth, and this is quite rare. I get asked. If you ask me to present the chances are I will say “ohh, yes please!!!!” We are back to my second point about being noticed, aren’t we? Caveat. If you ask me to present and you live thousands of miles from the UK, as I have no employer to fund my demands for limousines and four-star hotels, I might have to say no. Unless you buy me a LOT of beer.

That is why I present. You probably now expect me to say something to encourage you to present but, mehh, if you want to, just do it. If you don’t, don’t do it. It’s not for everyone. Except you over there. Yes, you know who you are. You need to present more. :-)

(*) I confess, when I do a talk that is aiming to be funny, I get very nervous. When I do my “disasters” talk I can get very nervous as the whole idea is that people will be amused. If I get the feel of the talk wrong I can look preachy or obnoxious or bitter or daft or all four and more. But it is still the talk I most enjoy giving.

Friday Philosophy – The Importance of Context November 23, 2012

Posted by mwidlake in Friday Philosophy, Perceptions.
Tags: , , , ,
1 comment so far

A couple of weeks ago I was making my way through the office. As I came towards the end of the large, open-plan room I became aware that there was someone following behind me so, on passing through the door I held it briefly for the person behind me {there was no where else they could be going}, turned left and through the next door – and again held it and this time looked behind me to see if the person was still going the same way as I. The lady behind gave me the strangest look.

The strange look was reasonable – the door I’d just held for her was the one into the gentleman’s bathroom. *sigh*

I was doing the correct thing, I was attempting to be helpful to a fellow person, I was in fact being very polite. But because I had utterly failed to consider the context, there is now a lady who works on the same floor as I who considers me, at best, as strange. At worst she thinks I am very strange – and more than a little creepy. I fear the latter given her reaction when she saw me in the kitchen area recently and turned around. {By the way, if anyone can think of a good way I can clear this up I’d appreciate it. After all, I can’t exactly go up to her and say “sorry about holding the door to the gents for you the other day, I did not realise you were a woman”}.

My point is that you can do what you believe is the right thing but, because you are not thinking of the context or are unaware of the full situation, you end up giving utterly the wrong impression. I had a work situation like this a while back.

Without going into too much detail, I was working with a client on a data warehouse project. The Oracle database bulk-processed large quantities of data, did classic big-data queries and was sitting on some fairly expensive hardware with dedicated storage and the intention of implementing Dataguard. One of the issues they had was with a subsidiary part of the system that created a very large number of small transactions, lots of updates. High volume OLTP on a DW setup. It was hammering the storage and eating up all the available IO. The data for this subsidiary system was transient, no need to protect it.
I realised that the hardware was not correct for this subsidiary system and it needed no archived redo. Archiving redo is an all or nothing situation for an Oracle instance. So happy that I had worked out what to do I proposed {with a smile} moving the subsidiary system to it’s own database on it’s own hardware.
When I said this to the client, their response was a stony look and the comment “We’ve just spent a fortune on this platform……”. Having dug my hole I proceeded to jump right in there “It’s OK, what I am proposing is only about 5, 7 thousand pounds of kit – nothing compared to what you spent already!”. The client now got very, very annoyed indeed.

You see, the context is that they had been sold a system that was very expensive – it was to do a demanding job. They had been getting poor performance with the system and that is partly why I was there. They also did not really understand the technical nuances well (at least, not the chaps I was talking to) and they did not appreciate why I said what I did. From their perspective, this smiling loon was suggesting that a system costing 2-3% of what they had spent on their data warehouse platform was going to be able to do the processing that the expensive system could not. Either they had spent waaay too much, this new “expert” was an idiot or else I was lying to them. And they did not like any of those options.

Looking back it is clear I should have been more aware of how they would receive what I said. I’ve done this before {several times}, bounded into a situation like a wide-eyed puppy and gone “Look! We can just do that!” without considering things like upsetting the guy who had suggested the original solution, or making the on-site expert look stupid or blowing away a salesman’s pitch. Or that I have missed a glaring and valid reason why they can’t “just do that”.

I suspect that a few people would say “no, you just tell them the way it is and if they don’t like it or you upset someone then tough”. Well, maybe, but not if you want to be there to help fix the next problem. Also, I know I am not great at appreciating the context sometimes. That is part of why I will never run a company or be a senior manager, I lack the skills to judge the impact of what I propose or say sometimes, in my rush to be helpful. I am slowly learning to just hold back on ideas though and to run things past friends or colleagues with more “whole picture” skills first though. I might be rubbish at it but I can learn I am rubbish at it.

In the case of the situation above, the expensive system was correct for what they wanted to do – and maybe not quite expensive enough. I was suggesting a slightly unusual fix for a specific problem and I should have been more laboured in explaining the problem and more circumspect in leading them to the solution. I should have taken more time.

I should have checked who was following me and where I was going before I held the door open.

Row Level Security 3 – In Pictures! November 22, 2012

Posted by mwidlake in database design, security.
Tags: , ,
3 comments

<..Part one intro and examples
<….Part two Permissions

I’ve noticed that there has not been a lot of traffic on this series on Row Level Security (data masking) so far – maybe due to how I am presenting the material? So here is a summary to date in picture/diagram format:

This is the flow of logic as to whether Row Level Security is applied to the SQL statement. Usually this is a query and usually it is on a table but the policy can be applied to other DML and also on views and synonyms:

These are the steps you need to add RLS to a table:

Below is a summary of how oracle fires and uses RLS functions and the predicates they provide:

An finally, a summary of what happens if an RLS function is invalid:

If you have not looked at the previous posts on this topic, more details are in there.

Enjoy!

Row Level Security Part 2 – permissions November 20, 2012

Posted by mwidlake in internals, security, SQL.
Tags: , ,
3 comments

<..Part 1, introduction..
..Part 3 summary in pictures..>

In this second post on the topic of “an introduction to Row Level Security” I want to cover a few things about what permissions you need to implement RLS and some of the consequences. In my introduction in part one I just said my main user has “DBA type Privileges”.

{NB This is all on Oracle V11.2 and I believe everything below is applicable to V10 as well. Also, I should point out that I am not an Oracle security expert – but despite repeatedly saying this, it seems like at least once a year I am asked to improve a system’s security on the grounds of “more than we have now is an improvement”}.

Firstly, you need the permission to create and drop RLS policies. This is controlled by having the EXECUTE privilege on SYS.DBMS_RLS. That is it. If you can execute this package you can create and drop RLS Policies, enable and disable them and also create, manage and use Policy Groups.

Obviously EXECUTE on SYS.DBMS_RLS is a privilege you need to restrict to only “empowered” accounts- but you also want to be very careful about your definition of empowered. If an account has this privilege, it can alter (disable!) any RLS policies that ANYONE created on the instance. You may need to think carefully about having several accounts that can alter all the RLS policies across all schemas in your instance.

You might want to create a single user that has the EXECUTE on SYS.DBMS_RLS and use that to implement and control RLS across the instance {and even have the account locked when it is not needed}. Of course, under normal Oracle anyone with SYS access can run the procedure – but you have SYS access under control already, don’t you?… :-).

By default, who has this privilege?

MDW> @obj_privs
Enter value for obj_name: dbms_rls
Enter value for obj_own: sys
GRANTEE              OWNER        TABLE_NAME             GRANTOR  PRIVILEGE    GRA
-------------------- ------------ ---------------------- -------- ------------ ---
EXECUTE_CATALOG_ROLE SYS          DBMS_RLS               SYS      EXECUTE      NO
XDB                  SYS          DBMS_RLS               SYS      EXECUTE      NO
WMSYS                SYS          DBMS_RLS               SYS      EXECUTE      NO

--
-- and who has that EXECUTE_CATALOG_ROLE?
--
select * from dba_role_privs where granted_role='EXECUTE_CATALOG_ROLE'

GRANTEE                        GRANTED_ROLE                   ADM DEF
------------------------------ ------------------------------ --- ---
DBA                            EXECUTE_CATALOG_ROLE           YES YES
IMP_FULL_DATABASE              EXECUTE_CATALOG_ROLE           NO  YES
EXP_FULL_DATABASE              EXECUTE_CATALOG_ROLE           NO  YES
SYS                            EXECUTE_CATALOG_ROLE           YES YES

So, our old friend the DBA role has that privilege, via the EXECUTE_CATALOG_ROLE and IMP_/EXP_FULL_DATABASE.

Let’s go back to creating policies. I’m sticking to row-level RLS for now, not column-level. When you create a policy you basically are saying “when object X is accessed, run function Y and apply the resulting string to all access, as a predicate”

Lifted an example from the manual, but adding in a couple of bits the authors were too lazy to put in { :-) } :-

DBMS_RLS.ADD_POLICY (
 object_schema   =>   'hr'
,object_name     => 'employees'
,policy_name     => 'emp_policy'
,function_schema => 'hr_rls'
,policy_function => 'emp_sec'
,statement_types => 'select');

The function HR_RLS.EMP_SEC returns a string P1

When someone issues a statement that access the HR.EMPLOYEES table the function HR_RLS.EMP_SEC is executed as part of the parse and, internally, the following view is created and placed into the orginal statement:

SELECT * FROM hr.employees WHERE P1;

As this takes place as part of the parse part of the statement processing, errors with the function are found before any effort to access the actual data occur. It is also why it is not simple to avoid RLS – it is done as part of the overall SQL processing carried out by Oracle.

If the function is invalid, generates an error or the supplied “WHERE predicate” P1 causes the final statement to be syntactically incorrect, you get an error.
At this point Oracle fails “secure” and simply won’t fire the SQL statement. Even if you own the tables and have full privilege access to them. I think it is worth a quick demonstration

I am logged on as the owner of the table and RLS function, MDW:-

MDW> select * from test1
        ID CP_NAME                        OTHER_NAME                        MASK_FL
---------- ------------------------------ ------------------------------ ----------
         1 ERIC                           ERIC THE RED                            1
         2 BROWN                          BOB THE BROWN                           2
         3 GREEN                          GARY THE GREEN                          1
         4 BLUE                           BILL THE BLUE                           3
         5 BLACK                          DAVE THE BLACK                          4
         6 PURPLE                         PETE THE PURPLE                         5
         7 RED                            ROGER THE RED                           6
--
--All works OK
MDW> @rls_func_damage
--
CREATE OR REPLACE FUNCTION hide_rows (
 v_schema IN VARCHAR2
,v_objname IN VARCHAR2)
RETURN VARCHAR2 AS
con VARCHAR2 (200);
BEGIN
-- The line below states loser not user
con := (loser='''||v_schema||''' or MASK_FL > 3)';
RETURN (con);
END hide_rows;
/
Function created.
--
-- Let us check the predicate coming from the function
--
select hide_rows(v_schema=>user,v_objname=>'TEST1') from dual;

HIDE_ROWS(V_SCHEMA=>USER,V_OBJNAME=>'TEST1')
---------------------------------------------------------------------------------------
(loser='MDW' or MASK_FL > 3)

-- "loser" is not a valid piece of syntax or a known variable. So...
--
select * from TEST1
              *
ERROR at line 1:
ORA-28113: policy predicate has error

-- I now damage the statement creating the function so that it is syntactically incorrect
CREATE OR REPLACE FUNCTION hide_rows (
...
-- missing quote
con := (loser='''||v_schema||''' or MASK_FL > 3)';
...

Warning: Function created with compilation errors.

select * from TEST1
              *
ERROR at line 1:
ORA-28110: policy function or package MDW.HIDE_ROWS has error

--re-create the function as I started with

Function created.

select * from TEST1
        ID CP_NAME                        OTHER_NAME                        MASK_FL
---------- ------------------------------ ------------------------------ ----------
         1 ERIC                           ERIC THE RED                            1
         2 BROWN                          BOB THE BROWN                           2
...
         7 RED                            ROGER THE RED                           6
--
-- Back to a working system

As you can see, you get ORA-28110 if the function used by the RLS policy is damaged and ORA-28113 if the function returns a non-valid WHERE Predicate. In either case, the data in those tables is now inaccessible. That could be a problem…

It is of course very interesting that RLS is kind of “blind” to the functions it uses until it uses it. You can alter RLS activity if you can alter the functions.

Last post I mentioned that, as you RLS security is based on these functions, you should give thought to who can see them and change them. Hopefully the above highlights that. There is another access aspect to these functions that is key.

In the first example where I use DBMS_RLS.ADD_POLICY, I state function_schema – this is who owns the PL/SQL function. It defaults to the person who creates the Policy. The function is fired in the standard way with DEFINER RIGHTS – i.e. the rights of the function owner. {You could override this when creating the function such that it runs with executioner rights – but that strikes me as a potentially very confusing and a source of much pain and anguish}.

So the function can see what the owner can see, not the executioner. This has two key effects:

  • You can restrict the access to any DB objects that the function requires to the owner of the function. The end user does not need to have access and it may be more secure if they do not.
  • You have to grant access to objects directly to the function owner, via either object grants or system grants. Stored PL/SQL does not “see” roles if executed with definer rights.

So eg if your function is owned by MDW and references a master table called SEC_MASTER.MASKED_USERS then you need to grant select on SEC_MASTER.MASKED_USERS to MDW. No using roles.
{of course, calling a table MASKED_USERS might give anyone acquiring access to the DB a clue that it is important. I tend to call such “security important” tables things that are “obviously” boring, like ROTA_DATA.}

Finally, that WHERE Predicate P1 is in effect added to your SQL statement. It can be pretty much anything. It can reference other database objects. If it references other database objects THE EXECUTING USER MUST BE ABLE TO SEE THEM. Not the owner of the function; that function has been fired and the WHERE Predicate generated already. On actually running the SQL, the executing user must have access to all objects referenced, including those introduced by the P1 WHERE Predicate. Oh, and remember synonyms!

There is there a way to sidestep RLS and you sometimes need to.

If your logon has the EXEMPT ACCESS POLICY system privilege then RLS functions are utterly ignored by you. No check is made to see if policies apply to your SQL statements, let alone run them. You may need to use this privilege if you need to access functionality that RLS has issues with (see comments by Dom Brooks and Tony Sleight on my first post). You might also need it if you have errors with the policies but you need access to the data to keep the business moving. You probably need a locked-down user somewhere with the EXEMPT ACCESS POLICY privilege if you use RLS.
Of course, great care has to be taken in making use of this privilege as it side-steps all RLS_based security.
If you have master users that have full access to the data, this privilege also removes the overhead of the SQL engine identifying any policies, firing the function and including the extra predicates.

One very final point. I have found myself writing RLS functions that all check if the user was the owner of the table and, if so, allowing them to see all data. By granting EXEMPT ACCESS POLICY to that user I was able to do away with those checks and make the functions much simpler. In this case I did not so much think of EXEMPT ACCESS POLICY turning off security but the lack of it turning it on for restricted users.

That will do for part 2.

Follow

Get every new post delivered to your Inbox.

Join 176 other followers