jump to navigation

Pragma UDF – Some Current Limitations November 11, 2015

Posted by mwidlake in performance, PL/SQL, SQL, Testing.
Tags: , , , ,
6 comments

There are currently some limitations to when pragma UDF will speed up your calls to PL/SQL functions from SQL.

In my post introducing the new pragma UDF feature of Oracle 12c I explained how it can be used to reduce the impact of context switching when you call a PL/SQL function from SQL.

In my example I showed how running a SQL-only SELECT statement that formatted a name for display over 100,000 records took 0.03 seconds went up to 0.33 seconds when the formatting SQL was put in a user defined PL/SQL function. This impact on performance is a shame as it is so beneficial to encapsulate business logic in one single place with PL/SQL. Stating that the PL/SQL function is a user defined one with the pragma UDF option reduced the run time to 0.08 seconds – which is removing most of the context switching overhead. Check out the prior post for full details.

This improvement in performance is great news and is as good, and sometimes better, than using the other new capability of 12c – allowing you to state a function as part of a SQL statement using the WITH clause, if you know about that (I plan to do a further post on that). As a quick example, here is my display name function code expressed within a WITH clause:

with 
  function l_disp_name(p_sn      in varchar2
                      ,p_fn1     in varchar2
                      ,p_fn2     in varchar2 :=null  
                      ,p_title   in varchar2 :=null )
return varchar2
is
v_return     varchar2(1000);
begin
  v_return := case when p_title is null then ''
                   else initcap(p_title)||' '
              end
            ||initcap(p_fn1)||' '
            ||case when p_fn2 is null then ''
                   else substr(p_fn2,1,1)||' '
              end
            ||initcap(p_sn);
return v_return;
end l_disp_name;
select  max(l_disp_name(p_sn =>surname           ,p_fn1  =>first_forename
                       ,p_fn2=>second_forename   ,p_title=>pers_title)        ) text_output
       ,count(*)
from pers
/

The above runs in 0.10 seconds, just slightly slower than 0.08 for my pragma UDF function

However, I need to warn you of some current limitations to pragma UDF. Nearly all the examples on the web so far are

  • using very, very simple functions that take in a number and return a number
  • Use a stand-alone stored function

And they work fine. However, I had real trouble getting a performance gain when I was working with my function that took in four varchar2 inputs and returned a varchar2 value. No error was given when I marked the function with pragma UDF but there was no performance gain (or loss).

I eventually worked out some limitations to pragma UDF on my version of Oracle – 12.1.0.2.0

  1. It gives a performance boost when the inputs and return values are NUMBER, VARCHAR2, multiple VARCHAR2 IN parameters
  2. There is no performance boost when either or both the IN parameter or RETURN value is a DATE
  3. There is no performance boost if there are any default values for VARCHAR2 IN parameters
  4. If the function gains a performance benefit from pragma UDF as a standalone stored function, it appears to also gain an advantage if it is a function defined as pragma UDF within a package – so you can still keep all your functions in packages.

You might notice that my example of using the WITH clause states a function that has default values. The WITH option gains the performance advantage of that feature just fine with IN parameter defaults.

The take-home message is that, at present, pragma UDF only seems to help functions with certain types of IN or RETURN values and is nullified by default values – so if you see no performance gain for your functions, this might be why. I need to stress that my tests were not exhaustive, I have not investigated many other combinations.

I’ve discussed the issue with a couple of people within Oracle and the relevant Product Manager is looking to investigate further for me, which is jolly decent of the fellow.

My investigation is of course only by empirical testing, it does not reveal how pragma UDF works. But, as I said in my first post, it seems to aid how information is passed between the PL/SQL and SQL engines as it is variation in those that seem to nullify the benefit of pragma UDF. If you want to duplicate my tests, you can do with the below scripts. I show my test output first, with comments produced with PROMPT commands. I then give you the SQL to create the test table, the functions and package I used and the test script. Feel free to take, expand and let me know of anything different or further you may find. I say nothing of interest after the scripts, so this is in effect the end of the post:-)

The output of my test, with prompts:

running udf_tests
investigating why pragam udf helps some simple functions and not others
---------------------------------------------------------------------- --

simple number in-number out function
NUMBER_OUTPUT   COUNT(*)                                                                            
------------- ----------                                                                            
  10000000000     100000                                                                            
Elapsed: 00:00:00.12

NUMBER_OUTPUT   COUNT(*)                                                                            
------------- ----------                                                                            
  10000000000     100000                                                                            
Elapsed: 00:00:00.03
** udf helps

simple varchar in-varchar out function
TEXT_OUTPUT                      COUNT(*)                                                           
------------------------------ ----------                                                           
XYYYY                              100000                                                           
Elapsed: 00:00:00.12

TEXT_OUTPUT                      COUNT(*)                                                           
------------------------------ ----------                                                           
XYYYY                              100000                                                           
Elapsed: 00:00:00.04
** udf helps

two varchar in-varchar out function, is the issue with more than one in parameter?
TEXT_OUTPUT                      COUNT(*)                                                           
------------------------------ ----------                                                           
XYYYYYYYYY                         100000                                                           
Elapsed: 00:00:00.14

TEXT_OUTPUT                      COUNT(*)                                                           
------------------------------ ----------                                                           
XYYYYYYYYY                         100000                                                           
Elapsed: 00:00:00.04
** udf helps

simple date in-date out function
DATE_OUTPUT            COUNT(*)                                                                     
-------------------- ----------                                                                     
14-MAY-2010 13:11        100000                                                                     
Elapsed: 00:00:00.15

DATE_OUTPUT            COUNT(*)                                                                     
-------------------- ----------                                                                     
21-NOV-2004 13:11        100000                                                                     
Elapsed: 00:00:00.15
***************************************************************SIMILAR TIME!!!
udf does not help

is date out the issue,  date in-num out function
NUMBER_OUTPUT   COUNT(*)                                                                            
------------- ----------                                                                            
      2454431     100000                                                                            
Elapsed: 00:00:00.17

NUMBER_OUTPUT   COUNT(*)                                                                            
------------- ----------                                                                            
      2454231     100000                                                                            
Elapsed: 00:00:00.18
***************************************************************SIMILAR TIME!!!
udf does not help

is date in the issue,  num in-date out function
DATE_OUTPUT            COUNT(*)                                                                     
-------------------- ----------                                                                     
07-AUG-2018 18:11        100000                                                                     
Elapsed: 00:00:00.21

DATE_OUTPUT            COUNT(*)                                                                     
-------------------- ----------                                                                     
11-NOV-2015 17:57        100000                                                                     
Elapsed: 00:00:00.21
***************************************************************SIMILAR TIME!!!
udf does not help

so back to my original function I had issues with
a difference with the multiple vcs in func and my orig func is my orig had defaults
thus I will try a version with no defaults
TEXT_OUTPUT                      COUNT(*)                                                           
------------------------------ ----------                                                           
Ms Wyyyyyyyyy W Wyyyyyyyyy         100000                                                           
Elapsed: 00:00:00.19

TEXT_OUTPUT                      COUNT(*)                                                           
------------------------------ ----------                                                           
Ms Wyyyyyyyyy W Wyyyyyyyyy         100000                                                           
Elapsed: 00:00:00.08
****************************************************************UDF has an IMPACT

now with one of the parameters set to a default
TEXT_OUTPUT                      COUNT(*)                                                           
------------------------------ ----------                                                           
Ms Wyyyyyyyyy W Wyyyyyyyyy         100000                                                           
Elapsed: 00:00:00.32

TEXT_OUTPUT                      COUNT(*)                                                           
------------------------------ ----------                                                           
Ms Wyyyyyyyyy W Wyyyyyyyyy         100000                                                           
Elapsed: 00:00:00.32
****************************************************************UDF has NO IMPACT
****************************************************************ALSO defaults cause both versions to be slower

now call the simple disp_name_udf function that benefits standalone from within a package

standalone
TEXT_OUTPUT                      COUNT(*)                                                           
------------------------------ ----------                                                           
Ms Wyyyyyyyyy W Wyyyyyyyyy         100000                                                           
Elapsed: 00:00:00.08

within package
TEXT_OUTPUT                      COUNT(*)                                                           
------------------------------ ----------                                                           
Ms Wyyyyyyyyy W Wyyyyyyyyy         100000                                                           
Elapsed: 00:00:00.08
*********************************************** WORKS - so long as neither spec of body have prm defaults

and just to round of, using a subquery factored function which my prior tests showed reduced overhead
TEXT_OUTPUT                      COUNT(*)                                                           
------------------------------ ----------                                                           
Ms Wyyyyyyyyy W Wyyyyyyyyy         100000                                                           
Elapsed: 00:00:00.10

the WITH function benefits even with defaults
TEXT_OUTPUT                      COUNT(*)                                                           
------------------------------ ----------                                                           
Ms Wyyyyyyyyy W Wyyyyyyyyy         100000                                                           
Elapsed: 00:00:00.10

Creating the test table

drop table pers;
create table pers
(pers_id             number(8)    not null
,surname             varchar2(30) not null
,first_forename      varchar2(30) not null
,second_forename     varchar2(30)
,pers_title          varchar2(10)
,sex_ind             char(1)      not null
,dob                 date
,addr_id             number(8)
,pers_comment        varchar2(2000)
)
/
insert into pers
select rownum 
      ,rpad(chr(65+mod(rownum,24)),10,chr(65+mod(rownum,25))) 
      ,rpad(chr(65+mod(rownum,24)),10,chr(65+mod(rownum,25))) @cre
      ,rpad(chr(65+mod(rownum,24)),10,chr(65+mod(rownum,25))) 
      ,decode(mod(rownum,4),0,'MR',1,'MRS',2,'Ms',3,'MR','DR')
      ,decode(mod(rownum,2),0,'M',1,'F')
      ,sysdate - (3000+mod(rownum,30000))
      ,rownum +1001
      ,rpad(chr(65+mod(rownum,24)),200,chr(65+mod(rownum,25))) 
from dual
connect by level < 100001
/

Creating the functions and a small package

--num_num
CREATE OR REPLACE FUNCTION normal_num_num(p_id IN NUMBER) RETURN NUMBER IS
v_num number;
BEGIN
  v_num:=p_id*p_id;
  RETURN v_num;
END;
/
CREATE OR REPLACE FUNCTION udf_num_num(p_id IN NUMBER) RETURN NUMBER IS
PRAGMA UDF;
v_num number;
BEGIN
  v_num:=p_id*p_id;
  RETURN v_num;
END;
/
--
-- vc_vc
CREATE OR REPLACE FUNCTION normal_vc_vc(p_id IN varchar2) RETURN varchar2 IS
v_vc varchar2(100);
BEGIN
  v_vc:=substr(p_id,1,5);
  RETURN v_vc;
END;
/
CREATE OR REPLACE FUNCTION udf_vc_vc(p_id IN varchar2) RETURN varchar2 IS
PRAGMA UDF;
v_vc varchar2(100);
BEGIN
  v_vc:=substr(p_id,1,5);
  RETURN v_vc;
END;
/
-- vc_vc_2
CREATE OR REPLACE FUNCTION normal_vc_vc_2(p_id1 IN varchar2,p_id2 IN varchar2) RETURN varchar2 IS
v_vc varchar2(100);
BEGIN
  v_vc:=substr(p_id1,1,5)||substr(p_id2,2,5);
  RETURN v_vc;
END;
/
CREATE OR REPLACE FUNCTION udf_vc_vc_2(p_id1 IN varchar2,p_id2 IN varchar2) RETURN varchar2 IS
PRAGMA UDF;
v_vc varchar2(100);
BEGIN
  v_vc:=substr(p_id1,1,5)||substr(p_id2,2,5);
  RETURN v_vc;
END;
/
--
-- dt_dt
CREATE OR REPLACE FUNCTION normal_dt_dt(p_id IN date) RETURN date IS
v_dt date;
BEGIN
  v_dt:=p_id+1000;
  RETURN v_dt;
END;
/
CREATE OR REPLACE FUNCTION udf_dt_dt(p_id IN date) RETURN date IS
PRAGMA UDF;
v_dt date;
BEGIN
  v_dt:=p_id-1000;
  RETURN v_dt;
END;
/
-- dt_num
CREATE OR REPLACE FUNCTION normal_dt_num(p_id IN date) RETURN number IS
v_num number;
BEGIN
  v_num:=to_char(p_id,'J')+100;
  RETURN v_num;
END;
/
CREATE OR REPLACE FUNCTION udf_dt_num(p_id IN date) RETURN number IS
PRAGMA UDF;
v_num number;
BEGIN
  v_num:=to_char(p_id,'J')-100;
  RETURN v_num;
END;
/
-- num_dt
CREATE OR REPLACE FUNCTION normal_num_dt(p_id IN number) RETURN DATE IS
v_dt date;
BEGIN
  v_dt:=sysdate+(p_id/100);
  RETURN v_dt;
END;
/
CREATE OR REPLACE FUNCTION udf_num_dt(p_id IN number) RETURN DATE IS
PRAGMA UDF;
v_dt date;
BEGIN
  v_dt:=sysdate-(p_id/100);
  RETURN v_dt;
END;
/
create or replace function normal_disp_name (p_sn      in varchar2
                       ,p_fn1     in varchar2
                       ,p_fn2     in varchar2  
                       ,p_title   in varchar2  ) return varchar2 is
v_return     varchar2(1000);
begin
  v_return := case when p_title is null then ''
                   else initcap(p_title)||' '
              end
            ||initcap(p_fn1)||' '
            ||case when p_fn2 is null then ''
                   else substr(p_fn2,1,1)||' '
              end
            ||initcap(p_sn);
return v_return;
end;
/
create or replace function udf_disp_name (p_sn      in varchar2
                       ,p_fn1     in varchar2
                       ,p_fn2     in varchar2  
                       ,p_title   in varchar2  ) return varchar2 is
PRAGMA UDF;
v_return     varchar2(1000);
begin
  v_return := case when p_title is null then ''
                   else initcap(p_title)||' '
              end
            ||initcap(p_fn1)||' '
            ||case when p_fn2 is null then ''
                   else substr(p_fn2,1,1)||' '
              end
            ||initcap(p_sn);
return v_return;
end;
/
create or replace function normal_disp_name_defaults (p_sn      in varchar2
                       ,p_fn1     in varchar2
                       ,p_fn2     in varchar2  
                       ,p_title   in varchar2 :=null  ) return varchar2 is
v_return     varchar2(1000);
begin
  v_return := case when p_title is null then ''
                   else initcap(p_title)||' '
              end
            ||initcap(p_fn1)||' '
            ||case when p_fn2 is null then ''
                   else substr(p_fn2,1,1)||' '
              end
            ||initcap(p_sn);
return v_return;
end;
/
create or replace function udf_disp_name_defaults (p_sn      in varchar2
                       ,p_fn1     in varchar2
                       ,p_fn2     in varchar2  
                       ,p_title   in varchar2 :=null ) return varchar2 is
PRAGMA UDF;
v_return     varchar2(1000);
begin
  v_return := case when p_title is null then ''
                   else initcap(p_title)||' '
              end
            ||initcap(p_fn1)||' '
            ||case when p_fn2 is null then ''
                   else substr(p_fn2,1,1)||' '
              end
            ||initcap(p_sn);
return v_return;
end;
/
create or replace package t_pkg as
function udf_disp_name (p_sn      in varchar2
                       ,p_fn1     in varchar2
                       ,p_fn2     in varchar2  
                       ,p_title   in varchar2 ) return varchar2;
end t_pkg;
/
create or replace package body t_pkg as
function udf_disp_name (p_sn      in varchar2
                       ,p_fn1     in varchar2
                       ,p_fn2     in varchar2  
                       ,p_title   in varchar2 ) return varchar2 is
PRAGMA UDF;
v_return     varchar2(1000);
begin
  v_return := case when p_title is null then ''
                   else initcap(p_title)||' '
              end
            ||initcap(p_fn1)||' '
            ||case when p_fn2 is null then ''
                   else substr(p_fn2,1,1)||' '
              end
            ||initcap(p_sn);
return v_return;
end;
end t_pkg;
/

The test script

-- udf_tests
set lines 100 pages 50
set feed off
col text_output     form a30
col number_output   form 99999999999
col date_output     form a20
spool udf_tests.lst
prompt  running udf_tests
prompt
set pause off
set autotrace off
set timi on
prompt investigating why pragam udf helps some simple functions and not others
prompt ----------------------------------------------------------------------- 
--
prompt
--
prompt simple number in-number out function
select /* mdw_16a */       max(normal_num_num(pers_id)) number_output
       ,count(*) from pers
/
select /* mdw_16b */        max(udf_num_num(pers_id)) number_output
       ,count(*)  from pers
/
prompt
prompt ** udf helps
prompt
--
prompt simple varchar in-varchar out function
select /* mdw_16c */       max(normal_vc_vc(surname)) text_output
       ,count(*) from pers
/
select /* mdw_16d */       max(udf_vc_vc(surname)) text_output
       ,count(*)  from pers
/
prompt
prompt ** udf helps
prompt
--
--
prompt  two varchar in-varchar out function, is the issue with more than one in parameter?
select /* mdw_16e */       max(normal_vc_vc_2(surname,first_forename)) text_output
       ,count(*) from pers
/
select /* mdw_16f */       max(udf_vc_vc_2(surname,first_forename)) text_output
       ,count(*)  from pers
/
prompt
prompt ** udf helps
prompt
--
prompt simple date in-date out function
select /* mdw_16g */       max(normal_dt_dt(DOB)) date_output
       ,count(*) from pers
/
select /* mdw_16h */       max(udf_dt_dt(DOB)) date_output
       ,count(*)  from pers
/
prompt ***************************************************************SIMILAR TIME!!!
prompt udf does not help
prompt

--
prompt is date out the issue,  date in-num out function
select /* mdw_16i */       max(normal_dt_num(DOB)) number_output
       ,count(*) from pers
/
select /* mdw_16j */       max(udf_dt_num(DOB)) number_output
       ,count(*)  from pers
/
prompt ***************************************************************SIMILAR TIME!!!
prompt udf does not help

--
prompt is date in the issue,  num in-date out function
select /* mdw_16k */       max(normal_num_dt(pers_id)) date_output
       ,count(*) from pers
/
select /* mdw_16l */       max(udf_num_dt(pers_id)) date_output
       ,count(*)  from pers
/
prompt ***************************************************************SIMILAR TIME!!!
prompt udf does not help
--
--
prompt
prompt so back to my original function I had issues with
prompt a difference with the multiple vcs in func and my orig func is my orig had defaults
prompt thus I will try a version with no defaults
prompt 
select /* mdw_16m */
        max(normal_disp_name(surname,first_forename,second_forename,pers_title)) text_output
       ,count(*) from pers
/
select /* mdw_16n */
        max(udf_disp_name(surname,first_forename,second_forename,pers_title)) text_output
       ,count(*)  from pers
/
prompt ****************************************************************UDF has an IMPACT
prompt
prompt
prompt now with one of the parameters set to a default 
select /* mdw_16o */
        max(normal_disp_name_defaults(surname,first_forename,second_forename,pers_title)) text_output
       ,count(*) from pers
/
select /* mdw_16p */
        max(udf_disp_name_defaults(surname,first_forename,second_forename,pers_title)) text_output
       ,count(*)  from pers
/
prompt ****************************************************************UDF has NO IMPACT
prompt ****************************************************************ALSO defaults cause both versions to be slower
prompt 
prompt now call the simple disp_name_udf function that benefits standalone from within a package
prompt
prompt standalone
select /* mdw_16q */
        max(udf_disp_name(surname,first_forename,second_forename,pers_title)) text_output
       ,count(*)  from pers
/
prompt
prompt within package
select /* mdw_16r */
        max(t_pkg.udf_disp_name(surname,first_forename,second_forename,pers_title)) text_output
       ,count(*)  from pers
/
prompt *********************************************** WORKS - so long as neither spec of body have prm defaults
prompt
prompt and just to round of, using a subquery factored function which my prior tests showed reduced overhead
with 
  function l_disp_name(p_sn      in varchar2
                      ,p_fn1     in varchar2
                      ,p_fn2     in varchar2  
                      ,p_title   in varchar2 )
return varchar2
is
v_return     varchar2(1000);
begin
  v_return := case when p_title is null then ''
                   else initcap(p_title)||' '
              end
            ||initcap(p_fn1)||' '
            ||case when p_fn2 is null then ''
                   else substr(p_fn2,1,1)||' '
              end
            ||initcap(p_sn);
return v_return;
end l_disp_name;
select /*mdw_16s */
        max(l_disp_name(p_sn =>surname           ,p_fn1  =>first_forename
                       ,p_fn2=>second_forename   ,p_title=>pers_title)        ) text_output
       ,count(*)
from pers
/

prompt the WITH function benefits even with defaults
with 
  function l_disp_name(p_sn      in varchar2
                      ,p_fn1     in varchar2
                      ,p_fn2     in varchar2 :=null  
                      ,p_title   in varchar2 :=null )
return varchar2
is
v_return     varchar2(1000);
begin
  v_return := case when p_title is null then ''
                   else initcap(p_title)||' '
              end
            ||initcap(p_fn1)||' '
            ||case when p_fn2 is null then ''
                   else substr(p_fn2,1,1)||' '
              end
            ||initcap(p_sn);
return v_return;
end l_disp_name;
select /*mdw_16t */
        max(l_disp_name(p_sn =>surname           ,p_fn1  =>first_forename
                       ,p_fn2=>second_forename   ,p_title=>pers_title)        ) text_output
       ,count(*)
from pers
/
--
spool off

Friday Philosophy – We Could Be Heroes! {just for one day}. November 6, 2015

Posted by mwidlake in Friday Philosophy, humour, Perceptions.
Tags: , , ,
3 comments

At Open World I overheard a snippet of conversation which went something like this:

Bob – “How’s it going? Did the last talk go down well?”
Bill – “Sure, it was on time, the audience seemed to like it.”
Bob – “Will you be here next year?”
Bill – “Errmm….” pause…*sigh*… “I don’t know…. I’ll see how I feel.”
Bob – “Oh? In what way?”
Bill – “It just that, at Open World… I have to fend off two dozen people just to go have a pee!”

Wild horses could not drag out of me the name of the person who said that (though several pints and the offer of a curry might do the trick – try me). It both made me smile and made me think. There are down-sides to becoming highly respected in your sphere.

There are definitely different levels of renown and respect in the relatively small world of the Oracle Database Technologist. I’m not doing bad in that respect; I’d put myself in the third of the seven circles, maybe tapping on gates of circle two. Occasionally I think it would be nice to be either technically or entertainingly good enough to join the Big Names in the innermost circle – but I really don’t think I can face the Hem-Touching!

What do I mean about “Hem-Touching”? It’s something a few friends and I came up with at the UKOUG Tech conference about 5 or 6 years ago to describe people who will approach one of the Oracle Names with a mixture of awe and fear in their eyes and just want them to acknowledge their presence,be allowed to speak, maybe to touch the hem of their cloak. If you go up to the balcony that is above the exhibition hall at the Birmingham ICC, you can sometimes watch an Oracle Name walk through the exhibition and see some people suddenly swerve and hurry towards them – especially if the Name currently has no one with them. I’ve even seen someone suddenly stop when another acolyte gets to their hero first. I don’t know why, these people will speak to more than one person at a time. And the thing is, people in the UK and Europe are generally more reserved than our cousins in other continents, so we are less forward in, well, being forward.

Am I being mean to these people? Well, a little I guess, but it’s mainly because of the little story I started with. I’m friends with some of the Names and I know a lot of them are uncomfortable with Hero Worship. Being respected and held in high regard is great, most of them are very happy about that, as they have worked damned hard and long to be knowledgeable enough to hold that position. But when people treat them like a living saint or the bestower of blessings, it’s just a bit weird. This is just an area of technology after all, not the eradication of Malaria. They are “just” people – OK, they are people who are usually very smart and very capable, but they are also people who are happy to share and teach – otherwise they would not be at the conferences sharing and teaching. Most of them are idiots in other areas of their lives too, we all are.

I’ve never felt the need to hero-worship myself. Not because I do not deeply respect people who achieve great things, it’s just not in my psychology I think. I did not put up any posters in my bedroom of the people I most respected when I was a teenager. I used to know a Nobel Prize Winner (though I doubt he’d recognise me in the street now) but when I met him the first time I had no idea who he was and just treated him like a person – and we got on fine. He treated me like a person too. I’ve been lucky enough to meet some very smart academics, many of the Oracle Names and even the odd traditionally famous person. It’s amazing how like people they are – if you treat them like people.

I’m certainly not above being pleased when someone I respect mentions me or refers to something I have done though. I’ll grin like an idiot on the rare occasions someone has name-checked me in a presentation or they tell me they liked something I said. I’m tickled pink when a Name follows me on twitter. But I feel hero worship is not what they want. Respect yes, being told you appreciate what they have taught you fine. Going shiny-eyed and asking to touch the hem of their coat, weird; don’t do it.

Oracle Names are people, treat them as such. They’ll probably appreciate you more if you do.

And if you ever find yourself in a group of several others, all trying to say “hello” to some gal or guy you just saw presenting, and they are looking a little uncomfortable and shifting from foot to foot and looking towards a door over there – let the poor sod go to the loo will you?

Pragma UDF – Speeding Up your PL/SQL Functions Called From SQL November 4, 2015

Posted by mwidlake in performance, PL/SQL, SQL.
Tags: , , ,
6 comments

A new feature for PL/SQL was introduced in V12, pragma UDF. UDF stands for User Defined Functions. It can speed up any SQL you have that uses PL/SQL functions you created yourself.

{please see this second post on some limitations of pragma UDF in respect of IN & RETURN data types and parameter defaults}.

We can create our own functions in PL/SQL and they can be called from both PL/SQL and SQL. This has been possible since V7.3 and is used extensively by some sites to extend the capabilities of the database and encapsulate business logic.

A problem with this, though, is that every time you swap from SQL to PL/SQL (or the other way around) you have to do a context switch each time, which can be quite cpu and memory intensive. If you are using your own PL/SQL function in the SELECT list of a SQL statement and you are selecting a lot of rows (say as part of a business report) then the overhead can be quite considerable as you could be doing a context switch per row. I won’t go into too much detail here (partly as I go in to considerable detail on the subject in a book I am working on for 2016) on how you can investigate the context switching and when exactly it occurs, but I will show you one of the two new ways in Oracle 12 to reduce the overhead, namely PRAGMA UDF. At present this seems to be a little used and rarely-mentioned feature on the blogsphere, with articles just covering simple examples of almost no-business-function, numeric functions.

I’ll give you a slightly less simple example but my next post will give you details on some limitations of pragma UDF. Here I am just setting the scene. I have the below PERSON table which has the parts of the names in distinct columns, with the contents forced to upper case (as is standard practice). We will create a function to provide a nicely init-capped and spaced display name and a second function which is identical but uses PRAGMA UDF.

PERSON
Name                                     Null?    Type
---------------------------------------- -------- ---------------
PERS_ID                                  NOT NULL NUMBER(8)
SURNAME                                  NOT NULL VARCHAR2(30)
FIRST_FORENAME                           NOT NULL VARCHAR2(30)
SECOND_FORENAME                                   VARCHAR2(30)
PERS_TITLE                                        VARCHAR2(10)
SEX_IND                                  NOT NULL CHAR(1)
DOB                                               DATE
ADDR_ID                                           NUMBER(8)
STAFF_IND                                         CHAR(1)
LAST_CONTACT_ID                                   NUMBER(8)
PERS_COMMENT                                      VARCHAR2(2000)


create or replace function normal_disp_name (p_sn      in varchar2
                                            ,p_fn1     in varchar2
                                            ,p_fn2     in varchar2  
                                            ,p_title   in varchar2  ) return varchar2 is
v_return     varchar2(1000);
begin
  v_return := case when p_title is null then ''
                   else initcap(p_title)||' '
              end
            ||initcap(p_fn1)||' '
            ||case when p_fn2 is null then ''
                   else substr(p_fn2,1,1)||' '
              end
            ||initcap(p_sn);
return v_return;
end;
/
create or replace function udf_disp_name (p_sn      in varchar2
                                         ,p_fn1     in varchar2
                                         ,p_fn2     in varchar2  
                                         ,p_title   in varchar2  ) return varchar2 is
-- The Below is the KEY bit
PRAGMA UDF;
v_return     varchar2(1000);
-- {Identical from here}

-- select some data with one of the functions, it does not matter which
select pers_title title,    first_forename    ,second_forename    , surname
      ,normal_disp_name(p_sn =>surname           ,p_fn1  =>first_forename
                       ,p_fn2=>second_forename   ,p_title=>pers_title) display_name
from person
...


TITLE   first_fn   secon_fn   SURNAME         DISPLAY_NAME
------- ---------- ---------- --------------- ----------------------------
MR      HARRISON   RICHARD    HARRIS          Mr Harrison R Harris
MRS     ANNEKA     RACHAEL    HARRIS          Mrs Anneka R Harris
MRS     NICKIE     ALISON     ELWIG           Mrs Nickie A Elwig
MASTER  JAMES      DENZIL     ELWIG           Master James D Elwig
MR      JEFF                  GARCIA          Mr Jeff Garcia
...
MRS     AMELIA     MARIA      ORPINGTON-SMYTH Mrs Amelia M Orpington-Smyth

So we have our test table, you can see my normal_disp_name function and that the *only* difference with the second version is the inclusion of PRAGMA_UDF in the declaration section. That is partly why it is such a nice feature, you can just add this one line to existing code and you should get the benefit. Should….. {see second post when I do it}

Finally, I show some code using the function and the output.

To demonstrate the impact of context switching I will select 100,000 records from my test table in 3 ways: using only native SQL functions and thus no context switching; using my traditional PL/SQL function which suffers from context switching; with my new “pragma UDF” function to reduce the overhead of the context switching.

select avg(length(
      case when pers_title is null then ''
                   else initcap(pers_title)||' '
              end
            ||initcap(first_forename)||' '
            ||case when second_forename is null then ''
                   else substr(second_forename,1,1)||' '
              end
            ||initcap(surname)
          )      )  avg_name_length
       ,count(*)
from person
where pers_id > 100100
and rownum < 100000

select  avg(length(normal_disp_name(p_sn =>surname           ,p_fn1  =>first_forename
                                   ,p_fn2=>second_forename   ,p_title=>pers_title)        ) ) disp_name_len
       ,avg(addr_id)
       ,count(*)
from person
where pers_id > 100100 and rownum < 100000

select  avg(length(udf_disp_name(p_sn =>surname           ,p_fn1  =>first_forename
                                ,p_fn2=>second_forename   ,p_title=>pers_title)        ) ) disp_name_len
       ,avg(addr_id)
       ,count(*)
from person
where pers_id > 100100 and rownum < 100000

One thing to mention in passing is that the code using either function is much easier to read and self-documenting. This is one of the benefits of proceduralising your code, as well as creating just one place to maintain it. If I had to re-use that native-SQL section in a half-dozen reports I would probably mess up at least one of the times I cut-and-paste it and I would now have several places to maintain that code.

I ran the test statements several times and took the average of the 2nd to 6th runs, to remove the initial parsing & caching overhead that comes with the first execution and to get more reliable figures than one further run would give me.

Version                      Run Time average (secs)
Native SQL                   0.03
Traditional PL/SQL           0.33
PRAGMA UDF PL/SQL            0.08

As you can see, just including PRAGMA UDF removed most of the overhead caused by context switching.

How does PRAGMA UDF work? I’m not sure, the official Oracle documentation is pretty light on it and just says:

“The UDF pragma tells the compiler that the PL/SQL unit is a user defined function that is used primarily in SQL statements, which might improve its performance

Note the italics (which are mine). “Might improve its performance” but no detail as to what it does. As I understand it, it alters the internal representation of data as it is passed between the SQL and PL/SQL engines via the IN and RETURN values (Note it does not change the data types!) – but treat that as a bit of wild speculation for now. I have some evidence for it that you will see in, yes, the next post.

Pragma UDF can slow down slightly functions being called directly from PL/SQL. So use it only for functions you know will be called from SQL.

I’ll make one other observation. Using PL/SQL functions increased the run time to process 100,000 records on my modest test system by all of 0.3 seconds. But that is 10 times the time taken for the native SQL statement. Pragma UDF removes around 80% of this overhead. It’s a nice saving but is probably inconsequential if your code is actually doing any physical IO at all (my example is processing already cached blocks). And if you are only processing a few records or one record in a GUI screen, the context switching is moot {meaning, of no significance}.

But if you have code that processes a huge set of data and uses a lot of user defined PL/SQL functions (and again I go into a lot more detail about this in the book) using pragma UDF in 12C could gain you quite a bit of extra performance. If you have code where even 0.00001 seconds is important (think trading systems) then again there may be a worthwhile benefit.

OOW Report – No List of Talks, No Cloud, Just Thoughts on Community October 30, 2015

Posted by mwidlake in conference, Perceptions, User Groups.
Tags: , ,
4 comments

As I type I am in my hotel, sipping a final beer (it was a gift that has been to a few talks with me in my backpack) and looking back at Oracle Open World 2015. I must confess I am a little drunk so we will see if this post lasts…
{Update – it passed the next-morning-sobriety test. I was only a little drunk}

OOW15 beers

I am on record as saying I don’t like Open World. I came to previous events in 2003 and 2004 I think (yes, over a decade back), both times at short notice on the behest of the Mother Corporation. And at those times I only knew people at the event from Corporation Oracle – not people in the Oracle User Community. It is miserable being 1 of xx thousand people who you *should* share interests with but simply don’t know. Oracle employees are generally excluded from the event so that removed nearly all of my contacts. It is such a large event that if you meet someone on Sunday and chatted to them – you may well never see them again! After all, it is 1 in x thousand people even for your specific area of interest. I’m not good at chatting to people “cold” and the whole “entering the US” is such a bloody awful experience (Immigration just shout at you and growl and are, frankly, as welcoming as a Rottweiler at a kitten party) that the total experience from beginning to end was just, well, less pleasant than a bad week in the office.

This time was very, very different (though not the growling Rottweiler bit, sadly). Because I am now an active member of a couple of oracle “clubs” (Oracle ACE and OakTable) I knew more people. Because I blog and tweet I knew a lot of USA {and other} people, if only via social media. As a result of going to a good few different user groups (and often presenting) I have become friends with people from several communities. And I have also got better at “Cold chatting”. So for several days I have been meeting people like Danny Bryant (still my hero as he got my conference pass back to me after I dropped it on a bus!), Bobby Curtis, That Jeff Smith, Sarah CraynonZumbrum, Zahid Anwar… and about 37 other people I had never met or only met once. I have re-connected with a couple of dozen old friends too and hung around with closer friends from the UK & Europe. And it has been great. This is one of the great aspects of being an active member of the Oracle Community, there is a pool of people I can now talk to and relax with.

I’ve loved my OOW15 experience and that is fundamentally because I felt I was inside rather than outside. At this point I was planning to say that not everyone you meet in the flesh will turn out to be people you actually get on with – but I can honestly say that everyone I have met this week has been at least polite to me, most have been welcoming. I’m not saying all will be life-long friends and I am at long last wise enough to recognise that someone being polite to me does not mean they did not find me annoying. But one of the great things about a user group community is that almost everyone in it is actually on the “friendly” side of normal. If you are not, user groups are not going to be your thing!

It makes a huge difference. Being able to find someone (and modern social media makes that so much easier than a decade back when I hated this experience) to have a coffee with and a nice conversation can make a potentially lonely gap between presentations into an enjoyable afternoon. I missed half a dozen presentations this week as the conversations went on much longer and were more illuminating than you planned. I could just position myself at a central location and pretty soon a friend would wander by. Or, at least, someone who would not run away:-)

Being mindful of the above, if anyone came up to me to talk, I talked to them. There is a phrase that seems current in the US of “paying it forward” which means if you have had a nice experience, try to make someone else’s experience nice. Or is it “paying it backward”? I don’t know for sure but I like both. If you have been helped, help someone. If you think people should help each other, start it by being helpful first. I was able to do this a little bit myself by making sure I was around if a friend called Stew needed some company, as he is not as tied into the user community as others as he is new to this. However, I don’t think this will last as he is making such a name for himself that next year he’ll be introducing me to people! In turn, another friend, Brendan, made time to make sure I had company as he knew I’d not liked my prior experiences.

So all in all I now don’t dislike OOW. I like OOW. And the reason is the user community is there for me. It’s there for everyone who wishes to be a part of it. You won’t like everyone, everyone won’t like you – but that is fine, we all have our different characters – but you will gel with a good few people.

Note I have not mentioned presentation slots. Some were good, some were bad, a small number were great and a similar number were awful. But I did learn a lot and I appreciate the fact. I will say no more as, frankly, if you were not at the conference then a discussion of the presentations is pretty pointless!

I just want to end on a final consideration. I know I am now a member of a couple of “clubs” and that helps me in knowing people. But a lot of people I now know are not members of either of those clubs and I know them due to my simply being social-media active, a user group attender and I make myself cold-chat more. It almost hurts me to say it, but social media can be a good thing. Nothing beats face-to-face socialising, but knowing people virtually first is a great help in getting started with meeting them for real.

I really love the user group community. Or is that just the beer talking (which I finished over an hour back!)

{update – OK, it was the beer, I don’t love any of you. But I like you a lot…}
 

The “as a Service” paradigm. October 27, 2015

Posted by mwidlake in Architecture, Hardware, humour.
Tags: , , ,
4 comments

For the last few days I have been at Oracle Open World 2015 (OOW15) learning about the future plans and directions for Oracle. I’ve come to a striking realisation, which I will reveal at the end.

The message being pressed forward very hard is that of compute services being provided “As A Service”. This now takes three flavours:

  1. Being provided by a 3rd party’s hardware via the internet, ie in The Cloud.
  2. Having your own hardware controlled and maintained by you but providing services with the same tools and quick-provisioning ideology as “cloud”. This is being called On Premise (or just “On Prem” if you are aiming to annoy the audience), irrespective of the probably inaccuracy of that label (think hosting & dedicated compute away from head office)
  3. A mix of the two where you have some of your system in-house and some of it floating in the Cloud. This is called Hybrid Cloud.

There are many types of  “as a Service offerings, the main ones probably being

  • SaaS -Software as a Service
  • PaaS – Platform as a Service
  • DBaas – Database as a a Service
  • Iass – Infrastructure as a Service.

Whilst there is no denying that there is a shift of some computer systems being provided by any of these, or one of the other {X}aaS offerings, it seems to me that what we are really moving towards is providing the hardware, software, network and monitoring required for an IT system. It is the whole architecture that has to be considered and provided and we can think of it as Architecture as a Service or AaaS. This quick provisioning of the architecture is a main win with Cloud, be it externally provided or your own internal systems.

We all know that whilst the provision time is important, it is really the management of the infrastructure that is vital to keeping a service running, avoiding outages and allowing for upgrades. We need a Managed Infrastructure (what I term MI) to ensure the service provided is as good as or better than what we currently have. I see this as a much more important aspect of Cloud.

Finally, it seems to me that the aspects that need to be considered are more than initially spring to mind. Technically the solutions are potentially complex, especially with hybrid cloud, but also there are complications of a legal, security, regulatory and contractual aspect. If I have learnt anything over the last 2+ decades in IT it is that complexity of the system is a real threat. We need to keep things simple where possible – the old adage of Keep It Simple, Stupid is extremely relevant.

I think we can sum up the whole situation by combining these three elements of architecture, managed infrastructure and simplicity into one encompassing concept, which is:

KISS MI AaaS.

.

.

And yes, that was a very long blog post for a pretty weak joke. 5 days of technical presentations and non-technical socialising does strange things to your brain

Friday Philosophy – The Small Issue of Planes, Trains and…Coaches. October 21, 2015

Posted by mwidlake in Uncategorized.
7 comments

Today I get on a plane. It is a long flight, 10+ hours, and throughout all of it, some people will hate me. I even expect some Hard Stares. Why? Because I’ll be sitting in a seat with a large space in front of it – and my legs dangling off the front of the seat. Those over 6 foot will be fuming I have that extra leg room. See me sitting there. See me smiling:-) .

King of all I see

King of all I see

Well, I picked that seat for reasons that the tall (and average) do not appreciate. On flights I generally have leg and arm room others may wish they had. It is one of the few, very few, benefits of being small. Relatively speaking I have more space in my allocated seat to place my body parts in comfort. But, unlike most of you, all I see for the whole flight is a wall of seat-back. And I can’t sit straight as my thighs are slightly shorter than the seat base, so I sink even lower with an unsupported back, staring at that seat back. Just that seat back. Nothing but that seat back. Unlike you I can only see ceiling if I tip my head back, I can’t take a long look down the aeroplane above the seats; the angle is too steep. And now the seat back has moved closer to me as the person in front has reclined their seat. On short flights this is a pain, on long flights it might mean the film I was watching, on a screen slightly above my eyeline, is now well above my eyeline and the colours have gone weird. The bottom line is, I spend the whole flight in a box that ends THERE, 14.2 inches in front of my face. The whole flight, in a seat too long for my legs, going slowly batshit due to mild agoraphobia that becomes major after 5 hours. I can sit cross-legged in my seat to solve many of the issues but (a) the flight attendants don’t like it (b) I lose blood supply to my left foot and (c) I start thinking I’m a Ninja. An evil Ninja.

There is another reason I pick such a seat. Looking out the window from time to time helps keep me sane from the Box I am in, so I like to be by the window. But those people in funny costumes, way too much makeup and with the fixed smiles keep offering me these big drinks (big to me) and I soon need a pee. A pint in a half-pint can only lead to one event. Asking 2 or 3 people to let me out every hour soon gets trying for all, so a seat where I can just step front and go find the loo is good.

At least on a plane the person next to me is likely to be normal (or my wife). On normal public transport, they often are not. This next consideration has been an aspect of my life for as long as I have traveled on my own. When I was a student almost no students had their own cars, including me. Which means I often had to get to and from my parental home and college home by public transport. In my case coach (bus, large vehicle driving down roads). I would get on, sit down, watch others file in and fill the classic “I want a double seat to myself so I will spread out my shit” pattern. Until each double seat was full with coats, bags and handbags strategically placed to warn others to sod off. And I knew what was coming. It always did. Normal sized people would generally get on and choose to sit somewhere near the door, forcing someone on their own in a double seat to move their defensive stuff. I always sat towards the back. But then some massive, often fat but sometimes just hulking, person would appear (usually a man) and would look gloomily at all the single spots left. And spot “The Small Guy” way back down there. They would be over to me in a shot – lord knows how quick given how much blubber they had to drag along with them – and into the seat next to me. And then they would Sppprreeeaaaadddd. First the thigh would come over my side, followed by the rest of the leg. And the arm would push up against my am and then over the arm rest (if there was one) and shove me to one side. Soon a torso would be shoving into me. Within minutes they would by laying claim to 25% of my seat, my space. One of my few benefits of being small was being taken from me.

I was young, I was brash and I’d learnt over the years to take none of this crap. I clearly remember one trip, I think in my final year when I was tired and annoyed, when one massive chap sat next to me and started to spread and I just shouted “Oi! Get out of my seat! Get your fat arse and your fat arm out of my space! I have few benefits in my life from being small and my space on public transport is a rare one of them! Keep to your bloody side!”. I did not hold back at all on mentioning his massive blubbery state or his encroachment into my space. Oh, he was full of “Oh I did not realise, I’m not taking your space, how could you insult me for being larger” that I knew from experience was really “I chose the small guy to sit next to so I could have more space.” I’d watched him scan the seats, spot me and come over. I let rip and said “well move and sit next to someone normal size! Go on! MOVE!” He didn’t. He knew he needed some of my space. It was not a comfortable journey for either of us from then on and I suspect he did not need to get off at Sheffield, but for f***s sake, I get few benefits in life from being small. At least we were the entertainment for the other passangers for a while.

It is an aspect that has not gone away. For many years I’ve commuted into London and watched the pathetic games played by other commuters. Get on, put your stuff in the seat next to you (exactly how hard would it have been to put that coat and that little bag in the rack above your head? About the same as to spread it evenly over the seat by you) and look busy or angry. If you are lucky you can sit opposite someone by the window who has already played the double-seat-claim-game and can sit in the isle seat and put your crap on the inner seat – any new player will see it is easier to make someone move stuff from the isle seat than move stuff from the inner seat AND then climb past them to the spare window seat. Utterly selfish evil people will get on an empty coach, sit in the isle seat and then fill the window seat with their stuff that could so easily go in the overhead racks. They know how the game works and they have no sense of shame in being so selfish.

When I get on I often look at the pattern of seating in front of me and pick one of the buggers in the isle seat to move. Almost no one else does.

When I get on a train and sit down, I usually put my stuff under the seat or in the overhead racks. And given my height, if I can do it all you buggers can. And I sit. I try to sit forward-facing, I hate facing back as it make me feel a little sick, but other than that I have no rules. If I am in the isle, I will stand as soon as you ask to let you in. This is my little play to show I am nicer than all you other commuting buggers. Anyway, this train will get packed, I might as well get someone to the side of me so I can relax. Others get on and I often get someone next to me pretty soon as I am not playing silly buggers. It’s fine, soon all seats will be taken unless someone is being especially obnoxious about double seat protection. But I have to say, if someone massive (and usually a man) gets on and I see them scan the carriage and eyes fall in relief on me… I pat the seat next to me and smile. Sometimes I wave. It’s the only defense I know that works 95% of the time.

As a social commuter I hate the games the antisocial ones play, but as a small person, I bloody hate my space being bloody stolen by fat/large people. You could lose weight you know, I can’t grow! I paid for my seat, you paid for yours, for once in my short life, I bloody well want the benefit of my short stature. Now bugger off over to your side of the double seat.

Where do my trace files go? V$DIAG_INFO October 19, 2015

Posted by mwidlake in development, performance, SQL Developer.
Tags: , ,
1 comment so far

Where do oracle trace files go? I don’t know why this piece of info will not stick in my head, I seem to have to look it up 3 or 4 times a year.

If only I had an easy way to find out. There is a very easy way to find out – and that piece of info won’t stay in my head either. So this really is a blog post just for stupid, forgetful me.

V$DIAG_INFO has been available since oracle V11. All the trace files go into the Automatic Diagnostic Repository (ADR) by default.

ora122> desc v$diag_info
 Name                                                                Null?    Type
 ------------------------------------------------------------------- -------- ---------------
 INST_ID                                                                      NUMBER
 NAME                                                                         VARCHAR2(64)
 VALUE                                                                        VARCHAR2(512)
 CON_ID                                                                       NUMBER

Quick sql*plus script to get it out:

-- diag_info
-- quick check of the new v$diag_info view that came in with 11
col inst_id form 9999 head inst
col name form a25
col value form a60 wrap
spool diag_info.lst
set lines 120
select * from v$diag_info
order by name
/
spool off

Contents:

 INST_ID NAME                 VALUE                                                            CON_ID
-------- -------------------- ---------------------------------------------------------------- -------
       1 Diag Enabled          TRUE                                                                  0
       1 ADR Base              D:\APP\ORACLE                                                         0
       1 ADR Home              D:\APP\ORACLE\diag\rdbms\ora122\ora122                                0
       1 Diag Trace            D:\APP\ORACLE\diag\rdbms\ora122\ora122\trace                          0
       1 Diag Alert            D:\APP\ORACLE\diag\rdbms\ora122\ora122\alert                          0
       1 Diag Incident         D:\APP\ORACLE\diag\rdbms\ora122\ora122\incident                       0
       1 Diag Cdump            D:\app\oracle\diag\rdbms\ora122\ora122\cdump                          0
       1 Health Monitor        D:\APP\ORACLE\diag\rdbms\ora122\ora122\hm                             0
       1 Default Trace File    D:\APP\ORACLE\diag\rdbms\ora122\ora122\trace\ora122_ora_7416.trc      0
       1 Active Problem Count  0                                                                     0
       1 Active Incident Count 0                                                                     0

I should add some notes later about setting the trace file identifier…
Ohhh, OK, I’ll do it now. To make it easier to identify your trace file, set tracefile_identifier

alter session set tracefile_identifier = 'mdw151019'

--Now if I create a quick trace file
alter session set sql_trace=true

@test_code

alter session set sql_trace=false

I now go to the Diag trace directory I identified via V$DIAG_INFO and look for my trace files. I could just look for the latest ones or do a wilcard search on my tracefile_identifier string and, pop, there we are:

19/10/2015 13:59 39,751 ora122_ora_7416_mdw151019.trc
19/10/2015 13:59 426 ora122_ora_7416_mdw151019.trm

If you want a taste of the numerous ways of initiating a 10046 trace, sometimes called a SQL trace, see Tim Hall’s excellent post on his Oracle Base website:

https://oracle-base.com/articles/misc/sql-trace-10046-trcsess-and-tkprof

Oh, one final nice thing. You can open trace files in SQL Developer and play with what information is shown. Maybe I should do a whole piece on that…

Actually, these two post from Oracelnerd and Orastory will get you going, it’s pretty simple to use in any case:

http://www.oraclenerd.com/2010/02/soug-sql-developer-with-syme-kutz.html
https://orastory.wordpress.com/2015/02/27/sql-developer-viewing-trace-files/

ScreenHunter_45 Oct. 19 14.25

Friday Philosophy – 3 months, 3 conferences October 16, 2015

Posted by mwidlake in ACED, conference, Friday Philosophy, Presenting, Tech15.
Tags: , ,
2 comments

Flights are booked, hotels reserved, plans made. Don’t ask about talks prepared, just don’t:-)

This is not the usual list of “I’m going to this talk and I’m seeing that speaker” blog that people write before an event – well it is a little – it’s more about the different flavors of conference we have available to us.

I have an Oracle conference a month until the end of the year and I’m really looking forward to all of them. Each is very different. I know I am lucky to be able to do this sort of thing, that is go to so many conferences, and partly it is because of being an ACED. But fundamentally it’s come about as a result of the decision I made back in 2003 to give something back to the community that I’d learnt so much from, and even more so when a couple of years back my wife gave me permission to do less stuff that pays and more stuff that I enjoy. Oracle Community stuff.

First up of course is Oracle Open World 15. This includes a couple of days before hand with the ACED briefings. We get a heads-up on what is happening with the direction of Oracle Tech and Oracle expect us to feed back what we think. After 25 years in the business and dozens of conferences, this will be a first for me so I will be a newbie again (hmm, maybe not so new thinking about it, I’ve been on Customer Advisory Boards and Beta tested in the past so it will be interesting to see the difference). I’ve said in the past how I was not so fond of my prior Oracle Open World experiences. Too big and too razzmatazz for my repressed British personality. But the huge difference between this time and 10 years ago is not my being ACED, it is being a member of the community and looking forward to seeing so many people, catching up and talking about all things tech.
Elton John is apparently doing the appreciation event. I’m hoping for “Yellow Brick Road” era stuff and none of that modern post Y2K stuff…
Oh, and don’t forget, there is also the Oaktable presence at OOW, OakTable World. It’s free to all at OOW15 and if you want technical meat on your presentation bones, that is where you will find it.

In November, Friday 20th to Sunday 22nd, it is a totally different experience, the Bulgarian Oracle User Group Autumn conference. This is purely a tech conference, no dancing girls, no laser-show keynotes and not a hint of Elton John. Just a shed load of top presenters (so many ACE badges next to names) with a good showing of local talent too. Several of the speakers are coming to it from DOAG, a conference I was seriously considering putting papers forward for but decided not to, as I felt I was too busy at the end of the year – and then I got sweet-talked into putting forward abstracts for Bulgaria. Next year I’ll try for DOAG. This will be my first time at a BGOUG conference but I know from my friends that it is like many of the smaller European conferences. It has a more inclusive, friendly feel as you see the same people over and over again for the couple of days and spend time getting to know people pretty well and often having longer, more involved discussions about whatever tech you are working with. I’ve been really well looked after by the organisers already, helping me sort things out and advising me on what to do outside the event.

I’m combining this one with a short holiday with my wife. (She speaks Bulgarian so she will be very helpful in ordering beer in local bars). One down side to going to more conferences is that, as she travels a lot herself for work, some months we don’t see a lot of each other. It will be really nice to wander around Sofia together for a few days. The ironic thing is that her employer, actually her department, is doing some work out there that week – and they did not schedule in the only person in the team who speaks the language!

Finally there is “my” conference. Mine as in I feel it is my home conference, being in the UK and one I have presented at or helped organise for 12 years now. The UKOUG Tech15 conference. This is from Monday 7th December to Wednesday the 9th, and if you get registered in time you can also be at Super Sunday on the 6th (half a day focused on deeper tech talks). Again, a conference that puts technical content at the top and the sales sides comes along for the ride. It is a very large conference, vying with DOAG to be the biggest after Oracle Open World. We are less show and more tell than OOW but it lacks the personal feel of smaller conferences. We are back in Birmingham for this one and I have to say it’s all looking set for a great event. Registrations are significantly up on the last couple of years at this stage, the exhibition is selling well and we have great content lined up. I need to tweet more about Tech15, both about how such an event is organised (I know some of you liked hearing about that) but also about some of the things that will be happening. I’m quietly excited about a couple of things. The only problem is that, by the time I get to the actual Tech15 conference I am usually a bit spaced out and knackered from all the prep work and by the end of Wednesday (the last day) I’m physically drained – but with a head full of new information.

As I said, all three conferences have a different vibe and which one you prefer is down to what you want from your conference.

After all that I’ll be done with conferences. I refuse to go to any more until the following year…

Which reminds me, I better start putting in some abstracts and seeing if I’ve got stuff people want in their conferences next Spring.

Friday Philosophy – Be Moral or Be Sacked? October 9, 2015

Posted by mwidlake in ethics, Friday Philosophy.
Tags: , ,
8 comments

How far will you bend your moral stance to keep your job?

This post was prompted by a Twitter discussion over the recent VW Emissions scandal development where software engineers are being blamed. Let’s just skip over the rather trite and utterly unbelievable proposition that a couple of rogue software engineers did this “for reasons unknown” – and the fuel engineers, mechanical engineers, and direct managers did not realise “hey, our engines are more efficient than we knew was possible, never mind seen”. Plus the testers, change control, release managers, etc were all circumvented by the rogue software engineers…. It would have to be incompetence of unbelievable levels for the whole stack of management up to the top did not in some way at least know about this – and I personally am sure they condoned or even demanded the results.

What made me think was a comment by a friend that the software engineers must have at least colluded and thus are at least partially responsible – and it struck a chord in me. What constitutes collusion? and would you or I do it? I’ve been in a very similar situation…

Back in my first job I worked for one of the regions of the UK National Health Service, as a programmer. An edict came down from high. Government high. We were to make the waiting list figures look better. “We” being the NHS management initially but, as I guess they were powerless to really do much about the reality of the situation, it come down the levels until it was realised it was the data used to show how the waiting times were doing that could so easily be changed.

I was given the job of altering the Waiting List Reports in a few ways. A key one was how the date you started waiting was measured. No matter how often the hospital cancelled your appointment or sent you home not having done the procedure, the date from which you started waiting remained the same. However, if you were offered an appointment and for any reason you could not attend – ANY reason, be you ill in another way, have a responsibility you could not avoid, were only given a day’s notice – the date you were waiting was reset to the day of the refused appointment. Of course this was utterly unjust and we were told it would not really mean Mrs Smith who had been waiting 3 months would now have to wait another 3 months – “it would be handled”. But it made the figures so much better.

I refused. In the first place it was a con, in the second I doubted all the Mrs Smiths would be handled as the NHS, even back then, was in a right state.

To this day I am proud I refused.

My colleague was given the task instead – and she did it. I asked her how she could do it? We had some shared political and philosophical views. How could she do something she knew was utterly false and misleading? Her answer was simple.

“You’re lucky – you can afford to take the risk. I’ve just got married, we have a mortgage and I have …other responsibilities – I can’t afford to damage my career or get sacked. You can.”

She was right. I did not know it then but she was trying for a baby, so yeah, getting sacked would have been devastating. On the other hand, I had no dependents (no one loved me), no mortgage and I was already muttering about leaving. She had in effect been bullied into doing a task she was morally against. And she knew, if she did not do it someone else would and she would have taken the hit.

And I confess, I did not simply stand up, shout defiance and proudly walk out the room, head held high. I had a long chat with my union rep about what support I could expect if things got bad before I refused. I knew he was ready to support me.

There were repercussions. I already had a poor relationship with my manager. After I refused to do that work I had an even worse relationship with him, and now his boss disliked me quite a lot too. It was a large part of me leaving to join some no-hope database company.

So, I think there is a very large difference in colluding and being coerced.

The same argument goes up the stack too. I can imagine there were lots of people involved in the VW scandal who knew what was going on, did not like it but, “hey, it’s my job I am risking and it’s not as if I’m the one *authorising* this”.

I can’t say I’ve always held to my moral ground so strongly, I’ve done a couple of things professionally I wish now I’d also said no to. But I’ve also said no to a couple more.

{I hope the statute of limitations on mentioning governmental evils is less that 25 years…}

PL/SQL bug with DBMS_RANDOM? October 8, 2015

Posted by mwidlake in bug, PL/SQL, SQL.
Tags: , , ,
7 comments

I think I’ve found an (admittedly obscure) bug with DBMS_RANDOM, group functions, PL/SQL and/or SQL.

Have a look and see if you also think this is odd – or have I missed the totally obvious?

(This is all on 12.1.0.2)

{Update – my conclusion is, and thanks to Joel and Sayan for their comments, that this is not a “bug”. Oracle do not promise us how PL/SQL functions are executed due to the way SQL can be re-written by the parser. I think most of us stumbling over something like this would treat it as a bug though. You have to look at the column projection, again see the comments, to see how Oracle is deciding to get the columns derived by a naked call to DBMS_RANDOM.VALUE (by naked I mean no inclusion of parameters passed in and, significantly, no reference to columns). It’s just the way it is}

Without going into the details (we would be here for hours if I did) I’m looking into the overhead of context switching between PL/SQL and SQL. It it fairly common knowledge that when you call a PL/SQL function from SQL there is a context switch when the SQL engine hands over control to the PL/SQL engine. I’ve been doing some work into how much the overhead is and that it is incurred for each distinct PL/SQL function (plus loads of other considerations around it).

In doing so I saw something unexpected (to me, anyway) which I have simplified down to this:

select /* mdw_z1 */ avg(dbms_random.value) avg_dbrav1,avg(dbms_random.value) avg_dbrav2
      ,max(created) max_cre
from all_objects
where rownum <50000

AVG_DBRAV1 AVG_DBRAV2 MAX_CRE
---------- ---------- -----------------
.502234063 .502234063 11-SEP-2014 09:31
Elapsed: 00:00:00.89

Note that the two averages of DBMS_RANDOM.VALUE are exactly the same. It is so improbably as to be pretty much impossible that over all those rows, the different random values generated for each column add up to exactly the same. They are getting the same values for each row. I’m very, very dubious of any “identical seeding” issue (ie they both get the same seed and from then provide identical values) as even if DBMS_RANDOM is basing it’s output on something like initial seed, SCN of statement and number of iterations, it is still being referenced twice for each row.

Some further evidence is that when I increase the number of calls to DBMS_RANDOM the elapsed time is almost identical and the statement CPU and PLSQL_EXEC_TIME (pulled from V$SQL) do not increase in any significant way (PLSQL_EXEC_TIME actually goes down):

select /* mdw_z2 */ avg(dbms_random.value) avg_dbrav1,avg(dbms_random.value) avg_dbrav2
       ,avg(dbms_random.value) avg_dbrav3,avg(dbms_random.value) avg_dbrav4
       ,avg(dbms_random.value) avg_dbrav5,avg(dbms_random.value) avg_dbrav6
      ,max(created) max_cre
from all_objects
where rownum <50000

AVG_DBRAV1 AVG_DBRAV2 AVG_DBRAV3 AVG_DBRAV4 AVG_DBRAV5 AVG_DBRAV6 MAX_CRE
---------- ---------- ---------- ---------- ---------- ---------- -----------------
.500367568 .500367568 .500367568 .500367568 .500367568 .500367568 11-SEP-2014 09:31
Elapsed: 00:00:00.84 -- (cf .89 before)

-- information about the two SQL statements from v$SQL, identified by my comments
SQL_ID        PLAN_HASHV  PRSE  EXCS     BUFFS DISCS     CPU_T PLSQL_EXEC    RWS
------------- ---------- ----- ----- --------- ----- --------- ---------- ------
SQL_TXT
--------------------------------------------------------------------------------
4y4jsj7uy12t3 1587414607     1     1     41481     0    843750     312691      1
select /* mdw_z2 */ avg(dbms_random.value) avg_dbrav1,avg(dbms_random.value) avg
gpdga3qars8p2 1587414607     1     1     41481     0    828125     316648      1
select /* mdw_z1 */ avg(dbms_random.value) avg_dbrav1,avg(dbms_random.value) avg

I’ve verified that adding other PL/SQL calls adds about 0.4 seconds to the execution time per extra (simple) PL/SQL function and you see the CPU Time and PLSQL_EXEC_TIME increase.

It is as if Oracle realises the call to DBMS_RANDOM is the same and so does it once per row and puts the results into each column. It does this for other deterministic PL/SQL functions but I would (a) be worried if DBMS_RANDOM was being treated as deterministic (as it’s purpose is to NOT be deterministic) and (b) I know I have used DBMS_RANDOM to generate test data many times, often for several columns in my generating SQL SELECT statement, and I never noticed this before. So, I decided to check that it was only returning 1 value per row to populate the results for each column derived from DBMS_RANDOM.VALUE:


select dbms_random.value drv1, dbms_random.value drv2,created
from all_objects where rownum < 6

      DRV1       DRV2 CREATED
---------- ---------- -----------------
.341919389 .020497964 11-SEP-2014 08:40
.569447631 .727672346 11-SEP-2014 08:40
.019986319 .709239586 11-SEP-2014 08:40
.286970852 .144263004 11-SEP-2014 08:40
 .14440676 .538196808 11-SEP-2014 08:40

Huh? That rather damages my theory and actually works the way I expected, hoped for and thought I remembered. And you can just look at the two columns and you know they are not going to add up to exactly the same! (the last digit adds up to 21 for DRV1 , 28 for DRV2) {How many of you checked that and got 27 for DRV1?}.

So I wrote some code that should be logically identical to my original SQL statement but does the data collection “manually”:

with source as
(select /*+ materialize */
        dbms_random.value  dbrav1    ,dbms_random.value dbrav2
       ,created
from all_objects
where rownum <50000
)
select/* mdw_z3 */ avg(dbrav1)  avg_dbrav1     ,avg(dbrav2)  avg_dbrav2
      ,max(created)  max_cre
from source

AVG_DBRAV1 AVG_DBRAV2 MAX_CRE
---------- ---------- -----------------
.497954489 .497633494 11-SEP-2014 09:31

Elapsed: 00:00:00.96

As you can see, all I do is force the data to be collected into an internal temporary table using the WITH clause and hint it to stop oracle merging the code together and then average the columns. And now I get two different values for the two DBMS_RANDOM.VALUE derived columns.

This version of the code also accrues more runtime and statement CPU/PLSQL_EXEC_TIME, as I mentioned above, when I add more PL/SQL calls. In the below the extended list of “columns” version takes 1.78 seconds, CPU time increases from 843,750 microseconds to 1,703,125 and PLSQL_EXEC_TIME increases from 316,589 microseconds to 917,208

with source /*2 */ as
(select /*+ materialize */
        dbms_random.value  dbrav1    ,dbms_random.value dbrav2
       ,dbms_random.value  dbrav3    ,dbms_random.value dbrav4
       ,dbms_random.value  dbrav5    ,dbms_random.value dbrav6
       ,created
from all_objects
where rownum <50000
)
select/* mdw_z4 */ avg(dbrav1)  avg_dbrav1     ,avg(dbrav2)  avg_dbrav2
      ,avg(dbrav3)  avg_dbrav3     ,avg(dbrav4)  avg_dbrav4
      ,avg(dbrav5)  avg_dbrav5     ,avg(dbrav6)  avg_dbrav6
      ,max(created)  max_cre
from source

AVG_DBRAV1 AVG_DBRAV2 AVG_DBRAV3 AVG_DBRAV4 AVG_DBRAV5 AVG_DBRAV6 MAX_CRE
---------- ---------- ---------- ---------- ---------- ---------- -----------------
.499007362 .501985119 .498591643  .50252316  .49939127  .49804233 11-SEP-2014 09:31

Elapsed: 00:00:01.78

--
--
SQL_ID        PLAN_HASHV  PRSE  EXCS     BUFFS DISCS     CPU_T PLSQL_EXEC    RWS
------------- ---------- ----- ----- --------- ----- --------- ---------- ------
SQL_TXT
--------------------------------------------------------------------------------
49zjaaj41dg00 3034557986     1     1     43465   962   1703125     917208      1
with source /*2 */ as (select /*+ materialize */         dbms_random.value  dbra
0rtbx97f14b0k 3034557986     1     1     42294   382    843750     316586      1
with source as (select /*+ materialize */         dbms_random.value  dbrav1    ,

I did have the execution plans in here too but the post is already quite long. They are identical though, as is shown by the same value of 3034557986 for the PLAN_HASH_VALUE for both statements

So in Summary, the below two versions of the (logically identical as far as I can see) code give different results. The difference is that one is allowing Oracle to do the averaging natively and in the other I am forcing the data to be collected into an internal temporary table and then averaged:

select /* mdw_z1 */ avg(dbms_random.value) avg_dbrav1,avg(dbms_random.value) avg_dbrav2
      ,max(created) max_cre
from all_objects
where rownum <50000

AVG_DBRAV1 AVG_DBRAV2 MAX_CRE
---------- ---------- -----------------
.502234063 .502234063 11-SEP-2014 09:31

with source as
(select /*+ materialize */
        dbms_random.value  dbrav1    ,dbms_random.value dbrav2
       ,created
from all_objects
where rownum <50000
)
select/* mdw_z3 */ avg(dbrav1)  avg_dbrav1     ,avg(dbrav2)  avg_dbrav2
      ,max(created)  max_cre
from source

AVG_DBRAV1 AVG_DBRAV2 MAX_CRE
---------- ---------- -----------------
.497954489 .497633494 11-SEP-2014 09:31

If no one can explain what I am missing, I suppose I should raise a bug with Oracle. Which could be tricky seeing as my access to MyOracleSupport is a bit tenuous…

In case you want to play, this is my whole test script, which does everything but query V$SQL for the statement stats at the end. I am sure you can manage that yourselves…

-- the two columns from dbms_random get the same result - which I did not expect
select /* mdw_z1 */ avg(dbms_random.value) avg_dbrav1,avg(dbms_random.value) avg_dbrav2
      ,max(created) max_cre
from all_objects
where rownum <50000
/

select /* mdw_z2 */ avg(dbms_random.value) avg_dbrav1,avg(dbms_random.value) avg_dbrav2
       ,avg(dbms_random.value) avg_dbrav3,avg(dbms_random.value) avg_dbrav4
       ,avg(dbms_random.value) avg_dbrav5,avg(dbms_random.value) avg_dbrav6
      ,max(created) max_cre
from all_objects
where rownum <50000
/
--
-- The below forces oracle to gather the data into an internal temporary table first.
-- and then I average that
with source as
(select /*+ materialize */
        dbms_random.value  dbrav1    ,dbms_random.value dbrav2
       ,created
from all_objects
where rownum <50000
)
select/* mdw_z3 */ avg(dbrav1)  avg_dbrav1     ,avg(dbrav2)  avg_dbrav2
      ,max(created)  max_cre
from source
/
-- and now I make it do more executions against dbms_random, to see if makes a difference
-- which will lend support to my idea it is doing more contect switching.
with source /*2 */ as
(select /*+ materialize */
        dbms_random.value  dbrav1    ,dbms_random.value dbrav2
       ,dbms_random.value  dbrav3    ,dbms_random.value dbrav4
       ,dbms_random.value  dbrav5    ,dbms_random.value dbrav6
       ,created
from all_objects
where rownum <50000
)
select/* mdw_z4 */ avg(dbrav1)  avg_dbrav1     ,avg(dbrav2)  avg_dbrav2
      ,avg(dbrav3)  avg_dbrav3     ,avg(dbrav4)  avg_dbrav4
      ,avg(dbrav5)  avg_dbrav5     ,avg(dbrav6)  avg_dbrav6
      ,max(created)  max_cre
from source
/
Follow

Get every new post delivered to your Inbox.

Join 230 other followers