jump to navigation

Analysing UKOUG Presenter’s – I Know How You Performed. June 16, 2015

Posted by mwidlake in conference, Presenting, UKOUG, User Groups.
Tags: , , ,
trackback

This last few days I’ve been analysing how well received previous presentations (since 2006!) have been at the UKOUG tech conferences. It’s interesting to look at the information. I’ve learnt some interesting things about all those well-known-names πŸ™‚

Like many conferences and user group meetings, during the conference and shortly afterwards the UKOUG ask attendees to feed back on the presentations, keynotes and round tables that people go to. If you chair a session, one of your tasks is to request people do the feedback forms. Talks are judged for several aspect (concept, quality of slides, presentation skills, overall value and a couple more) from 1 (very poor) to 6 (excellent). You can also add a free-text comment. The reason for an even number of possible scores it to prevent a non-committal middle score. Why not 1-8 or 1-10? I don’t really know, but I did see a blog post recently about using a wider range and it seemed to not really add to the overall benefit of the feedback as the very top and very bottom scores were never used. This information is compiled and fed back to the individual speakers, along with the average scores across the event. Speakers are very keen to know how they did compared to everyone else {no egos involved here πŸ™‚ } and also any specific comments on their efforts. It is important to us speakers, we need to know if you liked or disliked what we did so we can improve. Or sulk.

This is an example of the feedback we get (one of mine, of course).
Speaker Scores

Something that has annoyed me for many years is that the speaker scores are not formally analysed and fed into the speaker selection process for future UKOUG conferences. I used to get really quite vexed by this {ie bad tempered and, well, annoying & complaining to the UKOUG office}. I suspected they were going to do to me what you should always consider doing to some loud-mouthed complainer and say “well, if you are so passionate about this – you damn well do it!”. So I offered to take the data and process it.

The information I received was not the raw feedback forms, but the average scores per talk – the information actually passed back to the presenters. Which is perfect for my purposes. I analysed data from 2006 to 2014 {though 2008 was missing), so a pretty comprehensive data set. SIG talk feedback is also missing, I’ll work with the UKOUG office to incorporate that for version 2 of the analysis.

Compiling all the data into a single rating per speaker is more demanding than first seems. Isn’t any data analysis project? Rather than consider all areas speakers are judged on I decided to score people on only two dimensions (areas to you and I) – presentation skills and overall value of the session. When we judge papers those are the two main things we want to know – can the person present well and do they usually give a talk people value.

Some of the challenges were:
– What to score speakers on – I’ve just said what I chose.
– People’s names. I need to group talks on this but it’s a free-text field of course so I had to clean that. I stripped off title, edited variations and then reviewed. For women, their Surname can alter with marital status {I find it rather archaic that this is still very common and almost exclusively impacts women. But then, I offered to my wife we use her maiden name as our married one and she was fine to take mine.} But you also get variations on first name, spelling mistakes and alterations of title. I had to solve that to group scores.
– The number of feedback forms received for a given presentation. If a presentation only gets one feedback form, how reliable are the scores? If it gets 5 feedback forms, how reliable is that? 35 forms? I came up with a weighting based on the number of feedback forms where if only 1 feedback score was given, it held less weight than 5, that held less weight than 15… etc.
– Oddities of eg co-presenters or the “speaker” actually being a facilitator, as is often the case with Round Table sessions.
– The data, as held in Excel, being “damaged”. ie it caused my analysis issues as things had been done to the data to support other purposes – what was important to the UKOUG organising the conference that year. Sorting those issues out took up most of my efforts.
– The fact that I was using Excel as the analysis tool. I’m a SQL guy!!!! But the thing is, with a relatively small volume of data and a need to constantly visually check alterations, some things are just much easier in a tool like Excel than SQL. And some things are way harder.

In the end, I got a set of scores that helped us on the Agenda Planning Day (well, it did in the database stream) and hopefully will develop over the years. It would be wrong of me to discuss how specific Oracle Names did, especially any who did poorly, but the scores informed our deliberations this year and should do so for years to come. If you want to contact me directly and ask me how you did – I won’t tell you (or anyone else). But I can talk about more generic things I discovered.

Over all presentations from 2006 to 2014
the average number of feedbacks for a session is around 10
The average for Presentation Skills is 4.6
The average for Overall Value is 4.5

So almost 4.5 out of 6 as the average scores, which is “Good” to “Very Good”.

NB I do not calculate my averages in the same way as the UKOUG office.

Because I weight my scores and remove zero values (and probably a couple of other differences, such as I already have averages not the raw scores) my average scores do not compare and are higher to the ones they issue for events. I think I am a harsher judge πŸ™‚

So what were some of the interesting things I discovered?

  • Well, for starters, scores for a given presentation rarely hit as low as 3. In fact, except for a small number of stand-out-bad talks, most scores of 3 were where only 1 feedback score was received, some with 2. We don’t seem to like giving low feedback scores. The same goes for 6. I only saw 6 if the number of feedback forms was 1 or 2. So reliable scores are between 3.01 and 5.99 really.
  • As I was stripping off people’s title by manual replacement runs, I know how many Mrs, Ms, Miss, Dr etc we get. Miss and Ms go up – and down – over the years. It varies a lot, but Ms is becoming more common. What is disappointing is the consistently low number of presentations by women. But I know that in the years I have been involved, the proportion of presentations by women is in proportion to the number of submissions, or even a tad higher. Come on ladies, represent your constituents! Some of the highest speaking scores are by women.
  • Another thing I get from the person title, we get no papers submitted by Professors, Colonels (or any military bigwigs), members of the clergy or peers of the realm. Or members of royal families. They are simply not trying are they?
  • On a personal note – I am Utterly Average. Over 8 years I fall number 296 and 298 out of 623 speakers for Presentation Skills and Overall Value respectively. Have you any idea how much that damaged my ego?!? I was gutted! Where I am a little more unusual is my average number of feedback scores, which is 21.6, in the top 15%. I’m massaging my ego with that (it’s all I’ve got!).
    {what is really vexing is I dug out my scores from earlier years and they were better than my running average – and my scores are pulled down by one talk in 2010 where I really bombed. Have you any idea how tempting it was for me to delete that one talk out of the data set?}
  • Some speakers, a small number, always-always-always get high votes, mostly as they are excellent but with an added slice I suspect of of, well, they are deeply respected. But interestingly, even well known people (what I think of as the ‘B’ list and even a couple of ‘A’ listers in my opinion) can bomb. Some regularly. I mean, if you saw the scores for….no, I won’t say :-).
    But the scores for individual speakers can and do vary. I saw one speaker, who in my opinion is a brilliant technician and a fantastic speaker, be up in the high 5.8’s for one talk and then down in the low 3’s for another. That made me dig in further and there are several people I know and hold a similar opinion on who have high and low talks. So that makes me feel that the user feedback scores are generally reliable and even respected speakers will get a poor score if the talk misses it’s mark. The best just simply never miss the mark, or not by much.
  • Not to be too harsh, but if you score 4 or below for either presentation skills or overall value and got 3 or more feedback forms – you bombed.

But bombing occasionally is OK. I’ve bombed (well, this close to bombed) and I’ve learned. Many excellent presenters have bombed. We all alter in our presenting skills over time. Most of you get better over time – I’ve got a tiny bit worse! But if you bomb all the time? Then maybe presenting is not your thing. It is not the only route to spreading the word, maybe try writing. But, again to be harsh, if you can’t present we owe it to the delegates of the conference not to select you to present.

Those of us organising the content know, as a group, who the best speakers are. We ensure that they get slots. And we have a good feel for who the better speakers are and they get looked on “favorably”. We do this as we want the best content and experience for the audience. Eric Postlethwaite may be a genius at VPD and know it inside out, but if they present like a cardboard cut-out with bad breath then the session will be a failure. Judging scores are the top filter but we on the planning committee keep in mind how good a speaker is. What worried me was that this was not scientific, it was word-of-mouth and gut-feel, which is why I spent many days in Excel World to take the raw feedback and convert it into scores. I want the audience feedback to influence the content.

If you speak (or have spoken) for the first time and your scores are below average, don’t worry too much. As you can see from the above, you are up against a pre-selected set of known, excellent speakers. Hitting average is actually something of an achievement (and I would say that as I am Mr Average!).

One thing jumped out at me. I looked at all the comments (and I do mean all) for a couple of years and I noticed that you get the odd person who tries to “make a point” by adding the same comment to all the speakers’ feedback forms they fill in. Don’t do that. The speakers do not deserve your ire at the conference. It’s childish of you. If you want to raise an issue with the conference as a whole, don’t spam it on the speaker feedback forms, you dilbert, be an adult and contact someone involved in the conference organisation direct. Oddly enough, we DO like to have people come and say what you felt did not work, but spamming it on all the speaker feedback forms is just non-directed trolling.

I said I would not name names, it is not fair. But I’m going to name one though, and this is based on MY opinion of what I have seen looking at the stats. This is not official UKOUG opinion. Connor McDonald? Your presentation skills are awesome. I wanted to edit your scores down through pure envy. You are a good presenter, sir.

Advertisements

Comments»

1. Connor McDonald - June 17, 2015

Thanks for the compliment Martin, much appreciated.

Now that I’m inside Oracle, I imagine the challenges will be harder …. because given the mandatory white background, I’m expecting to take a beating on scores, just for strobe flash ! πŸ™‚

2. Scott Wesley - June 20, 2015

I concur with the outcome, Connor is one of the best speakers I’ve seen, though my sample size is relatively small and I shall revisit this opinion post Kscope15 ;p
I do hope is style isn’t cramped too much by the weight of Oracle. With enough elbow room he gets his point across very well, and people come out with a smile on their face.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: