jump to navigation

Friday Philosophy – When Not To Tune! Or Maybe You Should? May 20, 2022

Posted by mwidlake in Friday Philosophy, performance.
Tags:
add a comment

This week there was an interesting discussion on Twitter between myself, Andy Sayer, Peter Nosko, and several others, on when you do performance tuning. It was started by Andy tweeting the below.

As a general rule, I am absolutely in agreement with this statement and it is something I have been saying in presentations and courses for years. You tune when there is a business need to do so. As Andy says, if a batch process is running to completion in the timeframe available to it, then spending your effort on tuning it up is of no real benefit to the company. The company pays your salary and you have other things to do, so you are in fact wasting their money doing so.

Please note – I said “as a general rule“. There are caveats, exceptions to this rule as I alluded to in my reply-in-agreement tweet:

I cite “Compulsive Tuning Disorder” in my tweet. This is a phrase that has been kicking about for many years, I think it was coined by Gaja Krishna Vaidyanatha. It means someone who, once they have sped up some SQL or code, cannot let it lie and has to keep trying and trying and trying to make it faster. I think anyone who has done a fair bit of performance tuning has suffered from CTD at some point, I know I have. I remember being asked to get a management report that took over an hour to run down to a few minutes, so they could get the output when asked. I got it down to a couple of minutes. But I *knew* I could make it faster to I worked on it and worked on it and I got it down to under 5 seconds.

The business did not need it to run in 5 seconds, they just needed to be able to send the report to the requester within a few minutes of the request (they were not hanging on the phone desperate for this, the manager just wanted it before they forgot they had asked for it). All that the added effort I spent resulted in was a personal sense of achievement and my ability to humble-brag about how much I had sped it up. And my boss being really, very annoyed I had wasted time on it (which is why it sticks out in my mind).

However, there are some very valid reasons for tuning beyond what I’ll call the First Order requirement of satisfying an immediate business need of getting whatever needs to run to run in the time window required. And a couple of these I had not really thought about until the twitter discussion. So here are some of the Second Order reasons for tuning.

1 Freeing Up Resource

All the time that batch job is running for hours it is almost certainly using up resources. If parallel processing is being used (and on any Exadata or similar system it almost certainly is, unless you took steps to prevent it) that could be a lot of CPU, Memory, IO bandwidth, even network. The job might be holding locks or latches. The chances are very high that the 4 hour batch process you are looking at is not the only thing running on the system.

If there other are things running at the same time where there would be a business benefit to them running quicker or they are being impacted by those locks, it could be that tuning those other things would take more effort (or even not be possible) compared to the 4 hour job. So tuning the four hour job releases resource or removes contention.

I had an extreme example of this when some id… IT manager decided to shift another database application on to our server, but the other application was so badly designed it took up all available resource. The quick win was to add some critical indexes to the other application whilst we “negotiated” about it’s removal.

2 Future Proofing

Sometimes something needs tuning to complete in line with a Service Level Agreement, say complete in 10 seconds. You get it done, only to have to repeat the exercise in a few months as the piece of code in question has slowed down again. And you realise that as (for example) data volumes increase this is likely to be a recurring task.

It may well be worth putting in the extra effort to not only get the code to sub-10-seconds but significantly less than that. Better still, spend extra effort to solve the issue causing the code to slow down as more data is present. After all, if you get the code down to 1 second and it goes up to 5, even though it is meeting the SLA the user may well still be unhappy – “Why does this screen get slow, then speeds up for a few months, only to slow down again! At the start of the year it was appearing immediately, now it takes forevvveeeeer!”. End users MUCH prefer stable, average performance than toggling between very fast and simply fast.

3 Impacting Other Time Zones

This is sort-of the first point about resource, but it can be missed as the people being impacted by your overnight batch are in a different part of the globe. When you and the rest of your office are asleep – in America let’s say – Other users are in Asia, having a rubbish time with the system being dog slow. Depending on the culture or structure of your organisation, you may not hear the complaints from people in other countries or parts of the wider company.

4 Pre-emptive Tuning

I’m not so convinced by this one. The idea is that you look for the “most demanding” code (be it long-running, resource usage, or execution rate) and tune it up, so that you prevent any issues occurring in the future. This is not the earlier Freeing Up Resources argument, nothing is really slow at the time you do this. I think it is more pandering to someone’s CTD. I think any such effort would be far better spent trying to design in performance to a new part of the application.

5 Restart/Recovery

I’d not really thought clearly about this one until Peter Nosko highlighted it. If that four hour batch that was causing no issues fails and has to be re-run, then the extra four house could have a serious business impact.

Is this a performance issue or is it a Disaster Recovery consideration? Actually it does not matter what you label it, it is a potential threat to the business. The solution to it is far more complex and depends on several things, one of which is can the batch be partially recovered or not?

When I design long running processes I always consider recovery/restart. If it is a one-off job such as initial data take-on I don’t put much effort into it. It works or it fails and your recovery is to go back to the original system or something. If it a regular data load of 100’s of GB of critical data, that process has to be gold plated.

I design such systems so that they load data in batches in loops. The data is marked with the loop ID and ascending primary key values are logged, and the instrumentation records which stage it has got to. If someone kicks the plug out the server or the batch hits a critical error, you can restart it, it will clean up any partially completed loops and then start from after the last completed batch. It takes a lot more effort to develop but the pay-back can be huge. I worked on one project that was so out of control that the structure of the data was not really known and would change over time, so the batch would fail all the bloody time, and I would have to re-run it once someone had fixed the latest issue . The ability to restart with little lost time was a god-send to me – and the business, not that they really noticed.

Simply tuning up the batch run to complete more quickly, as Peter suggests, so it can be blown away and repeated several times is a pragmatic alternative. NB Peter also said he does something similar to what I describe above.

6 Freeing Up Cloud Resource

Again, I had not thought of this before but Thomas Teske made the point that if your system is living in the cloud and you are charged by resource usage, you may save money by tuning up anything that uses a lot of resource. You are not satisfying a First Order business need of getting things done in time but you are satisfying a First Order business need of reducing cost

7 Learning How To Tune

I think this is greatly under-rated as a reason to tune something, and even indulge in a little CTD even when the First Order business need has been met. It does not matter how much you read stuff by tuning experts, you are only going to properly learn how to performance tune, and especially tune in your specific environment, by practice. By doing it. The philosophy of performance tuning is of course a massive topic, but by looking at why things take time to run, coming up with potential solutions, and testing them, you will become really pretty good at it. And thus better at your job and more useful to your employer.

Of course, this increase in your skills may not be a First, Second, or even Third Order business need of your employer, but it could well help you change employer 😀 .

8 Integration Testing

Again highlighted by Peter Nosko, we should all be doing integrated, continuous testing, or at least something as close to that as possible. Testing a four hour batch run takes, err, more time than a 15 minute batch run. For the sake of integration testing you want to get that time down.

But, as I point out..

{joke – don’t take this tweet seriously}

As I think you can see, there are many reasons why it might be beneficial to tune up something where the First Order business need is being met, but the Second Order benefits are significant. I am sure there are some other benefits to speeding up that batch job that finishes at 4am.

I’ll finish with one observation, made by a couple of other people on the Twitter discussion (such as Nenad Noveljic). If you fix the First Order issue, when management is screaming and cry and offering a bonus to whoever does it (like that ever happens) you get noticed. Fixing Second Order issues they did not see, especially as they now may not even occur, is going to get you no reward at all.

Rescuing An Almost Dead Lawnmower May 3, 2022

Posted by mwidlake in off-topic, Private Life.
Tags: ,
3 comments

As I slowly slip into something like proper retirement I thought I’d do some more blogs on “retired” topics. And as I know some of my Twitter followers have shown a strange fascination with my lawnmower and wheelbarrow “collection”, I thought I’d show you how I saved my old Honda Izy lawnmower from it’s imminent demise.

I’d had my old Honda Izy since 2007 and, though it had served me well, I had not been as good at cleaning if after use as I should. And it was often used on rough ground or ground with lots of stones on it, so the inside of the deck (the grey body that the engine, wheels, and other parts are mounted on) had got a lot of abuse and rusted badly. Grass mulch is quite corrosive. Looking on the net the rusting of the deck is a common problem with the Honda Izy. If I’d taken more care of it I am sure the Izy would have lasted many years longer. Anyway, the deck was splitting all the way around the engine mount and the engine & cutting blades were wobbling, causing the blades to catch the inside of the deck whenever it got jolted. In time the whole engine/blade unit was going to come away, probably when at full spin. That could be nasty! As you can see, I had used duct tape to keep the mower going until I got it fixed or replaced it, but this was my third or so duct tape repair and the whole machine just felt dangerous.

I did enquire about getting a new deck for the mower but the person who maintains if for me, a garden machinery mechanic, refused to do it. “The deck is not cheap, well over £100, I have to take the whole lawnmower apart which can be a right swine, re-build it on the new deck and then sort out all the little issues found doing it. It would take me a whole day or more. I’d charge you almost as much as a new mower costs”. So, as you can see from the above, I did buy a new Honda Izy. I have to say, the engine is a little quieter and smoother on the new one.

I looked at a couple of videos online about replacing the deck myself, but it did look very fiddly and that is with all the tools you need. However, I resented throwing away a perfectly good lawnmower and cracking little engine just because the deck was knackered. How about a car body repair kit? Hmmmm. I’d used one 32 years ago on my Mark 1 Golf, I could do this…

To prepare the lawnmower I emptied the oil (it needed replacing anyway), removed the blades, took off the petrol tank & fuel filter and gave them a thorough washing. There was a fair bit of sediment in the tank and fuel filter so it was good get rid of that, it had been causing some running issues. I then scraped away most of the caked on dirt, grass, rust, washed away most of the remainder and rubbed it down with wire wool. I did try a wire brush but that was pretty ineffective, wire wool was much better. Finally, I gave it a rub over with some rough glass paper, a final wash, and I left it 24 hours to dry out. This was last summer, it was about 30C during the day so it dried quickly. It might not look that clean in the pictures but that’s due to the remaining specs of paint and pits with a little rust in. I could see the metal was scoured and ready to be a good surface to bond to.

If you have never used fibreglass body repair kits before, it is very simple. You have a sheet or two of glass fibre (the white stuff) that folds out like fabric, a pot of resin, and a little packet of hardener. And a pretty useless pot and stirring stick, plus a crap brush. I’d advise getting another pot (a yoghurt pot will do at a pinch), a second little brush, and some spare sticks. My brother keeps lolly sticks (like you get with a Magnum ice-cream) for such purposes, I use literally a little stick off the ground!

WEAR GLOVES! The fibres of glass can get into your skin and irritate like crazy and if you get the resin on your skin it is not coming off until the skin does. Most kits come with a crap pair of thin plastic gloves but you can buy a pack of 100 disposable gloves for a few quid and you will need more than one pair probably. Put on the gloves and then cut the fibre glass sheet up into patches that will cover the area you want to repair. Normal scissors will be fine for this. 32 years ago I tried to cut just one piece to fit the whole repair, it was not ideal. And in this case I am fitting the sheet to a curved, circular surface. I cut several smaller pieces, about 5, and put them in place to make sure all looked OK. Now remove the fibreglass pieces.

Still with the gloves on, put some of the resin in the pot, I used about a third of the tin. Add the hardener as described. Actually, don’t, I made that mistake. Add about half the hardener as described, especially if it is warm like it was this day, and mix quickly and thoroughly with the stick, do this in 30 seconds if you can. Generously paint the edges of the area you are fixing with a little of the mixture, covering all the area the fibreglass is going on, and then put the fibreglass patches over it. The mixture should make them stick in place. Now put the rest of the mixture over the fibreglass and work it in with the brush. If the fibreglass patches shift or you ruck it or in any other way make the fibreglass patches do something you don’t want, fix it ASAP. This stuff goes tacky and then gloopy very quickly, which is why I say only put half the hardener in. The aim is to soak all the fibreglass with the resin and get it worked into the fabric before it turns to treacle. If you have time, scrape any residue out the pot with the stick and either try and get it on the repair or use some scrap of waste fabric, piece of card or whatever to get the gloop off. This way you might be able to use the pot and stick again. The brush is probably history.

I left the first layer for about 4 hours, then I cut a second set of smaller pieces to cover the weakest areas (where, basically, there was no metal anymore), mixed up some more resin and (less) hardener and used it to apply this second layer.

I left it to cure overnight, then gave the repair a sand with glass paper and washed it down. I then painted the repair with a tough paint designed to go on rusty metal (“Hammerite” in my case) and once it was dry, a second layer over the whole area. Be generous, it fills little pits and missed bits in fibreglass that somehow did not get enough resin in it.

I should mention I took care not to get any resin or paint on the nuts holding the engine onto the deck, just in case I ever want to remove the engine.

I now had a workable repair and it looked OK from the inside. If I did it again I would uses less hardener from the start and take a little more care over applying the fibreglass.

I turned the mower right way up and I could see the fibreglass through the holes in the metal. I used wire wool to prepare the surfaces arounds the holes, pushed any proud metal firmly down onto the fibreglass repair, mixed up the last of my resin/hardener mix and used the scraps of fibreglass matt I still had to patch the top and smothered the repair and surrounding metal in resin.

Once it was dry it was Hammer(ite) time again and I painted the repairs and some other areas of scratched paintwork and mild rust.

The engine now felt rock solid and the deck was not flexing at all. I counted this as Job Done.

I gave the blade a damn good clean and sharpen (though it could have done with grinding back another 5mm to be fair), put that back on, put the cleaned fuel tank and engine cover back on, replaced the air filter & spark plug and put in the oil (standard servicing kit) and then finally a bit of fuel. Only a bit in case I needed to empty it again…

I set the choke, pulled the starter cord, and the engine fired straight up in a couple of pulls. I did a couple of swipes along the rough grass near the castle mound and it cut beautifully. The engine was a lot smoother now too, due to the clean fuel system and new air filter. I now filled the fuel tank.

I’m the first to admit it is not the cleanest, most professional looking repair in the world, but it has proven to be effective. I did this repair last year and I’ve used the repaired machine many times since.

I use the old Honda Izy on rough terrain and anywhere I think there may be stones or similar that might damage the new Izy. And I am cleaning off the new lawnmower more conscientiously as, though I now know I can patch a knackered mower deck, it took me a few days and I’d rather not have to do it again through sheer laziness.

The old lady is now a useful member of my lawnmowing family again…

(BTW the below is not the full set, there is the petrol strimmer. And the hover mower. And the hand-held trimmer for small areas around plants…)