jump to navigation

Friday Philosophy – Size is Relative February 15, 2019

Posted by mwidlake in Architecture, Friday Philosophy, Hardware.
Tags: ,
trackback

The below is a USB memory stick, a 64GB USB memory stick which I bought this week for about 10€/$. I think the first USB memory stick I bought was 8MB (1/8000 the capacity) and cost me twice as much.

Almost an entry level USB memory stick these days

This is a cheap, almost entry level USB memory stick now – you can get 0.5TB ones for around €50. I know, I know, they keep getting larger. As does the on-board memory of computers, disc/SSD size, and network speed. (Though compute speed seems to be stalling and has dropped below Moore’s law, even taking into account the constant rise in core count). But come on, think about what you can store on 64GB. Think back a few years to some of the large systems you worked on 10 years ago and how vast you might have thought the data was.

What made me sit back a little is that I’ve worked with VLDBs (Very Large DataBases) for most of my career, the first one being in about 1992/3. And that was a pretty big one for it’s time, it was the largest database Oracle admitted to working on in the UK back then I think. You guessed it – this little USB memory stick would hold the whole database, plus some to spare. What did the VLDB hold? All the patient activity information for a large UK hospital – so all patients who had ever been to the hospital, all their inpatient and outpatient stays, waiting list, a growing list of lab results (blood tests, x-rays)… The kit to run this took up the space of a large lorry/shipping container. And another shipping container of kit to cool it.

What makes a database a VLDB? Well, This old post here from 2009 probably still covers it, but put simply it is a database where simply the size of it gives you problems to solve – how to back it up, how to migrate it, how to access the data within it in a timely manner. It is not just about raw volume, it also depends on the software you are using and the hardware. Back in the mid 2000’s we had two types of VLDB where I worked:

  • Anything above 100GB or so on MySQL was a VLDB as the database technology struggled
  • Anything above 2TB on Oracle was a VLDB as we had trouble getting enough kit to handle the IO requirements and memory for the SGA.

That latter issue was interesting. There was kit that could run 2TB Oracle database with ease back then, but it cost millions. That was our whole IT budget, so we had to design a system using what were really beefed-up PCs and RAC. It worked. But we had to plan and design it very carefully.

So size in I.T. is not just about the absolute volume. It is also to do with what you need to do with the data and what hardware is available to you to solve your volume problems.

Size in I.T. is not absolute – It is relative to your processing requirements and the hardware available to you

That USB stick could hold a whole hospital system, possibly even now if you did not store images or DNA information. But with a single port running at a maximum of 200MB/s and, I don’t know, maybe 2,000 IOPS (read – writes can suck for USB memory sticks), could it support a whole hospital system? Maybe just, I think those figures would have been OK for a large storage array in 1993! But in reality, what with today’s chatty application design and larger user base, probably not. You would have to solve some pretty serious VLDB-type issues.

Security would also be an issue – it’s real easy to walk off with a little USB Memory stick! Some sort of data encryption would be needed… 🙂

Comments»

No comments yet — be the first.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: