Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it – Dan Ariely
Over the years, we’ve been to many events and have seen our fair share of speakers who tackled the topic of defining big data. All of them went in the same route: explain it with the technicalities like Hadoop clusters. The end result leaves the audience saying they will use big data but with an even more convoluted idea of what exactly big data is.
Now you can guess our surprise when we saw an email from Chamara Rupasinghe – Head of Information Security Business at Just in Time Group. It was an email with a simple and clear explanation of exactly what big data is WITHOUT complex technical jargon.
So what is big data?
Big data is a set of high volume, high velocity and high variety data, which needs innovative ways to process and analyse to bring out sensible decision making set of information – Chamara Rupasinghe
Simply put Big Data is literally a large pool of data. It’s a pool of data so large, that if you put it in a normal database and try analysing it with traditional SQL queries you won’t get anything remotely useful. This is because a Big Data set also has 3 unique aspects. They are:
- High volume: This refers to the vast quantities of data stored being very high. This data could easily be machine generated. As an example: take Facebook which stores 300PB of data.
- High velocity: This refers to the speed at which the data is generated being high. This can even be small chunks of data but it could be generated at a rapid pace.
- High variety: The types of data will also be different, simply unstructured as the multiple sources will provide the data in many different formats.
There you have it. That is big data in the simplest of terms.
Who can use Big Data and what do they gain?
Quite a lot actually. Big Data analysis can be applied to various industries, which when applied will see massive gains.
One example Chamara shared with us is supermarkets. By analyzing the details of transactions by customers, supermarkets can identify their buying patterns. This provides supermarkets with an opportunity to upsell and improve customer satisfaction.
Another example Chamara shared is the Insurance industry. Through Big Data analytics using past data from various source systems, insurance companies could save millions. This is because it would allow them to easily set premiums at the initial agreement stage and more importantly: identify fraudulent claims.
The biggest winners with Big Data analytics though, will be those in the healthcare industry. Let’s take the average patient in a hospital. On a daily basis he will be generating large amounts of data with his/her tests. Now let’s take an entire hospital full of patients, each generating this much data. That’s a large amount of data. If doctors could analyse this data alongside environmental and social economic data, they could potentially prevent outbreaks before they happen.
While Big Data does have it’s benefits, it’s not without it’s challenges. One being being security. Keeping vast quantities of data secure is not easy. Keeping vast quantities of data with varied formats from various sources which are spread across various geographic regions, is another challenge entirely.
Another challenge with Big Data is the series of ethical issues that inadvertently arise due to data privacy. That’s a moral gray area that we are all still debating.
Can we use big data in Sri Lanka?
If you asked that question one year ago, the answer would have been a definite no. Today, however things are different. Many companies in Sri Lanka have already started to implement some form of Big Data analysis in their operations. In fact, many service providers, retailers and online sellers use social media analytics to analyse the feedback from their customers and potential buyers in their decision making.
This of course just the start, going forward we probably will see Big Data being adopted more readily by companies. Hopefully when they do, their employees will be able to explain what Big Data in simple terms.