Perhaps you’ve heard it before? The excitement and promises on unlimited possibilities for renewed revenue and profit. The new rescuer that will save the corporate world and society – the technology that will answer all our questions.
From my leading paragraph you might sense that the idea of Big Data as a digital Messiah who will change the way we operate in firms and in society today haven’t fully seduced me just yet. But the concept and technology I think is really interesting. Furthermore, using Big Data as part of the development in firms is intriguing – however, I see an evolution more than a revolution.
But first– let’s set the frame of reference, as the hype cloud surrounding data is fairly dense. We already have the Data Warehouse and business intelligence solutions, but also analytics and advanced analytics – Why Big Data then? Let’s look at how everything is related to each other.
The connection – from data to Big Data
Data placed in a digital platform is not valuable in itself. The value from the data is generated when it’s pulled from the platform and the analytical process and is initiated, then it creates the basis for the subsequent processing. The analytical element is introduced with the concept of business intelligence. Business intelligence is like an umbrella, a bit edgy but still covers standard and ad hoc reporting as well as visualization of data.
A Data warehouse is a platform which main job is to collect data. We collect, clean, group and provide data for analytical purposes. Data is collected typically from a wide range of data sources and it’s integrated so that data from the acquired systems overall represents the company’s processes in a new context.
Software providers and consulting firms have for a period talked a lot about analytics and especially the notion of advanced analytics has become a hot topic. The analytics concept covers in reality a re-branding of the business intelligence paradigm. Everything is now more proactive; less focused on technology and more focused on the business.
Advanced analytics covers models that can predict the outcome of a hypothesis by finding hidden patterns in a great number of variables. As I recall, this was previously named data mining.
The technology supporting Big Data differentiates itself from the well know relational database platforms. Without diving too deep into the chemistry of ACID and BASE or in the CAP theorem, you could say that Big Data platforms focus is on availability and the ability to process large amounts of data through the distribution of workload in a cost efficient setup.
The area is starting to mature to a level where it actually supports value creation for the company, and not only serves as a ‘technological playground’ surrounded by immature open source technology that hardly integrates with your enterprise environment.
Why Big Data then? How does it fit in?
Big Data can actually fit in several places – it depends on the context you wish to apply Big Data in. Big Data may well occur as part of your operating environment, as well as your analytical environment.
Big Data applied in an operational context makes sense when you consider the automation wave which focus is on the manufacturing sector at the moment. The requirement for increased productivity drives the need for automation, which again is driven by data. Use cases for predictive maintenance of your production line with an increased OEE as a result, or the opportunity to do real time production planning holds a range of interesting possibilities.
From an analytical perspective, Big Data could be used to enrich your data warehouse even further. The data warehouse as we know it, still has its birthright in a Big Data era. With terms and definitions broad – and consistent anchored in the organization, there is a great opportunity to enrich the existing data with data from new sources – sources that traditionally have been hard to deal with in the current data warehouse infrastructure, but which the Big Data platforms are designed specifically to handle. Within all sectors and business’ there are opportunities to enrich your existing analytical foundation with Big Data, but with examples from banking- and insurance the opportunities to refine already existing churn- and lifecycle models through public data, data from social media, etc. is present.
I admit – you do not necessarily need a data warehouse to do analysis based on Big Data. In a context where focus is on an isolated process, Big Data can constitute the analytical foundation – and furthermore it’s possible to communicate results, but if the goal is data driven business development there’s a need for a wider reference, and here the data warehouse and its information model will serve its purpose.
What are the take ways?
My message isn’t that Big Data is the only true way – on the contrary. Big Data can be viewed as a supplement to the data bases already existing in the organization. The key point is: just use your data – keep being curious. That Big Data is not just seen as hype but rather as a concept, which, unfortunately, has been over-hyped. That you don’t deduct unstructured data as not eligible because it can’t be consumed by the tools you have available at the moment. That you are not scared off the frequency by which the data originates or the data’s sheer volume. The technology to support all of the above already exists – it requires an investment – both the technology itself and in its use, but it is after all just technology and education.
The true challenge is within the organization’s ability to ask the right questions, set up the right hypotheses, interpret the answers and convert those answers into action. So – keep being curious and creative in the way you work with data!