This is the first post of a series on Big Data. Watch for more!
The success of information technology until now has been built on our ability to comprehensively record transactions, events, or changes of state. We have made great use of this transactional data, optimizing inventory, streamlining processes, automating activity. Now, however, we can track and record the behavior that leads up to, or follows from, these transactions. People use computers, phones, the internet to do more and more. Each click and call is recorded and makes up a web of behavioral patterns. Computers are used for designing, making, selling, buying, trading – each step along the way is recorded and makes up a web of behavioral patterns. Markets are now computerized and each bid, each offer, each trade is recorded. Each item of news about a market, a company, a financial asset is also recorded and cross-correlated to the market activity. Systems are observed in finer detail, and instrumented in real time, not just when a transaction occurs or a state changes. By analyzing all this behavior we hope to be able to diagnose, to predict, to intervene; we hope to sell more, or price better, to make more efficiently, to diagnose disease and design treatments. We want this behavioral data because it promises to unlock value commensurate with its volume, velocity and variety, and this behavioral data, is, um, big.
This data is so big, in fact, that it is causing problems in the technology world. That’s why it has this name: big data. You might not have heard this term until now, but now you have read it here, expect to read or hear about it three more times in the next few days. What exactly is big data? All the definitions seem based on a notion that the problems of size make it noteworthy. Wikipedia offers “In information technology, big data is a loosely-defined term used to describe data sets so large and complex that they become awkward to work with using on-hand database management tools.”
This doesn’t tell us why we have so much data or really why we should care. That is why I started this series on Big Data with this observation: we used to track transactions, and now we are tracking behavior.
Make no mistake, Big Data is about behavior – of people, systems, markets and machines.
At some stage, technology companies will solve the problems that make this data hard to ingest, handle, process, analyze, understand. It will no longer be big data, because it won’t be too big to manage.
However, without any doubt, behavioral data is here to stay!