New resource not so much a challenge as an opportunity for firms

Over the past three years, many businesses have shifted from thinking of big data as a challenge to perceiving it as the opportunity it is.

In order to better understand enterprise big data trends, Microsoft Corp conducted a study among 280 top decision-makers last year. The findings were interesting in the sense that more than 38 percent of respondents' current data stores contained unstructured data (valuable, but unorganized information).

Nearly 53 percent of the respondents rated increased amounts of unstructured data to analyze as extremely important. This trend is only increasing as more businesses realize that unstructured data holds the key to new value not accessible in their existing structures.

CeBIT: big data, social business, mobile, cloud and more

Managing big data in a complex economy

When big data can lead to big profit

There are two key measures to tapping into ultimate value in big data. The first is time to gain insight. The second is return on accessible data. These measures are, in turn, enabled through a process of producing information, which means converting data or information from one domain into another.

Consider the following example. A fleet of ambulances is equipped with GPS units that collect telemetry. Information production techniques allow conversion of the raw GPS telemetry, a sequence of records containing time and location, into an incident response time.

The magic of information production is that it takes data difficult to deal with in traditional information systems, such as raw GPS telemetry, and transforms it into information that is both more structured and more business relevant. Once the incident response time is produced, patient outcome can be predicted.

Great information production tools can also allow a reduction in the time to gain insight.

They allow one to get from a hunch to validation quickly. In fact, there is an emerging class of these tools that can make great predictions by finding correlations in diverse data sets, which may hold the new untapped value.

Valuable answers require joining different data sets in logical ways. In traditional databases, what you would call "accessible" data is constrained to data contained in the database. This data has been normalized, cleaned and indexed so it can be used to efficiently answer a fixed set of questions over that data domain.

Big data and information production, on the other hand, enable a much larger definition of accessible data. Going back to the ambulance example, it could determine how many lives could be saved through shorter response times. By using accessible demographic and population data, one could determine how many heart attack victims could be saved by moving or adding ambulances.

We all know big data has become mainstream when it makes everyday experiences better. For example, by conducting an analysis on unstructured data with big data tools, the Microsoft Halo 4 (video game) team gained insights into the use patterns and thereby made changes to further improve the overall gaming experience.

There is no doubt that we are all starting to benefit from big data. Companies such as Microsoft are working to make sure that the future promise of big data is fulfilled in ways that we can now only scarcely imagine. It is focused on developing big data offerings that allow faster insight time and greater return on accessible data.

The author is chief technology officer of the Cloud and Enterprise Engineering Group, Microsoft Corp. The views do not necessarily reflect those of China Daily.

Chinese Internet firms dig big data gold	The history of cloud computing