The new databases, part one (of three): a proliferation of analytical engines

Most enterprise databases have come in two basic flavors over the past 25 years:

1)    relational databases optimized for transaction processing, and

2)    relational databases optimized for data warehousing.

But that has started to change dramatically.  George Gilbert, a Gigaom analyst and Co-Founder and Partner at TechAlpha, says there will be a great proliferation of databases and associated analytics over the next several years as new levels of optimization for different workloads become feasible.

As he points out, the Hadoop ecosystem is already replacing traditional data warehouses for an increasing number of situations. (He believes we won’t see the replacement of current, heavy-duty, transaction processing databases for another five years.) The Hadoop ecosystem encompasses a growing number of solutions that provide optimized analyses of various types of often lightly structured data that has been dumped into a Hadoop “reservoir”. These analytical systems are in effect complementary databases that feed into or draw from Hadoop data and, for optimal performance, can read and write on Hadoop data directly, without the need for full data extraction.

Driving this shift to new databases is the intersection of three factors:

1)    Vast volumes of new data—from social to machine generated—are increasing the new storage requirements of many enterprises by 50-100% per year.

2)    Clusters of smaller, commodity, Hadoop servers are able to store data at up to 100 times the cost efficiency of traditional Oracle, IBM, or Microsoft databases on very large-scale, database servers.

3)    The greater flexibility structuring and accessing data after it has been deposited in Hadoop than has been possible with traditional data warehouses.

In part two, we will look at examples of these proliferating databases, or analytical engines within the Hadoop ecosystem–and the problems that they are optimized to solve.

For part three, we will look at how George expects the market to shake out around this new technology—and how the mass of enterprises will ultimately tap the benefits of it.