The new databases, part three (of three): market development and enterprise adoption

How will enterprise customers and database vendors respond to the proliferation of databases within a Hadoop ecosystem? As discussed in parts one and two, Gigaom analyst George Gilbert sees a confluence of forces leading to a proliferation of analytic engines functioning as specialized databases within a broadening Hadoop ecosystem.

He anticipates another couple of years of messy startups and crowded competition, on the market side, with a gap between Hadoop technical demands and IT staffing within enterprises. MapReduce and other functions in a distributed Hadoop environment are challenging and time consuming, and they often require the specialized and expensive services of data scientists to create practical utility.

Moreover, many of the added capabilities of these solutions have been developed by startup firms, which introduce the added headache of vendor proliferation and potential vendor instability to enterprises looking to garner advantages from the technology.

Yet, the business case for the new technology is compelling. Compared to traditional, relational database data warehouses, these solutions will offer 10x or greater gains in economics over traditional databases, bring more flexibility to data query, be adaptable to changing data structures over time.

Many of the proliferating solutions profiled in part two lessen have begun to lessen that burden by providing a traditional, SQL interface, dashboards or/or another simplified interface. Still, George sees user organizations as currently falling into two categories:

  • Power IT enterprises, such as major Internet, retail and banking firms are already using Hadoop to reap the rewards of Big Data.
  • Mainstream enterprises, however, need an ease-of-use buffer to make the technology practical.

He expects this impediment to be solved by three developments in the market:

  • The usual pattern of market maturation. Through a combination of acquisitions and internal development, the major traditional players in the database market (Oracle, Microsoft, and IBM) will gradually add the capabilities of the startups’ offerings to their own products. This is similar to how the ERP market developed over the last 20 years, for example. While it won’t of itself solve the ease-of-use problem for many enterprises, it will help with vendor proliferation and provide stability to the market.
  • Product maturation. The advancement of business intelligence dashboards, incorporation of more established programming languages, and other interfaces will all increase the ease of use of these products over time.
  • Systems integrators. Again like the ERP market, a good portion of the ease-of-use gap will be bridged by the systems integrators. Many of these suppliers are incorporating the new database products into the often vertical-specific, integrated applications platforms by which they have started to deliver transformative technologies to customers on an end-to-end basis.

It is up to enterprises to determine the value of new Hadoop ecosystem investments to their business—and how they are best suited to bring such capabilities into their enterprise. For mainstream organizations, the short-term options range from implementation on a targeted, high-return basis to incorporation within a comprehensive migration strategy. Additionally, they must decide what mix of internal or external resources is most appropriate for them to exploit the technology in these next few, turbulent years.