Report: Hadoop in the enterprise: how to start small and grow to success

Our library of 1700 research reports is available only to our subscribers. We occasionally release ones for our larger audience to benefit from. This is one such report. If you would like access to our entire library, please subscribe here. Subscribers will have access to our 2017 editorial calendar, archived reports and video coverage from our 2016 and 2017 events.
herd of elephants
Hadoop in the enterprise: How to start small and grow to success by Paul Miller:
Hadoop-based solutions are increasingly encroaching on the traditional systems that still dominate the enterprise-IT landscape. While Hadoop has proved its worth, neither its wholesale replacement of existing systems nor the expensive and unconstrained build-out of a parallel and entirely separate IT stack make good sense for most businesses. Instead, Hadoop should normally be deployed alongside existing IT and within existing processes, workflows, and governance structures. Rather than initially embarking on a completely new project in which return on investment may prove difficult to quantify, there is value in identifying existing IT tasks that Hadoop may demonstrably perform better than the existing tools. ELT offload from the traditional enterprise data warehouse (EDW) represents one clear use case in which Hadoop typically delivers quick and measurable value, familiarizing enterprise-IT staff with the tools and their capabilities, persuading management of their demonstrable value, and laying the groundwork for more-ambitious projects to follow. This paper explores the pragmatic steps to be taken in introducing Hadoop into a traditional enterprise-IT environment, considers the best use cases for early experimentation and adoption, and discusses the ways Hadoop can then move toward mainstream deployments as part of a sustainable enterprise-IT stack.
To read the full report click here.

5 things to prepare the CIO for disruption

For years, IT organizations operated in a certain way. They provided a relatively standard service in a particular way. Of course, both of these evolved incrementally year over year. Over the past 5-10 years, that direction has changed pretty significantly. And it shows no sign of stopping anytime soon.

Data Center

10 years ago, if one said ‘death of the data center’ in a room of IT leaders, it would be seen as heresy. Today, IT leaders are actively looking for ways to ‘get out of the data center business.’ If you are one of the corporate environments not already thinking about this strategy, you are behind the curve. No longer is a physical data center a representative requirement to operate IT. Today, many options from colocation to cloud Infrastructure as a Service (IaaS) exist to replace this functionality. Not only does it exist, many solutions are already mature and more sophisticated than traditional approaches within the corporate data center.


At the other end of the spectrum, the organization is undergoing a significant shift too. Traditional organizations thought of their ‘customers’ as the internal users of the organization. The focus was predominantly on the internal operations of the company. Development may have spanned externally to partners and customers, but in specific ways. IT organizations are shifting to determine who their ‘customers’ really are. The shift in thinking starts with a change in focus. And that focus is one of the preparations.


The way IT organizations interacted with ‘customers’ was typically as two different organizations. The discussion typically included a distinction between IT and ‘the business.’ To some, this appears as an us-and-them perspective. The two were seen as very different and therefore required a different level of partnering within the company. At the same time, IT needed to clearly understand how ‘the business’ operated and at times translate between business requirements and IT deliverables. Part of the changes over time created a means to clarify this partnership. However, changes to the perspective assist with the introduction of disruptive methodologies. For example, Shadow IT, to some a threat, can become a real asset.

Changes in customers and users

Consumption expectations for customers and users changed as well. Consumers became more technologically savvy and demanded more. Overnight, consumers become familiar, and more comfortable with solutions quicker than IT organizations could adopt them. The technology available to consumers rapidly became more sophisticated. The combination of these two drove a change in consumer behaviors. Consumers, and customers became more demanding of technology…and by extension, corporate IT.

Getting ready for disruption

So, how does the CIO respond to these changes in a timely and meaningful manner? Start at the top and work down. That means, start with a business-centric approach that takes the perspective of the true customer (the company’s customer) and work your way down.

  1. Business-Centric Perspective: Change the culture and perspective to focus on a business-centric approach. Stop focusing on IT as a technology organization. The CIO needs to be a business leader that happens to have responsibility for technology. Not the other way around. Instill this change within the IT organization that is both meaningful but also helps staff adapt to the changing landscape. This will take time, but must be a mission for IT.
  2. Adopt DevOps: A fundamental premise behind DevOps is the ability for IT to work more holistically across traditional silos (applications & operations). Brining the teams together to work collaboratively and effectively is essential to the future IT organization and their customers.
  3. Stay Flexible & Responsive: Customers expect quicker response to change. Instead of building a fortress that will withstand the test of time, build one that will adapt to the changing business climate and requirements.
  4. Engage Cloud: Cloud is the single largest opportunity for IT organizations today. Plan a holistic strategy to leverage cloud in appropriate ways. For many this will look like a hybrid strategy that evolves over time versus a haphazard approach.
  5. Challenge the Status Quo: Lastly, do not assume that the way things were done in the past will work moving forward. Many organizations struggle to find success with newer methodologies because they apply past paradigms. In some ways, it is almost easier to forget the past and think about how to start from scratch. Momentum can provide some resistance, but it is healthy to challenge the status quo.

Each of these steps provides a different perspective that helps shift the thinking around IT. It starts with the CIO and involves both the IT organization and the business organizations outside of IT. Each of these five steps provides the change in perspective to evolve the IT organization and value it provides.

“The biggest hindrance to corporate growth”

IT departments are “the biggest hindrance to corporate growth”, and they need to learn from the “shadow IT” workaround of other divisions bringing SaaS solutions in directly. That’s what David Linthicum, the curator of Gigaom Research’s cloud coverage, sees in his role as an SVP at Cloud Technology Partners, a cloud consulting and implementation firm.

In his view, IT departments on one hand get in trouble by violating IT principles in their approach to the cloud. But on the other hand their lack of responsiveness is the cause for line-of-business managers bringing in SaaS products in a way that may violate the compliance, security and integration requirements of the organization.  Too often, firms know what they want and need to meet customer demand, but IT is too slow in delivering on it.

Cloud computing is the technology that enables companies to break that cycle, but it takes great skill to leverage the agility of cloud solutions within increasingly complex hybrid and multi-cloud environments.

IT department mistakes with cloud

Among the mistakes David sees IT departments make with cloud are

  1. Bypassing traditional prototype and pilot approaches, with the resulting data points factored into ROI projections—and thus ending up with unworkable solutions by the time they call in external assistance, and
  2. Not factoring in the dynamics and cost of staffing with the skill set needed to manage a cloud environment.

Sticker shock

The cost and difficulty of staffing with the needed skill set may be what most takes IT departments by surprise. Although some employees can be retrained effectively for the requirements of cloud—and cloud vendors are happy to train them on their products—David says many staffers are simply not suited for challenging cloud jobs. For both implementation and operation, new employees must either be hired or contracted for with a service provider.

A service provider may be an easy choice for the temporary role of implementations, but even operational staff can be difficult to hire. In smaller cities in the U.S., they may be difficult to find, while in larger tech centers such as Silicon Valley, Boston, or Washington, DC, they can be found but are very expensive. For those shops in markets where they are scarce, contracting with a service provider may be the way to go. Either way, the sticker shock of salary costs can confound companies that haven’t factored it into their business case.

Recommendations for “shadow IT”

With  SaaS, sales, marketing, HR or other departments can go around a slow IT department to bring the applications they need into an organization more quickly. But such side-door technology purchases can lead to breaches in legal, regulatory and security requirements. David advises departmental executives tempted to bring in their own SaaS solutions to follow these minimum rules:

  1. Tell the IT department about what is being brought in, so security and compliance concerns, at least, can be addressed;
  2. Work with the IT departments as well as possible; and
  3. Don’t hide it!

How the smart IT departments learn

David believes that smart IT departments learn from shadow IT initiatives by learning the priorities, needs, preferences and preferred delivery for technology in line-of-business departments. Proactive IT executives are enabling their non-IT peers to continue to take the lead in such decisions, while also assuring that such purchases meet the security, compliance and integration requirements of the firm.

Where are the boundaries in a rapidly changing world of business?

I had a number of interesting conversations with readers, colleagues, and vendors last week, some inspired by posts published here, particularly Shadow IT is growing because everything is IT, BYOD, and various discussions around the idea of placeforms (markeplace + platforms = placeforms). I think these are all working together to indicate a shifting of boundaries and certainties in the business, and I think it is critical to get a sense of where this is heading, both on the IT side, and for business strategy in general.

Everything in today’s business hinges on IT. There is no sector of any market where digital literacy and mastery is not critical, and in many areas, those skills and techniques are the most critical for competitive advantage. Every waitron, retail clerk, police officer, and GigaOM analyst of the near future will find their personal digital tools central to the majority of work the perform each day. Most importantly: these tools — both apps and devices — will become so embedded in our work patterns that we will no longer be able to accomplish our work without them, just like telephones were essential in 1970s business, and email was in the 1990s.

The difference is that today’s work tools are increasingly personal. They are ‘proximal’ devices: always close to hand, and they travel with us at all times: in the bus, in the rest room, watching TV, and in meetings.

As I stated in Shadow IT is growing because everything is IT,

Bring Your Own Device is really Bring Your Own Mind

These devices are a central aspect of personal productivity and identity. People want to choose these tools based on how they do their work. So BYOD should really be considered a shift of the boundary where the company’s control over the way we work — which equates to the way we think — is receding.

The takeaway from the BYOD trend is not just the company saving a great deal of money, or the furor about what can and cannot be used (The War On Dropbox). It is the strategic withdrawal of the business from one of the traditional border zones between the company and the employee: the sanctioned computing foreground.

As we have moved to an always-on world it is the average staffer who is always on, always connected. It is their hand on the tablet at 11pm on Saturday, dealing with a client emergency, or preparing for Monday’s briefing. So perhaps this is a tradeoff, a shifting of the social contract. But my bet is that in the final analysis, the decision on BYOD will be part of a larger withdrawal by the company, a part of a greater loosening and relaxing.

In a world where decision-making has to be decentralized in the company is to react in a timely way to market shifts and customer needs, what do you continue to centralize? Or, where do you decide to remain slow and resistant to change? Today, CIOs are drawing their Maginot line somewhere near  BYOD, but in a few years this will be a forgotten battle, and CIOs may be fighting for a table at the chair.

In a world where everything is IT, the CIO might start to seem like the Chief Breathing Officer. But the IT staff can’t do the breathing for everyone in the company. There is no boundary there, anymore, between the worker and their tools. And increasingly, full-time employees will want to be more like freelancers, in this regard. Every freelancers owns their own shovel, and digs their own hole. The days when a division VP would email the CIO about having a hole dug, and get back an estimate of six weeks and twenty staff days of work, well, those days are over.

There will be a very fast retreat of IT’s domain in the business, like the fall in sea level just before a tsunami.

One Certainty: The Retreat Of Business

The tsunami is the withdrawal of a great many conventional business functions in the business from direct interaction with their corresponding marketplaces. This first diagram is a (poor) attempt to characterize the traditional arrangement with functions in the business and their interaction with markets. Note that I have two-headed arrows since these functions may have already adopted social communications with the various people involved, like HR using Linkedin and Twitter communications with potential job candidates. But the shift I am talking about is post-social.


We are witnessing the emergence of placeforms that act as brokers, as intermediaries, between job candidates and possible employers (see HireArt is another placeform interceding in the broken labor market), between freelancers and client companies (oDesk, Work Markets, Elance, etc.), or even between interns and businesses seeking them. And that is just focussing on HR. The same sort of intermediates can appear in marker research, sales lead generation, and customer support (e.g., call centers).

How will that look? Where will the new boundaries be?



The business will retract from direct interaction with these markets, and individual marker participants — customers, freelancers, prospective employees, etc. — will welcome intermediation as well, and for simple reasons:

  1. These intermediaries will specialize at  narrow and deep knowledge, relationships, and technology, and all but the very largest companies will not be able to match the placeform’s ability to amass and mine big data regarding their niche. For example, a company that routinely has 20 or 30 freelancers working for it simply will not know as much about them and their relative value as a freelancer placeform company that works with hundreds of companies and tens of thousands of freelancers.
  2. The intermediaries will be able to scale their operations — based on working with large numbers of clients and large numbers of individuals in the market — in a way that is not possible for other market participants. So, they will be able to make serious profits while market participants benefit financially. For example, a company using a placeform to manage call centers will spend less and get better results, because a specialized placeform with scale advantages will be better able to match workers with the type of support needed, and can invest more in training, for example.
  3. The scaling of the intermediaries acts as a check on the information imbalance in many company-individual interactions. For example, a freelancer dealing directly with a company is unlikely to be able to determine if that company has a good track record of on-time payments to freelancers, or whether other freelancers believe the company is a fair partner. However, a placeform company can amass that information, and actually influence companies’ behavior by threatening sanctions, like closing their access to  its freelancers.

Businesses will still use conventional branding and marketing to get their story across, but the transition to marketing through social platforms — Facebook, Pinterest, Tumblr, Twitter — will accelerate, and increasingly those ‘social networks’ will be evolving into placeforms themselves, providing tools and programs to better position the company in a social mediated world. As these companies do more, the corresponding functions in the business are pulling back, and relying more on the data analysis and trend spotting of outsiders to help guide their direction.

So, on two major fronts, business is pulling back, redrawing the boundaries at the edge, and reconsidering where the edge should be. This is happening in parallel with the general loosening of ties within the business, and the overall increase of connection: the fast-and-loose business that I have been discussing here the past several months. The connections between the core of the business and the intermediaries that help implement the companies objectives will continue to grow, so along with the retreat from the direct market, the company is also advancing into the market through its intermediaries. A business working with oDesk, for example, is privy to more information about the freelance market than it could ever amass independently: it is made smarter by the sum of its connections.

The Final Word

So, the final word is that the enterprise is retreating form many areas it once considered central, and its is handing over a great deal to employees and to placeforms. This will make businesses leaner, faster, smarter, and looser.

BYOD blowback drives more IT underground

Enterprises unnerved by the bring-your-own-device movement that many had promoted are now trying to lock down employees’ own devices for security purposes. The unintended consequence is that many of those employees, frustrated by these restrictions, just use unsanctioned devices instead.

Move to cloud makes DevOps even more important

As companies move to the cloud, DevOps — the practice where developers work with the operations side of the house — becomes more important. That collaboration could lead to more satisfying IT implementations and the best — and sanctioned — use of cloud resources.