10 ways big data changes everything

Big data is now your company’s virtual assistant

By Bobbie Johnson
Big data is empowering an entire generation of smart businesses to gain fresh insights into their customers and build new products. But it can also revolutionize the way a company looks at itself, too.
That’s the premise of Autopilot, the flagship product from Frankfurt-based automation expert Arago. The system — a virtual assistant of sorts — uses a combination of data and artificial intelligence to take over the most boring and repetitive tasks of managing a large IT infrastructure, effectively becoming a new member of a sysadmin team.
After Autopilot is given access to the streams of information being logged by your servers and is taught about common problems your administrators encounter, it can use what it knows to ensure your services run smoothly.
For example, say a monitoring system reports a potential problem: An error is being thrown out by one of the company’s Oracle database nodes, which seems to have a nearly full tablespace. Autopilot picks up the ticket immediately, connects to the server, examines the database connections, retrieves data about the tablespace, and then adjusts it or reorganizes the database storage to resolve the problem. It has taken immediate action and completed its task in seconds, spitting out a full record of what it has done.
“People are used to tools, but this is more like a colleague,” says Chris Boos, the company’s CEO. “It writes tickets, it closes them, it asks you for help.”
The difference between this and other systems is that it is based on algorithms that learn how a problem gets resolved; it’s not just performing workflows that somebody else has input into it. That means it actually makes decisions for itself, interrogating previous data in order to understand and learn what’s happening now and what it can do to help.
Boos admits this prospect can be terrifying for some people who see Autopilot as a threat, and it can seem outlandish to others who struggle to believe it can make a substantial impact.
But the results can be significant. One commercial implementation of the software — which runs across a string of virtual machines at Amazon or VMware and sends data back to Arago’s own cloud-based cluster — achieves a 93 percent level of automation for IT administration.
“That means something previously done with 70 people is now done with 8,” says Boos. “And that allows those people to go and tackle more interesting problems, rather than deal with boring and repetitive tasks.”
But Autopilot is not just allowing Arago to help companies free up their talent from tedium. It is also helping Arago turn big data back on itself.
Now that the system is learning what happens inside large IT deployments, the company is starting to explore what that means, whether that is understanding the common problems faced by particular setups, helping CIOs avoid expensive mistakes, or simply giving customers the chance to benchmark their infrastructure costs.
“Since we collect all of this data on how IT systems run, we’ve started to think about how we can use it,” says Boos. “It turns out we can basically predict how much effort you have to put in to operate an environment, all without knowing anything specific — all we need is a very lightweight model of your system.”
All of this comes despite the fact that Boos doesn’t really think of his business as a big data company.
Sure, it uses Cassandra and Hadoop — what he refers to as “all that big data stuff” — but for him, the secret is in the artificial intelligence. The data is essentially a tool that helps people make better, more informed decisions. But it is a means to an end that can change not only the way you operate but also the products you offer.