Even the CIA is struggling to deal with the volume of real-time social data

Thanks to spy movies and other entertainment fare, we all have our own picture of what the Central Intelligence Agency is like — but the agency’s chief technology officer, Ira “Gus” Hunt, told attendees at GigaOM’s Structure:Data conference that just like any other company, the intelligence division has to somehow find the signal in an ever-increasing volume of noise. And that problem is getting harder and harder all the time.
For the CIA, Hunt said, there are three lines of “business” that are required: to collect information on America’s adversaries, to produce timely analysis, and to conduct covert action based on that analysis. All three of these jobs rely on understanding and interpreting increasing volumes of data, he said — not just from human beings and the vast quantities of social data that come from Facebook (s fb) or Twitter or YouTube (s goog), but also from devices and the growing field of smart machines and sensors.
As all of those forces combine to generate more information, the CIA’s analysts not only have to sort through it all somehow, but they need to be able to combine data in ways that they may not even know they require until the moment arrives where they need a specific kind of information, said Hunt. That means the agency needs to develop algorithms and tools that have some intelligence of their own built into them, so that analysts can sort through data in the same way they build an Excel spreadsheet.
That day isn’t here yet, the CIA official said, but it is coming. And just as Google and other companies are trying to find smarter and smarter ways of filtering the world’s information, so is the CIA, using many of the same tools and technologies — it is just doing so for a different purpose.
Check out the rest of our Structure:Data 2013 coverage here, and a video embed of the session follows below:
[youtube http://www.youtube.com/watch?v=isH8j0MPu-Y&w=560&h=315]
A transcription of the video follows on the next page