Big data analysis can thwart security threats

Michael Chertoff, the former head of the Department of Homeland Security under George W. Bush, said government intelligence workers have had a historical bias toward favoring the information it steals or quietly intercepts. But with the explosion of open data flowing from the Internet and social networks, the key for the government’s efforts in bulking up security is to tap this big data and mine it for the kinds of signals and traffic that yield important insights.

Speaking at a luncheon today hosted by analytics firm Opera Solutions, Chertoff said the nature of homeland threats has shifted dramatically from an old binary model, when the U.S. faced off against Russia, to a more nuanced spectrum of threats composing countries, terrorists, online hacktivists and criminal elements, both real-world and in cyberspace. That requires some new thinking, more private-public partnerships and, most importantly, better use of the data already available to the government:

When you’re dealing with a large social phenomenon, there is a huge amount of open-source data out there. If you can marry that together with sophisticated analytic tools and subject matter expertise, you can see a lot about trends and things that are developing that you’re not going to get in an intelligence report.

Chertoff said that the Arab Spring revolutions in the Middle East, which took many by surprise, could have been predicted by monitoring what people were searching for and how they were communicating online. Chertoff, who formed his own security advisory firm called the Chertoff Group, which advises Opera, said it underscores the importance of ingesting all of this online data and of working hard to understand how to use it properly to come up with real information.

The government, however, is hampered in part because it hasn’t developed the kind of framework or doctrine necessary to deal with the various threats that have emerged, Chertoff said. He has proposed a doctrine that would help guide the country as it deals with inbound attacks. The doctrine would assist government and law enforcement officials in understanding the various kinds of threats available, the vector of the attacks and the actors that might perpetrate them. And it would inform government officials how to handle and take responsibility for particular events, as well as serve as a guide for the private sector in knowing how to implement the doctrine.

This could help with some of the thornier issues around dealing with groups like Anonymous and LulzSec. Chertoff said it’s possible to be at war with these underground, leaderless groups, especially if they cross a line such as causing loss of life. But he said it takes a very fluid approach to deal with these entitites, who are hard to identify and whose threat and danger can change over time.

Chertoff said that the government isn’t bereft of talent or foresight and has a lot of bright technical minds. But he believes agencies can gain by engaging more with the private sector, something it’s already doing with Opera Solutions. But there are a thicket of older regulations, some legitimate but many outdated, that are hampering the kind of collaboration necessary for improvement. He suggested a simple cross-pollination program where companies would send a number of employees to work with the government for a year or two to help share their knowledge.

Ultimately, it takes a more comprehensive strategy that leans much more on extracting better insight from the growing amount of data available.

“We are now collecting a hugely greater amount of data even than we did 10 years ago,” Chertoff said. “Where security is headed in the 21st century, it’s about being smart, collecting information, assimilating the information and, most importantly, using the information.”