Scaling lessons from Google’s CIO

“I’m here to share a little disaster porn with you.”

And with that comment, Google (s goog) CIO Ben Fried kicked off his keynote at the Surge conference and shared two essential lessons for folks trying to build systems at scale. Fried’s keynote went beyond sharing his failures, though, all in an effort to help developers realize that the entire culture of IT must change.

The Industrial Age, when specialization ruled, is over, he said, and the focus should now be on having an end-to-end understanding of systems. Generalists, not specialists, are what matter when folks are trying to push the boundaries of what is possible. “We scaled up to deal with our own success and the way we scaled up was through specialization … but understanding how everything works is really important,” Fried said.

Fried described a time before his role at Google, when he led the development of a trading platform for Morgan Stanley’s high-value clients, and the series of mistakes the team made. But as he deconstructed what went wrong, he realized that one of the biggest issues was that until the product fell apart, the many, many people who helped build it had never been all in the same room before and had no real idea how the system was supposed to work. That was lesson one.

The second lesson was his realization that he and his team thought they could use smart software and some impressive infrastructure to deliver an application that would seem like a desktop app, over the less reliable and untrusted network that is the public Internet. “We made fundamentally unsolvable problems seem like solved problems,” he said. And his team believed it, until it all started unraveling.

Taken together, Fried explained that developers need to create a new culture built on general knowledge and realization that when you are building at scale, you are at the edge of the known world. Much like you would pack differently for a cruise than for an arctic exploration, those building at scale need to realize that they aren’t on the Loveboat and are instead like Admiral Byrd aiming for the North Pole. Explorers need to understand something about a lot of things so they have a sense of who to call and what to do when problems arise.

In the question and answer period that followed, Fried elaborated on these concepts, telling someone that IT generalists are probably born, not made. He said at Google, the company looks for folks that want to keep improving their skills, and even has a program to help give those people the tools to be better engineers when they find those traits in employees. He said the Google culture is one where the general engineers who understand the system have a lot of input and power, which is a cultural shift that organizations that want to build at scale should try to implement.

Finally, he emphasized that while change may be the root of all evil for a developer trying to build out complex web systems, it’s also inevitable and the reason they all have a job. Developers must expect change and adapt to it rather than assume that’s a solvable problem. The session went from disaster porn to a defining philosophy of the new developer culture, one that turns its back on the Industrial Age to embrace the webscale age.