Diffbot brings big-time search poobah aboard to help it scale

Diffbot is a small company with a big plan: to convert gazillions of web pages into machine-readable format that can be used and reused by lots of applications. Now, it’s bringing on some serious search engine firepower to get that job done. The new hire? Matt Wells, who created Gigablast, a pioneering search engine.

Matt Wells, founder of Gigablast, joins Diffbot.

Matt Wells, founder of Gigablast, joins Diffbot.

Some background: Gigablast was one of the first search engines to do real-time web indexing. At one point, back in the in the mid 2000s, its index hit the 12 billion page mark, second only to Google(s goog) but while Google was built with lots of people and resources, Gigablast was essentially a one-man show. Gigablast “from coding to crawling to marketing, it’s all Matt Wells,” according to SearchEngineWatch.

Prior to Gigablast, Wells was at Infoseek, where he worked with Robin Li, who went off to co-found a little company named Baidu, now China’s largest search engine and Wells created Gigablast.

To accomplish Diffbot’s ambitious goal of — to put it simply — ¬†converting the worlds’ web pages into databases, it needs machine learning and web-scale architecture experience which is what Wells brings to the table, according to Diffbot CEO Mike Tung.

Wells, he said, “is probably the only person in the world that has single-handedly written a commercial full-web search engine — even Larry and Sergey were a pair,” said Tung, referring to Google co-founders Larry Page and Sergey Brin.