Google, Yahoo, Microsoft Team Up for Crawling

Google has brought Yahoo and Microsoft on board with a standard format, called Sitemaps, for web masters to submit their pages for spidering and indexing by search engines. In order to get quicker access to dynamic content, the companies will encourage webmasters and content management systems to place a Sitemap XML file on their servers. Publishers will only have to generate one of these feeds to get their most recent content considered for inclusion in search results across all the participating engines.

Sitemaps is going to help especially with surfacing updates from complex URLs and frequently changing pages at e-commerce sites and user-generated content networks, said product managers Vanessa Fox of Google and Tim Mayer of Yahoo. Mayer (whose full title is senior director of product management at Yahoo Search) said the Sitemaps format is currently “very simplistic,” but it could be expanded to include meta information about URLs and geo-location data.

Google had first released Sitemaps over a year ago, while Yahoo and then Microsoft signed on recently. The format is now open to other search engines as well. We gotta say, it’s pretty funny to get an email from Google to set up a partnership announcement phone call (with no more details than just that), and then dial in to hear a recorded voice say “Welcome to conferencing at Yahoo!”

Danny Sullivan writes at Search Engine Watch,

Overall, I’m thrilled. It took nearly a decade for the search engines to go from unifying around standards for blocking spidering and making page description to agreeing on the nofollow attribute for links in January 2005. A wait of nearly two years for the next unified move is a long time, but far less than 10 and progress that’s very welcomed. I applaud the three search engines for all coming together and look forward to more to come.