Pinterest bought Kosei because recommendations are really hard

Pinterest announced Wednesday that it has acquired Kosei, a Palo Alto, California-based startup that focuses on machine learning for product recommendations. It’s a smart buy for Pinterest because the company’s path to profitability depends on its ability to connect users, products, and the companies or people selling them.

Here’s how Pinterest explains the acquisition in a blog post:

Over the past year, Kosei has been building a unique technology stack that drives commerce by making highly personalized and powerful product recommendations, as well as creating a system that contains more than 400 million relationships between products. As we build a discovery engine for all objects, Kosei is a perfect fit for our team.

. . .

As people use Pinterest to save and discover the things they want to do in the future, we have a unique and growing data set of more than 30 billion Pins that will only get more powerful over time. With the addition of the Kosei team, we can supercharge our existing graph to help brands reach people at the right moments, and improve content for Pinners.

For Pinterest, as the post goes on to note, the Kosei team will add to several other machine-learning-based teams at Pinterest, which are responsible for everything from spam detection to deep-learning-based object recognition (via its Visual Graph acquisition in early 2014). Kosei joins an existing “Discovery” team that’s already working on recommendations and user-behavior models.

Pinterest's guided search feature

Pinterest’s guided search feature

But the bigger picture here (and something several speakers will no doubt cover at our Structure Data conference in March) is that, despite years of effort by companies such as [company]Amazon[/company] and [company]Netflix[/company], recommendations — a driving factor behind the entire big data and data science movement — are far from a solved problem. Data science teams at those companies, as well as at places such as [company]Facebook[/company], [company]Google[/company], [company]LinkedIn[/company] and [company]Twitter[/company], are always testing out new variables and tweaking their models in an effort to put the right content — ads, users or otherwise — in front of the right people.

And they have some of the smartest people and most-advanced systems around. For laypersons and smaller companies, recommendations can be a much more daunting task, although there are now startups, open source projects and other efforts trying to address the situation.

As long as the web continues to be a hub for our shopping, education, socializing and media consumption, companies will strive to personalize it the names of user experience and revenue. Which means they’ll also keep pumping money into the graphs, models and algorithms that make personalization possible.

Twitter open sourced a recommendation algorithm for massive datasets

Twitter recently open sourced an algorithm designed to ease the process of running recommendation engines at large scale. Called DIMSUM, the algorithm pre-processes pairs of possible matches so the other algorithms in the process don’t waste resources on poor choices.

GraphLab thinks its new software can democratize machine learning

A Seattle-based machine learning startup called GraphLab is releasing the first official version of its software, which the company hopes can democratize an historically difficult space. Called Create, the software is focused on simplicity, speed and being able to handle a wide variety of applications.

Boostable is putting machine learning to work for marketplace sellers

A startup called Boostable is helping individual sellers in large marketplaces like Etsy and Airbnb target potential customers on sites such as Facebook. It’s yet another company trying to package up some advanced algorithms for users with little or no IT budget.

Spotify acquires The Echo Nest and its musical smarts

Streaming music service Spotify has acquired The Echo Nest and its graph of musical data spanning more than 35 million songs and 2 million artists. It’s an easy way for Spotify to match companies like Google and Pandora on the data science front.

Personalized e-retail tech gets Reflektion $8M from Intel, Nike

An e-commerce startup called Reflektion has raised an $8 million series B round of venture capital for its technology that helps retailers personalize the online shopping experience for consumers. Intel Capital led the round, and Nike and several private investors also pitched in. This seems like the latest thing in marketing — not just targeted advertising but entire tailored experiences for individual shoppers. It does add an Amazon-like recommendation experience, although one might fairly question whether most product catalogs are large enough to warrant it.

On the path to personalization

This post from the New York Times‘ Open blog talks about the architecture and algorithms underpinning its content-personalization engine. Its experience speaks to some larger trends around companies moving from batch to stream processing and to cloud services overall. The Times’ recommendation engine used to rely on MapReduce jobs that ran every 15 minutes, but now relies on a homegrown real-time system. It used to run on Cassandra, but now runs on Amazon’s DynamoDB service.