Twitter’s big problem: It still needs better filters

Things couldn’t be better at Twitter, CEO Dick Costolo told attendees at a conference on Tuesday, saying the business is going “phenomenally well,” and the company is “confident we have a hit on our hands” with its Promoted Tweets feature for advertisers. So why launch yet another new version of its Discover tab? Because even Costolo admits that Twitter needs to get better at figuring out a user’s “interest graph” and recommending topics and content to them, something the CEO says he plans to devote a lot of time to this year. And that makes sense, because services like, Zite and Prismatic are also busy trying to fill that gap.

The original launch of the Discover tab, for both the web version of Twitter and its various iOS and other apps, came last fall amid a revamp of the website — one that founder and product lead Jack Dorsey said was designed to make it easier for new users to find things that might interest them. Now, the algorithms that power the Discover tab have been upgraded with the addition of new personalization features, which look at links shared by your network and try to recommend related content. It’s also easier to see who posted links, and to reshare them from the Discover tab.

Will the new discovery features bring more users?

I was interested in this news because, like many users — including those described by Google staffer Hunter Walk in a recent blog post on discovery tools within Flickr and other networks — I rarely use the Discover tab, mostly because it always seems to be filled with spammy-looking trending topics and hashtags that I have no interest in. In my initial use of the upgraded one (which is being rolled out to all users over the next few weeks), I found things somewhat improved, but only in the sense that the obvious spam was gone. The recommended topics still were somewhat hit and miss:

To take just one example, the top link on the tab at one point on Tuesday was an article from Diner’s Journal, in which a writer answered questions about Mexican food. The link was shared by Julia Moskin, the dining reporter for the New York Times — who was described by Twitter as “people who share your interests.” I don’t follow Ms. Moskin, and I hardly ever tweet about food of any kind, Mexican or otherwise. So why is she someone whose link might interest me? Is it because I follow a lot of NYT staff, and they follow her? I don’t know. In any case, I wasn’t just somewhat uninterested in her link, I had zero interest — possibly even negative interest.

Obviously, any system based on algorithms is going to be hit and miss — especially one that must sift through the half a billion or so tweets that go streaming through Twitter every day, according to Costolo. And figuring out a user’s broader “interest graph” is no easy task at the best of times, especially when the only thing Twitter has to go on is 140 characters of text and perhaps an image now and then. Recommendation services are a little like voice-recognition, in that no one notices when you get it right but everyone hates you when you get it wrong. But more than anything else, that is what Twitter has to figure out — and soon, before someone else does it better.

Curation and filtering are the holy grail for media

As we noted at the time, the company’s acquisition of Summify was an attempt to get a leg up in this particular race — the race to curate the never-ending stream of information coming from social networks. from Betaworks is trying to be one aspect of that filter, and to create an Instagram-style social network around the news as well, a niche that Twitter would like to fill (and already does, in some ways). The CNN-owned iPad app Zite is also pursuing the curation-and-recommendation market, as is Flipboard.

Some users, like Hunter Walk, have said they’d rather see one of those services, or even the page-saving app Read It Later, instead of the Discover tab. I’ve been using a new service called Prismatic a lot to filter Twitter, and so far its algorithms have been doing a pretty good of recommending links I might want to read — substantially better than Twitter’s own filters, even though Twitter should have more info about me. And like Zite, you can help Prismatic learn and improve by voting on the stories you see, while Twitter’s recommendation engine remains somewhat of a black box.

Getting the “interest graph” right is about more than just users. Twitter needs to solve this problem for its advertisers as well, because if their promoted tweets don’t go to the right people then they will be ineffective. As I’ve tried to argue before, Twitter is a new-age media company, and as a new breed of media player it has to be the best at what new-media companies need to do to succeed — and that is curate and filter better than anyone else. The Discover improvements are nice, but there is still a long road ahead.

Post and thumbnail images courtesy of Flickr users Rosaura Ochoa and Luc Legay