Content personalization still has a long way to go

[protected-iframe id=”fd915e4d2c43b51d2f72ccec46aa1f7d-33319749-25766478″ info=”″ width=”640″ height=”360″ frameborder=”0″ scrolling=”no”]
Session: The Impact of Personalization and Algorithms on the Attention Economy

Speakers: Chris Albrecht Ernie Sander Aria Haghighi Mark Johnson

Chris Albrecht 00:03

Okay you may not have heard it, but there is [chuckles] personalization in media. And now I’m excited to bring on the panel, that’s going to be talking about the impact of personalization and algorithms on the attention economy. It’s going to be moderated by Ernie Sander; he is the executive editor of GigaOM. And he is going to be speaking with Aria Haghighi, Cofounder, and CTO of Prismatic. And Mark Johnson, CEO of Zite. Please welcome our next panel.


Ernie Sander 00:38

Mark things for coming, I’m super interested in this topic as Chris suggested. Think there’s some people that this is the future of personalization is the future of content and modernization. And so it’s a great topic and there are not only a lot of new age players – and you guys are two of them. But companies like Google and Facebook and Amazon and Apple are also poking around in that area. There’s interesting privacy issues, but also a lot of really exciting consumer interests in making this all work out. I’m curious to pick your brains about it. Matthew and I were kind of joking earlier about the newspaper days when you get four-section paper and you would toss away the three sections that you weren’t interested in. And obviously personalization allows us to really laser in on what we want, and not have those three sections that we just throw out every day. I’m curious to talk to you guys about where we are in terms of figuring that out. I wanted to start by just asking you guys whether–when somebody says personalization, to me, that means a combination of stuff of I know I want and stuff that I don’t know that I want, but I do want. But obviously there’s complex algorithms that go into finding this pieces of content and I’m curious everybody in this space kind of agree about what the factors are they go into the perfect personalization without getting knee-deep into the algorithms themselves? The inputs and the waiting, can you talk a little bit about that, are there philosophical differences?

Aria Haghighi 02:14

It’s not that there’s worked out philosophical differences. I think it’s – personalization is really young. I still think we don’t all agree necessarily on what personalization means. I agree with your definition, that it’s a lot of– their stuff that you see, and you want to only see this stuff that you care about. The whole pool of other thing that you might care about, that you’re not seeing. I think that’s pretty reasonable. Some people think it’s only getting signal from noise, or whatever. I think were still really early, and I don’t think we even matured to the point where we have complete philosophical differences yet, because I think we just mean way too many. We had way too many things, and no one speaking concrete really about what it means to do personalization. And frankly to me, I come from a technical background; I’m a PhD in computer science. And so for me I think of algorithm as basically operationalization of definitions of personalization, so for me algorithm that we use is very opinionated. And it’s very opinionated in a way that doesn’t normally enter into the public discourse. I think when two people think personalization they almost certainly means something different.

Mark Johnson 03:17

You might see this kind of as the prequel or–surely people here remember the Excites and AltaVista of the world, and they were really crappy. In the Google came along, it was really amazing. I think, Prismatic, Zite are really good at finding you stuff, but we can be so much better. And I think that we still haven’t had that Google moment where people realize, ‘ah ha’ that’s what personalization is, that’s what the Google paper really was,’ oh that’s what web search is, that’s how you make a good’

Ernie Sander 03:45

So was between us and there?

Aria Haghighi 03:47

Lots of research and refinement, both in terms of stuff as well as getting a deeper understanding, and looking that up with user research. Even the Prismatic– I run a lot of the relevance that goes on there and every– by just looking numerically at the numbers of how kind of our metrics are increasing for all of this. I know were nowhere near the cap just because it’s too easy to gain points and the probability that a user clicks or interacts with the first article is climbing up so steadily that I know where nowhere near the ceiling of what we could be doing.

Ernie Sander 04:17

You talk about metrics, how do you measure yourself, where are you trying to get to in terms of relevance how fast are you getting there, that’s an interesting question. How do you measure yourself?

Aria Haghighi 04:27

Yeah I think the way that you measure personalization’s important. It says a lot about what you think, the problem is obviously. So a lot of the simple metrics look a lot like what search engines do. They look at what are the ranks – did you click on the first search result or did you IE, did you share, or read the first story our first piece of content? But then there’s also for personalization things are slightly different where part of our goal is to push people into say, new areas, and new topics. And so were looking at things like – did somebody who’s never read a story about interior design, read about interior design. Again, the metrics and code a lot of what you think personalization should be doing, and so that’s a mix of finding people interested contests, like you suggested. And even more kind of metric points were saying, we pushed you into a new area and we’ve got you out of your local maximum.

Ernie Sander 05:13

So the interior design point, you’re both interested in getting them to read interior design articles, but also interested in them telling you they don’t want those. That’s a piece of feedback can use or one or the other?

Aria Haghighi 05:24

Yeah, and both of those are kind of baked into the numbers that we look at internally when we judge all of the relevance machinery.

Mark Johnson 05:30

I don’t think one of the challenges here is that in the search world the interface kind of dictates how you measure, right? There’s a ranked ordered list of 10 results on the page, and if you– most people click on first, if you were to click on the second and so on. And one of our challenges I think in general as an industry and figuring out what metrics are good for personalization. Everybody’s app looks different, everybody’s page is different. So it’s hard to have generalized user metrics that say is good or that’s bad. It’s really a feeling of people – although to use Google as an example again, people just kind of knew it when you used AltaVista and then finally use Google, you’re like, it gets me the right result more often. I think that’s often the feeling with personalization where people don’t know why a site is better, they just say, Oh, it gets me lots of stuff I care about reading.

Ernie Sander 06:18

I have this image if I’m sitting on a train and everybody on the train has their Zita, Prismatic app open that everybody’s got a totally different– or largely different feed. Is that true how much variation – differentiation is there and people’s personalization? Like Avin Ava is everybody getting politics and sports and technology, and then with a couple subtopics or is there massive variation? It’s just you guys now have a lot of customization where you can add subtopics and you can add feeds from a whole bunch of different places. And so I would imagine there’s the potential for real variation.

Mark Johnson 06:52

It’s shockingly different. A few data points one is that of all the publishers that we have only a few are in single digit percentage points in terms of clicks. That curve is very, very, very– the long tail is extreme along with then Zite. Another point that I thought was really interesting is our CTO Mike Klass who’s been working on this for seven years. We were going to look at our two technology sections, he’s like, dude, “We care about the same stuff, they’re probably going to be identical.” And one day we flip through them, and we were like, wow, there’s– we share to articles. And they weren’t actually even shared articles; it was about a topic we both cared about, but from different publications. And even he, the guy who created the technology was shocked at how different our technology sections were.

Aria Haghighi 07:40

Yeah. We share the same thing, they look radically different. A topic like science fiction for instance, some people only care about the books and kind of literacy aspects. And some people care about movie aspects and some people care about the big push in small publishers. And speaking to the point, the other interesting thing is, I noticed is how much traffic is going to kind of smaller publishers that might not otherwise I think get it. And so a lot of these small blogs that are publishing that kind of need to interest, which people in the Prismatic community have – they’re getting a lot more play than they might otherwise. And that actually feels really great, that yeah, almost no people’s feeds have no– no two peoples have very similar feeds at all.

Mark Johnson 08:19

I think that it’s almost like we have to do that as personalization technologies and personalize magazines, because if you want to get headline news you go to CNN, if you want technology news you go to GigaOM. But if you just want to see really interesting articles that are on the fringes of the Internet that you had no idea that existed, were that you wanted. That’s really I think one of the core challenges of personalization.

Aria Haghighi 08:46

Then on the flip side of it natural question to ask, after that is to say, ” What about the sense of community, now that everyone’s looking at something slightly different.” I think one of the things that you can do when you kind of look at people’s interests and you’re willing to personalize and now say, “Yeah, I can’t guarantee that the person next to you is looking at the same feed as you.” But now you’re hooked up to all these people who share all these fringe interests don’t share with your friends. And you guys can have this simplistic community of people who are interested in science fiction, or interior design. And they can be interacting on the same article. So I think it actually– while it decreases the chance with that any particular or the random person you’ve got a lot in common, it can actually do a better job of hooking you up with people with whom you share a lot more common interests.

Ernie Sander 09:26

That’s an interesting question. We all tend to get conditioned to place a high premium on sharing and yet personalization leads you more into yourself, I think. Do you see a lot of sharing, is that even important? Couldn’t I be a totally satisfied reader, consumer, and not share much at all? If it’s really niche targeted to my interests and even my social raft, because it’s so targeted might have different interests.

Mark Johnson 09:47

I would love to know if there are other people interested in astronomy, taxidermy in San Francisco. It’s kind of a rare group, I may be the only one, but if there someone else in San Francisco that shares those interests I want to know they are. I think there’s other magazines out there that build themselves as a social magazine. And I think that I social still hasn’t been nailed in this world. There’s something special about that graft of people that you don’t quite know, but who share a lot of interest as yours. I think sharing it’s just kind of the tip of the iceberg. Okay, fine I want to share interesting things from my Zite with my twitter feed. That’s kind of the basic level, but I think there’s something deeper you can do social within these personalized applications that no one yet has nailed. And I think that the next 12 months are going to show a lot about.

Ernie Sander 10:42

And what would that look like, take me through scenario. Is it you finding the other person who has those three interests easily, or is it– what would you like to be able to do that you can do now?

Mark Johnson 10:51

I think that there’s kind of conversations around affinity groups that– one particular feature that I would love to see, it’s conversations around affinity groups. Again, there’s people who care about certain things and you kind of want to have conversations with like-minded people, even if you don’t know them, really well I think. That’s certainly one feature that I’d like to see. Another one is just what are people that I care about reading? If you look at your Twitter feed, or your Facebook feed right now– I mean, there’s a reason that you have to separate your Twitter feed into people I really care about. I care about everything my mother posts on Twitter. Whereas just people I follow randomly, I don’t care as much. I think that there’s a lot to be done there around just figuring out what are my friends, reading that I really care about?

Aria Haghighi 11:38

I think that’s absolutely right, like I alluded to in the last question. I think the notion of a complicit community around topics and interest groups are really, I think the way that social makes sense with personalization. And if you think about it what happens in Prismatic, and I imagine Zite as well. There’s this unique thing when I just recommend click, or do anything with an article. I’ve actually benefited implicitly tons of people around me in a soft way. People I’ve never met before, I’ve kind of made their consumption experience a lot better, just merely by reading an article. Other people are interested in science fiction or taxidermy who have dead bears on their floor, or whatever. So other people–

Ernie Sander 12:13

So you’ve been to Mark’s house?

Aria Haghighi 12:14

Yeah, I’ve got a telescope yeah [exact later?]. There’s this powerful thing that happens when you have a social component of personalization, which currently exists on the backend Prismatic. Everyone’s actions are contributing and benefiting the lives of others. And surfacing that to the top so that you understand when you actually taken action, you’ve done something for a ton of people you don’t even know. You’ve done something for the implicit community of interior design people, and having conversations that. That’s definitely coming very near future, at least at Prismatic and I imagine it’s kind of the direction everyone sees is probably the right one go into.

Mark Johnson 12:47

What was interesting, I think that you need some kind of technical foundation to enable some of these next-generation social features. Again, looking at some of the competition out there, they are really pretty RSS feeds. But you need to have this underpinning of technology, you need to understand the user is what they care about who their connected to. And have done some analysis to know what’s most important among this delouse of information that they get. That’s why where up here is that technology is really driving this, even though both applications have really nice user interfaces and we spent a lot of time thinking about what the interface looks like. You’ve got to have the technology underpinning.

Ernie Sander 13:33

You’re sort of talking about feedback loops here, which is interesting. Are you getting enough feedback from your users? I guess there’s never enough, but are you getting a lot, obviously, what you’re trying to do is create a line of thumbs-up, thumbs down kinds of signals from people. Seeing you need a critical mass of that in order to create [crosstalk]

Aria Haghighi 13:52

The way to think about how you do artificial intelligence and you mix UX and learning– you learn from behaviors that users naturally take within an application. And anyone who builds good applications you instrument the hell out of the thing. For us, even the thumbs-up, thumbs down thing. I don’t necessarily think are great ideas. Anytime you call attention– anytime you draw attention to a user that they’re making a decision that might affect classifier, and that action has no bearing on the rest of the application, sort of like Pandora’s thumbs-up, thumbs down. I actually think that’s a bad UX call. You should only be learning from the pieces of the application that makes sense, a similar example, say, in Facebook. Facebook does a lot machine learning based upon your likes. The Facebook news feed is actually the most consumed machine learning industry in the world, bar none. But unlike has meaning within their product that has nothing to do with kind of whether or not there happens to be learning behind the scene. To answer your question we get enough feedback, because it’s an application and we just instrumental the things that a user my normally do. And I think the right way to set up your machine learning is to actually learn from those natural behaviors that make sense in the application.

Mark Johnson 14:54

I think your point about feedback loops is really on point. A good friend of mine, Dr. Paul Pangaro was a cybernetician. And cybernetics is really the study of feedback loops and I think that these applications are in a unique position where we people want to have a conversation with them. And I agree with you that thumbs up and thumbs down our kind of the sledgehammer you use to say, ” Okay, do you like it or not.” But people want to give you more information they want to say hey, Hey, I like only this author from this publication not the publication in general. Give me less of this publication only when it’s really important, and I think those conversations difficult to have. They kind of look like power user features and I think it’s a challenge to us in general is we’ve got this deep technology that understands your likes and interests underneath. How do you converse with that in a way that’s really effective in generating you the kind of magazine that you want to come back to every day?

Ernie Sander 15:50

Is tough thing designing the system or getting the people to participate in it? Because obviously is like having a focus group, if you do a survey, you ask people 10 questions you get those 10 people in a room, you asked them another 30 questions you get a lot more information. Is the problem that you can’t get the people to do the focus group or it’s hard to design a focus group of people that are ready to play?

Mark Johnson 16:15

I think it’s a sink question. There is no private language you have to– to have a feedback loop. You have to have someone want to participate in it, and they have to see what the effect is. A conversation really is about, “Okay, I took this action and now this other actions happening.” This is how we’re having a conversation right now. So I think the design inherently has to have a few of those features, it has to be clear what’s going to happen, and it has to show you what’s happened. And it turns out that’s really hard to do in machine learning, because there’s a lot of stuff happening underneath that even a really smart guy with a PhD in computer science doesn’t necessarily know – it’s kind of a black box to you sometimes.

Aria Haghighi 16:57

That’s one of the other things that I think are challenging. What Mark’s talking about is right. How do you do mechanism design and that’s touching on AI and product design. And that’s an important issue. Another one is the black box issue, even if you do the world’s best job recommending articles or context to people, if they don’t have an limitation of why, people just feel uneasy. So for instance Netflix’s spends quite a lot of time engineering those things. Because you like light romantic comedies, or because you like cerebral sci-fi, definitely not the two categories I get those frequently. Yeah, I think you have to just engineer enough time to surface parts of the system to give an explanation to a person, otherwise it just doesn’t work. And that’s kind of place where user responses intersects with AI design. You have to build those things and pending foundation right. Kind of what I think what is unique about Prismatic is old AI researchers, including myself, are super interested in these kind of factors. What is it about AI products that turn people off? There are things like there’s explicit of a feedback mechanism with this thumbs-up and thumbs down. It is to black box, there’s no explanation to why the hell I should see this thing. Am I looking at this, because the author’s one that I like or the topics are one that I like? And surfacing those reasons is part of that conversation that I think we’re talking about.

Ernie Sander 18:08

Should publishers embrace you guys and love you guys scared by you guys. Are you creating an ecosystem that exists out where you grab the content, create a great reading experiences for users– super personalized reading experiences, everybody spends a ton of time on your platforms and your apps, or is there some win-win here for publishers if you guys figure this out?

Aria Haghighi 18:34

They should love us. Thing is here look– the problem that I think we’re both addressing– and we’ve just been talking about news. But, of course Prismatic interested in a broader range of things. Is the discovery problem, there’s tons of things out there give me the things that I love. And from a publisher’s point of view, that’s also just called distribution. The thing that’s unique about Mark and I’s apps is that were not just a place you go to say, “I like the New York Times, please give me the New York Times.” Or you already have a declared interest in it. We’re the people who are pushing New York Times articles to people who may not be reading them at all. The kind of the goal of our accessory readers of the world, they’re not necessarily grow the pie for publishers, they’re not going to help the smaller publisher connect up with an interest community or something like that. When you’re us and your business is knowing people’s interest and understanding what they want to be reading, and part of the kick of the personalization is I want to see something news, I would think.

Ernie Sander 19:31

Do publishers get their data back to find out that there’s– they have a New York Times as a massive following in astronomy. Does that go back to them–

Aria Haghighi 19:41

Yeah, yeah. For at least me I read a lot of comic books news. I think USA Today comic book articles; I’ve never read USA Today and like that gate or whatever. But I actually consistently read their comic book content, because they actually have quite good comic book content, they got in the customer. I definitely think we should be– and there’s a revenue model– there’s definitely a revenue story that goes along with this, that there’s probably not necessarily time go into.

Ernie Sander 20:07

We can make some time for that. I was actually going to ask about that. Let’s talk about the revenue model, if the revenue model that–Mark you’re getting into advertising now. Is the revenue model that you become a content platform supported by advertising, for example, is it different – is there a subscription fee, is there publisher fee? Talk a little bit about how you make money here.

Mark Johnson 20:34

One of the challenges with these apps is that were innovating in several areas simultaneously. We have any method of delivery and mobile. We had a new idea of personalization and aggregation, and I think if you take the old model of just straight up advertising, people are going to reject it. When think I think about in Zite a lot is that so used to Zite knowing your interests really well – if we give you really that ads don’t aligned to your interests going to some is going haywire Zite. We think a lot about this and what we found is that, yes we’re actually owned by CNN now. We’ve gotten to the point and the maturity of the business where it’s time to start thinking about making money. And what we found is that there are a lot of brands out there that are looking to do really innovative interesting things. As we sit down with them, even though the conversation starts off with, okay, well can we do [inaudible], okay fine. But then after lunch in they’ve realized that there’s some deep technology and that were willing to work with them and innovate. I think that’s a really powerful combination, because I remember I was at this conference last year on this stage. Someone asked a question I’ll never forget they said, Dollars and print are turning to dimes in digital and are going to be in mobile. And I’ve never forgotten that, and it terrifies me. We should all – everybody in this room should be terrified about that.

Aria Haghighi 21:57

The 30 second version of this is discovery is the new utility of the Internet, so discovering, and personalization, whatever you want to call it, is the new utility of the Internet, much like searches. And search had a native revenue model associated with it. There was kind of two sides of a search business, there’s the side of the utility itself going to a search engine as well as what actually happens on the publisher end, the idea the ads and all of these things. And I think discovery, like Mark alluded to we’re innovating a lot of fronts. And I think there is a native revenue model for discovery. That native revenue model is direct commerce. The reason why wanted to make a Prismatic with Bradford Cross is that, I used to go to– I used to be one of these music discovery junkies. I would go to NPR, I would hear about new album and I sure as hell would buy. Discovery was associated with taking some kind of action. I did find some awesome review of a book that I’d never heard of before by some fringe science fiction author, I would buy that book. So I think discovery and serendipity are associated with taking new actions, find in an article about some local museum that you haven’t been to before – having an exhibit in one of your interest areas. I don’t normally go to the muma, but if there’s a video game exhibit at the muma I’ll probably go. There are actions that are naturally associated with it, there’s commenced that’s naturally associated each of these actions. Prismatics – I won’t say long-term short to medium-term vision is actually doing recommended feeds of things like apps, like TV shows, like podcasts. I think the natural revenue model with that kind of discovery utility is actually just directly selling people things. I think it’ll make everyone happier. Everyone knows what’s happening to display advertising and content-based advertising; it’s not doing so well. So on direct commence is the only sustainable– the only way to hurdle over the drop in prices for ads.

Ernie Sander 23:46

Let me ask one more quick thing and see if there’s a question. You mentioned that you guys have aspirations well beyond content.

Aria Haghighi 23:52

They’re more beyond aspirations at this point, it’s a prototype, so yeah.

Ernie Sander 23:56

You want to be the recommendation engine for people’s lives I think, right?

Aria Haghighi 23:58


Aria Haghighi 23:59

And kind of scotch to buy, what trips to take, what restaurants go to. That would seem to bring you into a collision course with a lot of big, established companies that also do– people like– I mentioned them earlier, Amazon and Netflix and Google and Facebook and all. So how do you win that battle?

Aria Haghighi 24:19

Companies are more or less limited what comes with the DNA – more or less what comes limited to the DNA of the founders. Google’s really good at search because they had to information retrieval researchers that founded the company. The more they stray from a lot of these core things – they can occasionally do a good job. But on average, the thing that they relate–

Ernie Sander 24:41

Take companies like Amazon and Netflix that are very much– you talked about in starting back in the ’90s. They’re very much built on personalization.

Aria Haghighi 24:48

They don’t have that, so are people with whom we would actually work with. Amazon does an affiliate program where they encourage everyone in the world to go and manually do this kind of discovery for them. It’s not competitive with the fact Amazon on their home page is going to want to recommend items to people. This is something that they don’t necessarily want to be doing; it’s just that they have to do it to help themselves out. I don’t think Apple – their hearts really in the Genius tab on say–

Ernie Sander 25:17

You’d help them be a better Amazon?

Aria Haghighi 25:18

Yeah. Discovery is its own unique utility, and everyone is going to one in various channels. People of course are going to do it within their own space, but that doesn’t mean that there’s not a wider world in which they would want to have discovery within their own commence stores. So I definitely don’t see these aspirations as in conflict with any of those players. I think the contrary, Netflix want someone who reason article about a director to go watch the Netflix movie, that that directors made on their site. That’s not something they’re doing natively. So I definitely don’t see it as competitive.

Mark Johnson 25:49

I’ve also add that I don’t think a lot of the– either the problem is fairly constrained and easy. I’ll let Netflix – not to take anything away from them, but you’re dealing with a corpus of 20, 30, 000 movies as opposed to 2 million articles written a day. There’s more constrained simple or as opposed to generalized discovery problems. The other thing is, there are a lot of companies as seen as recommendation companies who aren’t very good. Amazon still recommends me the most bizarre and ridiculous things, when I come to the home page. I bought vacuum cleaner for a friend’s mother five years ago, and I still see Hoover bags on the homepage. It’s like why, why?

Aria Haghighi 26:32

Beyond the quality I think [laughter] just this– beyond just the quality of say like Netflix, I think there’s a lot to be had in getting discovery cross content type that you’re dealing with. The thing I suggested where I read an article about someone in– all of a sudden there’s an event– events probably going to do own recommendation. But they’re not going to know that I just watched a video within Prismatic about something, and now maybe this event makes sense. Discovery is one of these things that were talking about a person, and a person does lots of things. They don’t just– when I buy an app in an app store that might have implications for where I want to eat. That’s why discovery is this core utility to a person’s life, and there’s going to be a company, of yes, that is that one stop discovery destination. And that’s going to be what the company does, it’s going to be what the founders were interested in– it’s going to be a focus. It’s not something you grow want to existing company

Ernie Sander 27:26

Any questions? I have one more, if nobody else has– let’s talk about filter bubble for a second. So there is this notion that you get somebody so personalize that you near their interests rather than expanding their interest. Is that something we should be concerned about? Whole books have been written about it, how do you guard against that. If you know them so well, and you keep serving up exactly recipe of stuff that they’ve told you they want – if it’s three or four things or whatever, how do they ever read the three other things that the opposition might say, or that there, critics might say, or…?

Mark Johnson 28:03

I would say Fox news is a filter bubble. MSNBC is a filter bubble and that our job is to actually break people out of that editorial. I think you said it earlier is that when discovery really works you find stuff that you didn’t even know you cared about. That kind of serendipity has to be baked into the core to any great discovery engine, and as long as we do that we will show people– one of my favorite articles I ever read Zita is something I disagreed with everything in it, and I still gave it a thumbs up, because I was so glad I read it.

Ernie Sander 28:39

Any final thoughts?

Aria Haghighi 28:40

I think, actually I– I agree with Mark.

Ernie Sander 28:43

Did you read that same article in love it?

Aria Haghighi 28:45

No. But you can show it to me later. No, yeah, I think the questions kind of wrong. Filter bubble is just a term that people use when they interact with crappy personalization frankly. A lot of people have been conditioned to hear the personalization and see a lot of bad stuff, and rationalize, “Oh well’ is just giving me more of a– yeah, it’s just bad”. This is why the term personalization kind of bad I think, it’s like you know that’s just bad personalization, it’s what you like and if you happen to not like seeing the same thing and you want serendipity, as we all do to some extent. I have numerical data that supports this within the Prismatic, then yeah, you’ve done a bad job.

Ernie Sander 29:16

When I’m checking box– when I’m adding things to my feed that I want, aren’t I telling you exactly what I want? How much of my feed is serendipitous, and how much of it is exactly what I’ve told you I want?

Aria Haghighi 29:29

It depends on how much you seem to be interacting or skipping things we think you like. So there’s an audit– there’s explore exploit dynamic that’s always at work, that’s looking at– there’s always a diversification component. And some people do just want more of the same, and we can learn that and stop trying to like push them out into new areas, and some people don’t. Because they get stultified, so even though there was all this stuff that they look like– it appears as though they liked their not interacting with that anymore. Okay, it’s time to throw some serendipity their way. And serendipity’s not just less strong something from the entire uniform distribution of all the articles out there. Is something that I didn’t know about that was close to my interests. There’s an algorithm make implementation of serendipity, at least on end that I think that is meant to be an explosive solution to this equipment filter bubble problem, which is really just another word for bad personalization.

Ernie Sander 30:17

All right, I think were out of time. Thank you so much. That was great.

Aria Haghighi 30:19

Thank you.

Mark Johnson 30:19

Thanks so much.