[protected-iframe id=”d85c3cf0f11cfc2e67e68dabac53b044-14960843-61002135″ info=”http://new.livestream.com/accounts/74987/events/2117818/videos/21951707/player?autoPlay=false&height=360&mute=false&width=640″ width=”640″ height=”360″ frameborder=”0″ scrolling=”no”]
Input sound file:
Session Name: Constant Progress: Keeping One Billions Users
Announcer Stacey Higginbotham Jay Parikh Najam Ahmad
I have to tell you, this is great. This is the only time anyone listens to me, so that’s pretty cool. Well, we had Google talking about operating at scale. There’s another company out there operating at scale, some of you may have heard of, I won’t waste any more time, let’s hear what they have to say. The Facebook panel, moderated by Stacey Higginbotham.
Stacey Higginbotham 00:29
Thank you, Joe. I love that we’ve got the Facebook panel. Can you imagine if we didn’t have Facebook on stage, this would be terrible, come back. All right, today, we have Jay Parikh, who is in charge of all of Facebook’s infrastructure operations, the guy who keeps everything up and running, right?
Jay Parikh 00:49
Stacey Higginbotham 00:49
So if the site goes down, he’s like a flash, man. And this is Najam, who is just Najam, in networking circles, who is in charge of their networking operations, yes?
Najam Ahmad 01:02
Stacey Higginbotham 01:03
All right, so this is exciting, because for years I’ve begged Facebook always, I’m like “What’s your networking strategy? You’ve got open compute, you’ve got this cold storage stuff,” and they would always put me off. Today, they are ready to talk to me about what they are doing. They’re introducing a new networking fabric for inside Facebook, that’s what you’re working on?
Jay Parikh 01:26
Yes, so I think before we jump into that, I just want to set the stage for why we’re having to do so much innovation on the networking side. We’ve been doing a lot of work on the hardware side via open compute, storage, compute, et cetera. It’s all been pushed because the need of our application is such that everything is virtually real-time and interconnected. Our traffic going across from machine to machine far exceeds the traffic that’s going out from machine to user. I think our ratio roughly is around 1000:1, so there are 1000 more bytes sent out between our data center, across our data center, than there are going from machines out to users. So there is a huge amplification of traffic that is generated and necessary for the real-time nature of our application. This is to generate pages, it’s also doing a lot of the analytics and ranking and all the stuff that’s happening in the back-end. For us, in order to eliminate the bottleneck or free up the application builders, the services inside the data center, we want to make sure that we can eliminate more of the bottlenecks and the oversubscription in this hierarchy that we’ve been building inside our data center. To make everything more real-time, and sort of give more flexibility in our infrastructure.
Stacey Higginbotham 02:47
As an end user, because I always have to bring it back to the end user, will they notice anything like this, or is this mostly a cost or business agility, savings kind of thing?
Jay Parikh 02:58
There’s two things, one is it’s a cost, it’s a flexibility part of it, so that we can continue to move faster, build more services. It’s sort of a flat topology across the data center, it’ll free up and make things like capacity planning and new product launches easier and better for us. The user should notice that the products we build will be better, they should be faster, for us it’s mainly a flexibility and operations manageability play.
Stacey Higginbotham 03:27
We were talking about this earlier, and I think this is a really important point, the whole point of Structure is to talk about how things are changing inside the data center, and one of the things you mentioned is the vendor community had dictated your network design for so long, and that led it to look a certain way, and now you’re trying to – I don’t know if undo that is the right word, but you’re trying to modify that. So what did it look like, and what are you going for here?
Najam Ahmad 04:02
I’ll take that one, if you even just want to go back to when networking technologies were built, they were primarily designed around ISP requirements and ISP networks. If you look at protocol development and things like BGP were developed primarily for interconnection of ISP networks. Technology wasn’t developed as much for content and application services. There are certain fundamental gaps in what requirements we have from an app perspective versus an ISP to be specific, there’s really no way to communicate between an app and the network today. The app really just puts the packet on the net and hopes that it gets to the other end. There’s that level of technology development where we can come up with structures and technology that allows the app to have a better feel of what the network is doing, where the network has capacity, where there are issues and how you route round them, things of that nature, how do you find better cheaper things of that nature, paths across the network. Then the other piece is, if you look at the technology developed in those days it was primarily around platforms and the platforms are very vertically integrated. You have the chipset, the hardware, the operating system protocol stack, everything developed by one vendor and you have to buy the whole thing as a package. It doesn’t matter if all the things met your needs or not, you had to buy the whole package. You had to essentially make it work in that environment. Then you look at the hierarchical nature of the network architectures, if you look at the traditional architectures, where traffic was primarily north-south. Network architectures evolved and platforms evolved in the direction of catering to that. It’s not that any more.
Stacey Higginbotham 06:07
Right, instead of traffic going up from a server out to a user, things are talking across the data center to each other and that’s all kind of trouble.
Najam Ahmad 06:16
All kinds of trouble. The apps are humongous, they’re all over the place, they move around, without the network actually knowing what’s going on. In that environment the hierarchy is actually a bottleneck, because we see flash floods, we see choke points, all sorts of interesting scenarios. Going from more of a cluster-by-cluster hierarchy where clusters are connected by aggregation layers and things of that nature, that just doesn’t scale for us anymore. Building more of a fabric which is flat, if you will, and taking the entire data center, if you can build a fabric that every cabinet, every rack is at the same level, connectivity between any two racks is uniform, then you have a better chance of actually managing the applications, then the construct becomes more logical than the physical hierarchy based systems that we have today.
Stacey Higginbotham 07:14
When you talk about a fabric, it sounds a lot like some of the software defined networking stuff that people have been talking about for the last couple of years. Are you talking about an entirely software defined network for all of Facebook? Or am I confusing things?
Najam Ahmad 07:34
No, If you think about how you solve the problem, networks are a distributed system, if you will. How they manage this fabric is through protocols. But again the protocols were developed more for an ISP type environment, not necessarily for an application. It doesn’t provide the same level of controls that are required for an application. How we build those controls is interesting, and the way to do that is through SDN, these technologies. What we want to do is we want to be able to build a fabric or a forwarding plane that can pass bits, that’s massive in scale, but it’s fairly static. We can build a software layer on top that allows us to develop new functionality on that fabric much more rapidly. Not having the control of the network embedded into the platform itself. If we take the control out and are able to iterate over it, then we can move much faster. If you look at all the application development in our environment, the innovation in applications is much faster than what happens at the network layer. There’s a huge speed mismatch in the app versus the network, so how do we make the network innovation happen much faster, that’s what SDN brings. We can’t actually build a fabric without solving that problem.
Jay Parikh 09:07
Our applications are really– there’s just a huge variety in them. We have some systems that have really high packet rates, and just overall bandwidth is not that interesting, we have other applications that have huge bandwidth needs and consumption, because of replication or something like that. One of our services today handles two and a half billion ops per second, and that’s a lot of packets being sent over the wire, and latency matters, and this goes into page generation, and that’s a lot of packets that we have to handle. The network is a choke point, it can be something that is disconnected from the application, the application needs. The other thing that I’d say that also is really important here in the fabric is just an ease of manageability of the fabric. Today you have these black boxes that all have different interfaces in terms of how you configure them, how you monitor them, how you repair, remediate them, how you set up the different protocols, how they fail, et cetera. This is really clunky and it slows us down in terms of being able to adapt the network to the needs of the software. With this effort that we’ve launched within OpenCompute, around the network switch, we really want to have this OS agnostic box that allows us to do the things that Najam mentioned, but also just the configuration, the manageability, repair, remediation, et cetera, all in software.
Stacey Higginbotham 10:39
Except for the box, because there always is a box. You brought it up before I got to it. You guys in May announced that you were working on a switch, and if I look at you idealized network then, what we’ve got is the OpenCompute switch, we’ve got some sort of Linux-based OS that you guys are working on, I think? On top of that is your controller that you have built. Then on top of that will be the fabric, or the controller is kind of like the fabric? I’m trying to create stacks, because that’s how I make sense of the world.
Najam Ahmad 11:19
Actually your view might be slightly different–
Stacey Higginbotham 11:24
But not wrong, just different.
Najam Ahmad 11:24
Najam Ahmad 11:30
The switch itself is a component of the fabric, and there will be different sizes of switches. The first one we’re starting in a pretty simple, basic, top of rack type environment, but then we’ll have the fabric switches and the spine switches which will form the fabric itself. They’re essentially cogs in that fabric, devices that we’re building. The operating system will allow us to have better programmability and configuration management and things of that nature, that we can use a lot of the knowledge base and expertise we have from the Linux world, from the software side of the world. We manage hundreds of thousands of machines–
Stacey Higginbotham 12:10
Hundreds of thousands of machines?
Jay Parikh 12:12
A lot of machines.
Najam Ahmad 12:13
Did I say a number? [chuckles]
Jay Parikh 12:14
A lot of machines.
Najam Ahmad 12:17
Which are running our apps, and we have tools that manage and configure those devices, but for the network we have a separate set. It tends to be network engineers we have to go and find so we can actually do that, why can’t we have the same system to manage the network itself, as well?
Stacey Higginbotham 12:36
We’re going to talk about some of that I think tomorrow. So now, let’s take it outside the data center, because you guys are also doing some kind of interesting things with undersea cables, and– I love undersea cables. If you read the blog, you would know this, but I love them. So what are you doing around– factually, tell me what you’re doing with buying dark fiber and what you might be doing here, what you’re doing there, and why this is so important.
Jay Parikh 13:13
I think in some ways it’s a similar story. 82% of Facebook users of the 1. 1 billion monthly actives that we have are actually outside the US. So for us, being able to reach them reliably, reach them quickly, that’s sort of everything that we’re trying to do. We’ve announced two things, which is, one is participating in the Asia-Pacific Gateway undersea cable in Asia, and then also the fiber loop that we’ve been building and operating for powering our Luleå data center in Sweden, and connecting it throughout Europe. Really what this does for us is just gives us that flexibility to move faster, so if we do need to launch a new application, we do need to reach our users in different developing, growing countries and regions, then the investment in fiber, whether it be cables or other forms, then that’s what we’re trying to do. So some of this is a cost thing, capex versus opex, but then the other thing is flexibility in being able to turn up capacity much faster as our needs, the machine to machine traffic increases.
Najam Ahmad 14:25
To quantify that “much faster,” we’re talking seconds and minutes rather than weeks that it takes today. We want to be able to point and click, or through a single linking mechanism bring up capacity in a matter of seconds or minutes.
Stacey Higginbotham 14:39
It sounds like, listening to you, because– inside the data center you’re kind of constrained by the boxes that vendors have given you, and protocols and things designed for ISPs, and then going around buying up your own fiber, it sounds like, if I look at the world I feel like we’re moving from the maybe ISP dominated world to a world dominated by content providers. I feel like there’s a lot of tension there, happening between the two sides. Will that side ever come around, does it need to accelerate in terms of– you guys are huge, but what about all the other people who are trying to build companies, that are going to work over these networks. Do they have to build their own, do you see that shifting?
Najam Ahmad 15:33
When you say two sides, you’re talking ISPs or vendors?
Stacey Higginbotham 15:36
Well, okay, you’re right, there are three sides! Some of the vendors are really in tune with the ISPs, is kind of my thinking there. We’ll say ISPs, and then maybe just the old way of doing it, the telecom versus the IP network. Maybe that’s a good way to think about it, I don’t know.
Najam Ahmad 15:55
I think ISPs definitely need to solve the end-user connectivity, the access problem. Every user in the world needs connectivity, and that problem needs to be solved, and that’s not really what we’re trying to solve from that point of view. There’s definitely that access that those guys need to focus on. The thing that– and I talked about this at ITW last time, was that the tension is over the content being considered over the top of ISP networks. ISPs traditionally have two models, retail and wholesale. We don’t fit in either one on them. We’re not a wholesaler, we’re not buying wholesale and reselling stuff, so what really needs to happen is more partnerships in trying to figure out what’s the best way to serve this particular segment, and that’s where things have been slower, that newer models haven’t developed. I think where the ISPs and the industry have opportunities is where building cloud type infrastructures. If you think about this we’re building our own private cloud, and for our own use, not to sell. But there are enough companies who are building public clouds as well, and if you look at the startups today, they are just going to those clouds and starting their apps, that’s how those startups are doing it. There’s a lot of opportunity for the industry to provide really huge, massive, scalable infrastructure for all the startups that are coming out. At some point you get to the scale where building your own is the only way to do it, but there’s lots of opportunity there.
Jay Parikh 17:36
I think you get to that point where you need to have better control around cost, or you need the flexibility in terms of time to market and being able to turn things up that you can’t get from some set of vendors or ISPs or partners.
Stacey Higginbotham 17:51
Derek thinks that you guys should open up your infrastructure to startups. Maybe next year at Structure. All right, well thank you guys, very much.