Anatomy of a security fix: Postgres launches massive update to address vulnerability

It’s the kind of call any software vendor — or open source project — dreads: A large customer (in this case NTT) flagged a vulnerability in PostgreSQL, the popular open-source database also known as Postgres. That happened March 12. The  next step for the Postgres community (and it is a community, not a single vendor, which complicates things) is to assess the vulnerability, evaluate whether it’s really an issue and then figure out what it takes to fix it.

In this case, the security SWAT team deemed this to be a real problem and scrambled to address it. “The team evaluated it and wrote the code for the fix, which actually took very little time and then started scheduling the release,” Josh Berkus, Postgres core team member told me in an interview. And it’s in that scheduling and coordinating the rollout to thousands of Postgres repositories that was really tricky.


When it comes to any big fix or patch, the practice is to create an installer to ease its application. In Postgres’ case, because it runs on virtually every flavor of Linux as well as Windows, they needed to come up with 80 different packagers. “That’s why delay was built into the process,” Berkus said.

“If it were a normal, minor update for bugs, we don’t worry about making them all available at once, but for this, we felt we needed to.”

There  was also the issue of disclosure. You want users to be alert as to what’s happening but you don’t want to “provide a roadmap” of the vulnerability to the script kiddies, Berkus said.

A March 28, message posted up on the Postgresql message board alerted folks of a patch to come April 4. Then Heroku(s crm) the popular Platform as a Service, which supports lots of Postgres users, posted that it was issuing the patch starting April 1. That timing set off some of the Postgres faithful who felt that Heroku was getting special treatment. It also garnered some press attention and Hacker News comment.

Berkus said there’s a reason for that. Heroku provides the database as a service, not the binary code itself. Heroku requested that early access because it has lots of machines. And, as it turns out, the vulnerability could impact any Postgres user that has port access to the database even if he or she does not have a valid account.

The nature of that vulnerability meant Heroku — which runs on Amazon(s amzn) Web Services– or any Postgres user running on AWS or other public cloud could be vulnerable depending on how they set up their servers. Many customers running on public cloud leave ports open, probably because they don’t know better.

From the Postgres FAQ about the issue:

Any system that allows unrestricted access to the PostgreSQL network port, such as users running PostgreSQL on a public cloud, is especially vulnerable. Users whose servers are only accessible on protected internal networks, or who have effective firewalling or other network access restrictions, are less vulnerable.

This is a good general rule for database security: do not allow port access to the database server from untrusted networks unless it is absolutely necessary. This is as true, or more true, of other database systems as it is of PostgreSQL.

Berkus and the folks at Heroku who spoke to me on this issue were quick to assert  that while this was a big vulnerability — much bigger than the last vulnerability back in 2005 —  there was no sign of any exploits.

So, net net net, Heroku rolled out its fix earlier this week. Minor drama ensued. But as of now, the rest of the world is covered as well.