Caching out of Hadoop: How New York Times Embraces New Technology
It can be said that an IT organization reflects the business from which it has grown. When it comes to the New York Times, this is definitely true. As a news organization, the company’s collective journalistic head is always on the swivel, always racing towards the newest story. The company’s IT organization, similarly, watches trends and makes bets early, for an enterprise organization.
Nick Rockwell, Chief Technology Officer of the New York Times, is a big fan of the new and innovative. That’s a bit different from the traditional enterprise CTO, who’s typically charged with limiting progress and maintaining two versions of critical systems just in case one has to go end-of-life.
Rockwell, on the other hand, favors finding the best horse to bet on and making that bet when it makes sense. In practice, that can mean abandoning things quickly and moving on to what might work better. But it can also mean giving your teams a much longer attention span and allowing them to solve problems for the long haul.
This is exactly what Rockwell did with Hadoop. Leaving Hadoop in favor of Google’s BigQuery, said Rockwell, resulted in a “Massive cost savings, a huge headache savings, and a giant productivity increase — the third being the biggest factor. Plus, my analysts are happy and smiling, instead of taking long walks in Bryant Park waiting for their jobs to finish,” said Rockwell.
“Scale-wise, we’re not necessarily the leading edge, but I would say BigQuery is making a lot of noise. I think most of our technology folks tend to be a conservative lot. A lot of people need a lot of validation, and then they want to do two things for a long time. I am willing to do things quickly based on conviction and keeping our eyes open. At our scale, we’re picking one solution. We’re going to be multi-cloud, ideally, but it’s an adaptation, not a strategy. I’d prefer to be single cloud. When we look at any service we want to do it once, not twice,” said Rockwell.
Another area Rockwell has managed to save money is in the New York Times’ mobile clients. The client’s ability to push alerts to users when news occurs means that when something does take place, there can be 20 to 30 million push alerts sent out in a single minute.
That spike in traffic is too short lived to be compensated for by autoscaling in the cloud, and too unpredictable to be ready for ahead of time with extra servers. Thus, Rockwell and the New York Times use Fastly as a Content Delivery Network to mirror those alerts and take load off the central systems. That’s saved the company around $25,000 per month by putting Fastly in front of its alerts API.
So happy with that caching layer is Rockwell that he is now looking for internal systems to rip out and replace with Fastly’s caching layer. “We’re poking through our infrastructure looking for ways to leverage Fastly within our infrastructure,” said Rockwell.
That goes along with some of the larger work going on at the Times, said Rockwell. For a start, the company is undergoing a “Massive amount of re-platforming, so this is very good timing for us. We’re generally trying to simplify and purge older implementations in the infrastructure and purge unnecessary apps and internal caching,” said Rockwell.
Rockwell is currently intrigued by Google’s Cloud Platform as a target for future application deployment. “We think some of Google’s higher-level abstractions are demonstrably better than their competitors today. We think Google is a superior engineering company, for a whole bunch of engineering problems than their rivals,” said Rockwell.
When it comes to managing herds of nerds, Rockwell has some advice for others in the technology world. “Get to know the herd. Know the individuals in the herd. Pay attention to the dynamics between the individuals. Figure out who you can rely on to do what things,” said Rockwell.
“It’s all about collaboration, and understanding which programmers can work with which designers and why. Nobody wants it to be that way. Everyone wants programmers to be interchangeable, but they’re just not,” said Rockwell.
Rockwell also has some thoughts on agile in the enterprise. “Agile is very important and specific, but not necessarily about speed. It’s more about accuracy. That is one of my key things. I am a firm believer in the methodologies you could call agile, but I think there’s a lot of misunderstanding about what agile means and why it was created, honestly,” said Rockwell.
“I don’t care that much about specific methodologies or how they work themselves, but the principles behind almost any methodology anyone would call agile tend to be consistent. Amongst those principles are time box iteration, and shipping value at the end of every iteration if at all possible. That’s one that gets lost a lot,” said Rockwell.
Another major factor in succeeding with a development team, said Rockwell, is, “The team having its own commitments, and not having commitments imposed upon them,” said Rockwell. He added that it’s important for the team to have an “Attitude of continual improvement.”
That doesn’t mean becoming completely buried in the minutia and metrics of agile, said Rockwell. Common terms are still important, and running to management with velocity statistics isn’t necessarily going to be giving them useful information.
“For certain metrics, understanding those metrics are only important to the team. Velocity is not something the business can use to see if its development team is good or bad, but it is something the dev team can use to make itself better,” said Rockwell.
In terms of implementing true agile, said Rockwell, he said it takes a “Certain amount of courage, because you don’t want to over plan how to get there. You make a commitment that six months from now we will broadly reach milestones. That becomes a personal commitment, not the outcome of a planning process, but speed is not necessarily an output,” said Rockwell.
But that’s Rockwell’s point: developers can use agile to improve the overall quality of their work, but it is not a path to going faster. Nimbleness comes from elsewhere.
In terms of getting a more nimble team, Rockwell prefers true agility. “I don’t really want to be that nimble. A lot of nimbleness is because you didn’t make good choices in the first place, or because executives are reserving the right to change their minds every five minutes. I hope we have the ability to stay on task longer than that. I prefer to think of nimble in the sense of learning quickly and abandoning things that aren’t working quickly,” said Rockwell.
