How to: bring agile into the newsroom

Wander into a room of developers and you may hear the term "agile". But what is agile and why and how should journalists learn from developers and adopt some of the principles?

This guide looks at how a number of newsrooms have taken aspects of agile on board, and how some journalists have actually been using some of the principles without knowing it.

Journalism.co.uk speaks to Cory Haik, executive producer for digital news at the Washington Post; Brian Boyer, who was a news apps developer at the Chicago Tribune and is now news applications editor at NPR; John Stefany, a project manager at NPR; Martin Belam, who was lead UX (user experience) at the Guardian and now runs digital consultancy Emblem; and Richard Sheridan of Menlo Innovations, software developers who train others in agile.

What is agile?

"Agile is a set of principles used to improve the way groups work together on projects," NPR's John Stefany told Journalism.co.uk. "It's most commonly used in software development but it has relevance to other fields, including newsgathering."

The principles were codified in the "agile manifesto" in 2001 but the ideas had been around for a couple of years before they were formalised.

More recently newsrooms have been adopting some of the principles. "Agile became a buzzy thing for newsrooms a couple of years ago, with Brian Boyer and others really champions of it in other places", Cory Haik from the Washington Post explained.

"As the challenge of newsrooms became about integrating the technologist and the journalist together, agile just became part of the language."

Move fast

Agile is about pushing projects out without delay. "It's a recognition that in a competitive market there's value in bringing new products out as quickly as possible," Stefany said.

Agile recognises that "humans are naturally wired to make mistakes", Richard Sheridan of Menlo Innovations said. "Our view and the view of the agile industry is, how about if we create a system that allows us to make those mistakes quickly, review, discover those mistakes as fast as possible and correct them before they kill us?

"Being agile simply means reorganising your processes in a way that allows you to make those mistakes quickly, get feedback on them, so setting up short communication and feedback loops, and then iterating your way to the right conclusion even if it that conclusion is well different to what you imagined at the beginning."

"The whole process is all about refinement and improvement," Martin Belam added.

"Agile allows developers to focus on the things that are important, which is actual working software rather than things that are unimportant like extensive documentation," Belam said. "One of the things that's really important is to value working software above all else."

It really tries to emphasise individual interactions over processes and plans and tools, delivering working software or other products over creating comprehensive documentation, and it favours customer collaboration over legalistic negotiationsJohn Stefany, NPR

Seeing a working product helps the whole team, Stefany explained. "Agile recognises that customers sometimes need to see a product to assess whether that's what they really need. You can describe things in requirement documents all you want but until it's something that people can actually get their hands on, they don't really know whether its what they wanted or not."

Agile "also recognises that all of us are bad at estimating how long things will take and it really appreciates is that needs change over time", Stefany added.

"In a competitive landscape you need to be able to respond to that in short increments rather than waiting months and then changing your game plan."

Stefany summarised: "It really tries to emphasise individual interactions over processes and plans and tools, delivering working software or other products over creating comprehensive documentation, and it favours customer collaboration over legalistic negotiations."

If a newsroom adopts agile, constantly refining and improving, the project will never be finished. The Washington Post thinks of its website as "always in beta", as Haik explained when she delivered the keynote presentation at this month's news:rewired digital journalism conference. When I spoke to Haik from her newsroom in Washington on Friday (20 July) afternoon, she illustrated the point by describing the build of a new liveblogging platform the Washington Post is creating using agile principles.

At the time I spoke to Haik they were using the unfinished liveblogging platform to report the Aurora Colorado shooting despite it "not being fully baked".

The platform has been in development for the past three or four months with the goal to have it ready for the Olympics, but he Washington Post team started using the platform during the US electoral primaries, Haik told me.

"It was not complete but we would use it one night and then we would come back and do bug reporting then the developer and the designer-developer would work on things, put out a new version of it .... and the next night we'd do it again, and now we've got something that's pretty great." 

Putting the user first

The news apps team at the Chicago Tribune team thinks of the process of building a product as "deadline-driven development". The team starts by thinking about who the users are. "That should be really familiar to an editor, who should be an expert in their audience," Boyer said. "The next question is 'what are their needs?' and the third question is 'what features can we build to fulfill those needs?'"

The team writes these features on notecards, putting them into piles labelled "must", "want", "nice" and "meh". "The secret of the process is you sort them and then throw out the bottom two categories. You then put them in order of priority for the user. You execute by working through the list and when you hit a deadline you stop.

"Prioritising lets you break your work into iterations. We usually work in week-long iterations, which means that at the end of every week we have completed a unit of work. It prevents you from getting stuck with three half-done features and your deadline is approaching."

Prioritising means that "when the deadline hits you can cut off the bottom of the page", Boyer said. "And that's exactly what the inverted pyramid is all about."

How does agile work?

There are a number of variations on the agile methodologies and ways of actually applying these principles, Stefany explained.

• The scrum

The scrum is used both at NPR and at the Chicago Tribune. It is named after the rugby term and is based on the idea of small teams working together. "In our case there is a product owner, who sets the vision, one or more developers, designers, a user-experience architect and the project manager assumes a role called scrum master," Stefany said.

• Removing impediments

Agile also encourages the removal of obstacles, such as lengthy meetings. As scrum master, Stefany's role at NPR is to "keep the team on track following the rules of the process, and more importantly removing impediments that they have to doing their work".

He also looks for opportunities to streamline. "Maybe they've been scheduled for a whole bunch of meetings that aren't really necessary, so I'll try to go and get those removed, for example."

The Washington Post recognised that "there are too many meetings in newsrooms, particularly in digital", Haik said, and have made moves to tackle this.

• Sprint cycles - the start

At NPR, as with many other teams of software developers, the team meets every two weeks to plan what it is going to do for the next two-week sprint cycle.

"It's usually about a two-hour meeting and we go through in fairly good detail as to what the plan is going to be for the next two weeks," Stefany said.

• The daily stand-up

Agile believes in stand-up meetings to keep them as short as possible. "We start each day with a daily stand up," Stefany said. "It's intended to be no longer than 15 minutes, and we go around the group and each person says what they did yesterday, what they plan to get done today, and what impediments they have to getting that work done".

The Washington Post also adopts the 15 minute stand-up for its weekly meeting of the digital leadership group, Haik told me. They too take the "rapid fire" approach, asking what colleagues are working on and discovering if connections can be made.

• Sprint cycles - the end

The team at NPR holds a sprint review at the end of the two-week sprint cycle. "That's where the stakeholders of the software come in," Stefany said. If it is a mobile application, for example, the key people can see what the team has actually produced. "The rules are it has to be working software, it can't be smoke and mirrors," Stefany said.

"This is their opportunity to provide feedback on whether it meets the needs, or whether for the next cycle the team needs to shift gears and rethink priorities."

After the sprint review the members of the team who built the software get together and have a retrospective. They look at the process, assess how well the two-week sprint cycle went and ask "how we can in change the way we work together in the next cycle to make our process more efficient", Stefany said.

• Pair working

Some versions of agile encourage pair working, Boyer told me. "A lot of software developers think it's absolutely mad, but I firmly believe that two programmers sitting at the same keyboard are more productive than two programmers sitting at separate keyboards. They catch bugs, they help each other talk through problems."

Pair working is also is a way of helping people focus, Boyer has found. "If you are sitting at a keyboard next to someone for eight hours, you work the entire time. No one is checking their emails, no one is tweeting, and frankly it's exhausting."

Pair working is something the Washington Post has been doing for "a long time", before it started thinking in terms of agile. When close to a deadline two journalists often sit together on one computer. "That limits the distractions, which is definitely key," Haik said.

So what are the downsides to agile? Belam says that agile can "make design more difficult in that you could produce what you expect to be a finished design and then at the end of the fortnight the developers say 'we got 70 per cent of it working' and here's our best guess of how that will appear on the page".

Being agile in the newsroom

Stefany believes that journalists already use elements of agile. "Journalists have been doing this for a long time," he said. "The assigning editor is the product owner, the voice of the customer, he/she prioritises what gets worked on on a daily basis.

"Your team could be a reporter, photographer, graphic artist and videographer, and they are going to work best when they are all in constant communication with each other, ideally face-to-face, but at least checking-in regularly."

Newsrooms could apply agile to "longer, investigative projects that take months", Stefany believes. "Traditionally we've approached this by a long research phase, then photographers are brought in, then at the end graphic artists. But you really don't have anything concrete until the very end of that process. In the meantime the story could change or your competitor could scoop you on part of it, so if you were to apply agile principles to that, you would be constantly developing publishable content."

"We work in two-week sprints at NPR on the software side but there's no rule to how long a sprint is," Stefany added. "You could envision it as a daily story that has multiple deadlines during the day, so each deadline is a sprint."

Breaking news doesn't make it more difficult to do agile in a newsroom or a news environment, it just means you have to plan for itMartin Belam

"After a story breaks and you do your first pass on it and after that deadline you reconvene to ask 'what did we get? what do we need to focus on now?' and then you try to improve that for the next one.

"If a tornado rips through a town, your first set of questions isn't to find out what is means for the apple blossom festival in two weeks."

But can agile really work for breaking news? "One of the criticisms I've seen of using agile methodologies in a news organisation is that it doesn't make room for the sudden big breaking news event where you need to do an infographic or change the style of something or the tone of something," Belam says.

"I can't see that myself. The thing with agile is you always set aside a time in a sprint to deal with bugs, to do refactoring, which is where you go back and make code that's more efficient. A well organised agile team should have 20 or 25 per cent of their capacity to do those things, so it's not beyond those realms of possibility that you also assign 15 per cent of capacity for breaking news events.

"Breaking news doesn't make it more difficult to do agile in a newsroom or a news environment, it just means you have to plan for it."

Agile at the Washington Post

Rather than following "hard core agile" the Washington Post newsroom has developed a "beta mindset", Haik said. "We are borrowing some of the things that work in agile, like scrum meetings, stand-up meetings everyday or, if it's a longer project, perhaps once a week".

Teams at the Washington Post may consist of a developer, designer, an editor, a reporter, for example. "Everybody does their updates and then we keep notes," Haik said. "It's a very light documentation, it's almost like a liveblog of what everyone is working on on the project.

"The idea is we are going to get something live as soon as we can and then we are going to do public-facing testing in some way, get feedback from users, get feedback from the group, and then continue to work and to develop on it and iterate."

This is how the Washington Post's liveblogging platform developed, but have they tried using agile for investigative or purely editorial projects where there is no software element?

Haik did not reveal details of one of the projects her team is currently working on but did say that they are "taking an agile approach to it".

"We are going to have to build some digital things," she said. "We'll need a data journalist, for instance, but there's no heavy platform building, it's more about storytelling and the content."

The team members on the project are going to be doing "check-in meetings, people are going to go away and work on things and come back and explain what they have".

And the starting point is always the user, Haik said. "The team starts by asking 'what is desired effect of the project? what story do we want to tell users?'" They then may "prototype some of that", working out what direction they can take the data in.

"Instead of a long planning process, as reporters are working on the stories, the data journalist and a designer/multimedia journalist are going to work together and build things at the same time," she added.

As at the Washington Post, the newsroom of the Chicago Tribune has also "cherry picked" elements of agile, selecting parts that "fit in really rather nicely with the news cycle", according to Boyer, and at NPR, the newsroom is "just starting to scratch the surface" of agile, according to Stefany.

"We think there's tremendous potential to bring some of these ideas into the newsroom, but journalists are not attuned to these kinds of processes in the same way as software people are," Stefany added.

Parallels between journalism and software development

Belam sees one of the parallels between software development and journalism as the cycle of improvement and refinement in the traditional print publication.

"The old model where you published a first edition and then you spotted some errors with it and you published a second edition and then a story develops slightly and you publish a third edition and by the fourth edition you've got the thing right. I think that very much mirrors the iterative approach to software development.

"I think the thing that is difficult for news organisations is the fact that things are never finished with agile and although you could say that news stories are never finished, there is a final layout and design of a page and you know that's the thing that you are sending off to print."

But that may have changed with the development of digital. Belam feels the "circular iteration process where you do something, you test it, you see what works, what didn't work and you improve the things that work" mirrors online reporting. "Meg Pickard [head of digital engagement at the Guardian] published a blog post about the publishing process and the opportunities for community collaboration," Belam said.

"She looked at how the model used to be that you would commission and article, do some research, write it, get it subbed, publish it in the paper and then wander off and go and do the next one. She's suggesting that with digital there's much more of a circular thing there."

Belam said the process of crowdsourcing may mean part of the journalist's research is done in public and building on the story with the help of the audience has parallels to agile.

What else can journalists learn from developers?

Boyer believes that although not agile, one thing that journalists can learn from developers is reserving time for self improvement.

"One of the things we had at the Tribune is that every person on the team in addition to being able to go to a conference was given five hack days, as we call them. They are sort of a paid vacation in which you are allowed to do whatever you want as long as it is cool. If you want to work on a project or attend a conference or you want to go and build a radio station in Africa, you are encouraged to.

"That something that seems really missing in newsrooms, when everyone is hustling for the deadline all day long and everyone is being encouraged to do more with less.

"I would love to see newsrooms give reporters a day a month or two days a month to learn a technology, to learn how to code, to learn how to use Twitter. It would benefit both the reporters and the organisation and the industry."

The Washington Post has held hackathons, which have involved journalists as well as developers. The "social reporting" news app @mentionmachine, which monitors Twitter and media across the web for US election candidate mentions, was developed as a result of one such event.

Hack days and off diary projects are "very key to the culture" of the Washington Post, Haik said. "We feel very strongly that people have to have some time to do projects that don't necessarily fit in to exactly what we are doing now but ideas may come out of them or they could be an launchpad for something else."

Advice for journalists

So what is the advice for journalists who want to begin bringing agile into the newsroom? "Go and talk to your news organisation's technical department," advises Stefany. They are probably using agile and can share advice.

You have to try things and you have to try them quickly. Go fail, that's my adviceCory Haik, Washington Post

Sheridan's tip is to be open minded and embrace new ways of working. Although the agile manifesto was not codified until 2001, the concepts arrived in Sheridan's team a couple of years before.

"When I went to my team and suggested what we were going to do and that we were going to take a different approach the first reaction was one of my programmers raising his hand and said 'blood, mayhem, murder. Don't do it'." Thirteen years on and there is still some resistance but Sheridan is adamant that agile is the best way of working and shares his belief and best practices with those who attend his training courses.

And what tips does Haik have for journalists wanting to learn from software developers? "You honestly have to embrace the 'fail fast' mentality. Think of your failures not just as failures but as a path to something else. Is there a way to take whatever didn't work and build something else? Learn lessons and just keep moving on.

"You have to try things and you have to try them quickly. Go fail, that's my advice."

Related links:

• Podcast on agile for newsrooms
• Blog post by Brian Boyer

Free daily newsletter

If you like our news and feature articles, you can sign up to receive our free daily (Mon-Fri) email newsletter (mobile friendly).