This article first appeared here and has been cross-published with permission from the author
Have you ever wondered how news leaders measure the impact of their articles? So did we, and we asked eight top media professionals and data analysts to lift the curtain on their evaluation methods. Our question was simple: 'How do you evaluate the success of an article?' Read what unique metrics they lean on to define article success.
Game changer: conversions over page views
Dávid Tvrdoň, Subscription Growth and Retention Specialist at Denník SME (Slovakia)
"A few years ago our primary focus used to be purely on looking at the number of page views of articles, which is undeniably still a very relevant metric.
"However, nowadays our newsroom has evolved and looks at subscription conversions attributed to a single article. While the editorial team has access to many other metrics, such as time spent on the article, which we like to look at globally, subscription conversions have become a key metric we prioritise in our evaluation process.”
Tomáš Bella, Chief Digital Officer at Denník N (Slovakia)
"Our key metric for assessing article success is the number of new conversions per article, basically representing the number of readers for whom the article was the last trigger to motivate them to buy a new subscription.
"While this metric is straightforward and universally understandable, it has its shortcomings. Mainly, it doesn't take retention into account, only new subscriptions. Nevertheless, it is so simple and easy to understand for everyone that we use it as the key indicator in the newsroom.
"Our journalists see it displayed on live dashboards, they get daily overviews of all articles published and weekly summaries of their own articles. Additionally, every month they get a list of the best converting articles, and journalist bonuses are partially based on this list.
"In addition to new conversions, we also analyse various article metrics. This includes:
- conversion rate of the article (pageviews vs. number of conversions)
- the number of new registrations triggered by the article (through registration paywalls)
- the number of unique browsers that visit the article
- ratio of visits by subscribers versus non-subscribers
"By examining these comprehensive metrics, we gain a deeper understanding of article performance and its impact on audience engagement and subscriptions."
Lars K Jensen, team lead audience at Berlingske Media (Denmark)
"We look at multiple parameters: we take into account both quantitative measures, such as views, as well as qualitative measures that assess the quality of consumption. Additionally, we prioritise understanding user needs and preferences.
"Currently, our primary focus is on conversions, which involve people purchasing subscriptions or signing up for user profiles. However, we are also committed to delving deeper into understanding our users and subscribers on a more granular level. We are good at “keeping” them and ensuring they feel welcomed.
"To achieve a well-rounded assessment, I would suggest other publishers to think of a way where you can semi-qualitatively measure and assess your journalistic output on a quantitative level through the eyes and experiences of your readers. I have been doing this as a part of my daily work for the past two years, and I wish we had started it earlier, as it has proven invaluable."
The subscriber shuffle: acquire, keep, repeat
Lukas Gorog, data strategist at Die Presse, founder of Predictea (Austria)
"When evaluating the success of an article at our organisation, we employ two distinct audience approaches: subscriber acquisition and audience retention.
"For subscriber acquisition, we rely on a combination of page views and engagement metrics. An article is deemed successful when it performs strongly in both these areas. Engagement metrics adhere to a precise formula that includes user events, like newsletter subscriptions, time on page, author subscriptions, and number of loyal readers per article. This score ranges between 0 and 1.
"To quickly assess the effectiveness of our new subscriber acquisition tactics, we conduct various A/B tests. For instance, we A/B test the performance of our recommendation algorithms, the impact of various page layouts, and the effectiveness of different paywall positions on our website.
"In terms of audience retention, we analyse page views and engagement metrics, focusing specifically on our loyal subscribers. This approach allows us to evaluate the quality of our product and its alignment with our long-term strategy. By analysing these metrics, we gain valuable insights into how articles impact audience retention, empowering us to make informed decisions about our content strategy."
Emanuele Porfiri, senior data analyst at FT Strategies (UK)
"Though obvious, we believe there isn't a one-size-fits-all solution when it comes to measuring content performance (it depends on the organisation and content's goals, as well as the data available).
"At the FT, we evaluate the performance of an article based on the volume and quality of engagement it generates. Whilst volume is perhaps a straightforward metric, quality of engagement is a trickier one to define and we opted for a metric we call ‘quality reads’, that is based on a set of page views and article indicators, and estimates the percentage of page views where the reader has read at least half of the article.
"Ultimately, as there are endless ways in which to measure performance, at FT Strategies we recommend testing different versions (if the data is available) and validating them against a desired user action (e.g. registration, subscription, retention…). At the FT, we can see that subscribers with higher quality engagement are less likely to churn, and that’s exactly one of the behaviours we want our content to generate."
Delving into deep reader engagement metrics
Andrey Boborykin, executive director at Ukrainska Pravda (Ukraine)
"While we start with the fundamental metric of page views to gauge initial reach and interest, we recognise that true engagement goes beyond mere clicks.
"To gain a more nuanced understanding, we delve into scroll depth and engaged reading time. By analysing how far down the page readers scroll, we can infer the level of their engagement. Furthermore, we measure the amount of time readers actively interact with the article, which provides valuable insights into their engagement.
"These metrics are then evaluated against our internal benchmarks, allowing us to assess not only the absolute numbers but also the relative success of each article. Such comprehensive analysis helps us refine our content strategy and ensures that we consistently deliver relevant and engaging content to our readers."
Joao Rodrigues, data specialist at Observador (Portugal)
"Our approach to measuring the success of an article has changed over time. Through numerous iterations and collaborative efforts between the editorial and subscriptions teams, we have arrived at a formula that is now widely accepted in our newsroom.
"We believe that a combination of three key factors is crucial in defining success: reach (page views), engagement (time and scroll depth), and conversions (number of subscriptions resulting from a specific article). Evaluating these three factors together helps us achieve a more comprehensive assessment of an article's success and continuously refine our content strategy, ensuring that we deliver impactful and engaging articles to our audience.
"We circulate these metrics to our journalists, editors, and directors daily. Directors use them to know what content to highlight on the homepage and which articles to pick in their daily and weekly newsletters.
"Editors analyse these metrics to understand which articles (and their respective subjects) have performed well in recent days. With these insights, they make informed decisions on what to write (or not to write) next. Journalists use these metrics, offering their unique perspectives on why they believe an article performed better or worse."
Measuring the immeasurable? The challenges of quality and impact assessment
Greg Piechota, researcher-in-residence at International News Media Association (INMA)
"Evaluating the success of journalism is a subject of wide debate. Although many agree that the purpose of journalism is to make an impact, they agree less on how to measure it.
"The work of journalists can be easily measured by activities and outputs, but evaluating output quality and impact is trickier. It includes checks for adherence to the craft standards, metrics of user engagement, and recognition by peers, e.g., citations and awards. The impact of journalism on individuals, society, and institutions is the hardest to quantify.
"User engagement is often measured with usage metrics, with the most common showing popularity - how many people visit the website or click on the links. Then there are metrics showing how engaged readers are with a website, how many articles they read, or how much time they spend each time they visit. One often measures whether users read or add comments, share articles with others.
"Metrics of loyalty are also crucial. For example, how often users visit the website, how many were present on a particular day or in a given week or month, how much time has passed between visits. In the news subscription context, a common metric is 'lifetime value' (LTV) which measures the expected revenue or profit over the lifetime of the relationship - from the day someone subscribes to the predicted day of cancellation.
"Engagement is highly individual in digital media use. People have different ways of engaging and engagement varies across website types and pages. It also differs based on the context of a visit and user intent. People behave differently when they just want to kill time and find out what's going on in the world, and differently when they are looking for specific information. Understanding these diverse behaviors is essential, otherwise comparing them will be difficult."
Success is more than just numbers, right?
As we have discovered, the magic of a successful article goes beyond numbers - it ignites curiosity, sparks conversations, and provokes thought. In the quest for success, one should not be confined by data alone.
Rather, go and seek out the intangible connections that resonate deeply with your readers. It is these elements of empathy, authenticity, and a genuine passion to enlighten that truly set an article apart. After all, the road to success is not just paved with data - it is built on the foundation of genuine human connections and lasting impact.
Martina Klárová is research and development manager at Fatchilli for Publishers, a one-stop shop to help publishers monetise their content
Free daily newsletter
If you like our news and feature articles, you can sign up to receive our free daily (Mon-Fri) email newsletter (mobile friendly).
