The study “Anticipating Attention: On the Predictability of News Headline Tests” by Nick Hagar and Nicholas Diakopoulos from Northwestern University, and Burton DeWilde from b Chartbeat Inc. measured the importance of textual features to headline performance, in other words how effective a headline is in capturing the attention of the reader.
Headlines in newsrooms are optimized for the audiences with two goals: making the user click the article and presenting concisely what the story is about. Thus, editors and journalists seek to create optimized headlines to increase the audience engagement.
The data for the study comes from Chartbeat, a company providing analytics to digital publishers. service they offer is Engaged Headline Testing, or A/B testing, where multiple variations of a headline are experimentally compared to find out the one most attractive to readers.
The total dataset consisted of 1,023,996 A/B headline tests with 2,662,572 headline variants, run across 1,314 web domains between April 1, 2015, and April 30, 2020. Due to compatibility issues, non-English headlines were filtered out.
Also, all anomalous zero-click headlines were filtered out. The final dataset after filtering was 140,918 A/B headline tests and 334,976 headline variants across 293 domains.
The analysis used predictive modeling on how headline writing affects performance. Textual features were grouped into four categories: linguistic construction, news values, individual tokens, and semantic embedding.
The model achieved a modest performance, but A/B test outcomes are influenced by factors outside the writing and textual features, such as audience behaviors and contextual factors.
When the authors scrutinized the textual features in more detail, they still found only marginal importance for any of the textual features. Of the categories, semantic embedding had by far the most importance.
In discussing their results, the authors acknowledge that the model is limited since it fails to capture any contextual features, but nevertheless, the limited predictability of headline performance by content alone is a result.
As the authors note, having static writing guidelines for headlines failures to capture the dynamic nature of audience engagement, and is insufficient in itself. The results highlight the importance of adopting A/B testing for headlines in newsrooms to measure actual performance.
The article “Anticipating Attention: On the Predictability of News Headline Tests” by Nick Hagar, Nicholas Diakopoulos and Burton DeWilde is in Digital Journalism. (free abstract).
Picture: Untitled by bennett tobias (@bwtobias).
License Unsplash.