Does an experimental setting affect news audiences’ behaviour? The question is a relevant concern to researchers who consider sending participants to a mock news website as part of their methodology: if the knowledge of taking part in a study affects the participants’ behaviour, the method is not externally valid.

University of Texas at Austin researchers Natalie Jomini Stroud and Emily Van Duyn (the latter is also affiliated with Stanford University) probed the question through two comparative experiments. They collected audience browsing data from one American and one Canadian news website. Some of the data came from knowing participants to an online experiment, while some came from unwitting visitors to those websites.

Both of the news websites were going through a redesign, and the authors attempted to find out which version -the old or the new- produced more favourable audience metrics. So, some research participants were directed to the old version and some to the new version, ditto for the regular visitors. Then Jomini Stroud and Van Duyn compared the data sets: did the experimental setting yield different results than observing natural audience behaviour?

Both methods produced a similar outcome: the new website designs were superior to the old. This also means the audiences behaved roughly the same regardless of whether they were knowingly taking part in an experiment or browsing naturally.

However, the results were not uniform in magnitude. For example, the increase in page views was far greater in the natural than in the experimental condition. The authors also noted there were differences between the Canadian and American data sets, suggesting people might react differently to the experimental setting in different countries.

The experimental methodology appears valid in studying the direction of change, but less so in studying the magnitude of change, Jomini Stroud and Van Duyn conclude.

The research brief, entitled “Assessing the External Validity of Using News Websites as Experimental Stimuli” was published by the journal Communication Methods and Measures. It is available online on the publisher’s website (abstract free).

Picture: Untitled by StockSnap.