Our recent article about altmetrics introduced the concept and highlighted some of its benefits. We’ll now dive behind the metrics and explore what this young concept needs in order to achieve its full potential. Read also our introduction to altmetrics: “Altmetrics, Part I: Tracking research”
Two experts on the topic, Stefanie Haustein, a post-doctoral researcher at the University of Montreal and Kim Holmberg, a research associate at University of Turku, were interviewed for this article.
Some unsolved problems
The benefits of using altmetrics include demonstrating the diffusion of knowledge beyond the research community, speed compared to traditional metrics and the ease of tracking where research is discussed. Possible problems with using altmetrics have also been noted widely. The following list is summed up from the works of Bornmann  and Haustein :
1) Altmetrics are comprised of many types of metrics. There is not a common definition, and activities in social media vary a lot, making assessment difficult.
2) There is a dependence on technology and availability of data providers. Different technologies provide a variety of, sometimes contradictory, altmetric scores. Data companies are under pressure to highlight the value of altmetrics. Currently altmetrics focus on article DOIs, removing other types of research input from the equation.
3) Low data quality. There are inconsistencies between data aggregators. Also the lack of qualitative assessment in practice makes the numbers unreliable.
4) Ease of manipulating altmetric scores .
Some of these problems are similar to traditional metrics. Numbers can be persuasive, and there exists an underlying assumption that higher altmetric scores correlate with better articles . However, research of low quality can gain a lot of online attention, especially if the topic is entertaining and trending. Examples are easy to find.
Information from behind the numbers
Both Holmberg and Haustein hope that altmetrics would not be used as a basis for research evaluation. “We don’t know yet enough about altmetrics. As a comparison we can think about bibliometrics, which have been studied for about 60 years. The debate about whether we can even use for instance citations to previous research for assessing future funding is still a hot topic”, states Holmberg. As altmetrics data is so varied, its value for evaluating research outputs remains unmeasured.
The most interesting part of altmetrics data is found behind the numbers:
“The fact that an article has been tweeted two thousand times or mentioned in some blogs does not really tell us anything. But if [we] look at what these numbers represent, we’ll find the really interesting stuff: in what context the research article is mentioned and how it is being discussed”, Holmberg explains.
This kind of contextual information can be of great use to the researcher or for someone interested in the impact of the study.
More nuanced evaluation needed
Altmetrics are already in use widely. A problem occurring here is similar to the one with counting citations: “If these numbers are used as a type of impact indicator, researchers will change their behaviour to achieve higher scores”, Haustein warns.
Online attention does not equal research impact. Therefore altmetrics would need to be combined with qualitative evaluations and other metrics . Assessments done after the publication, such as the post-publication recommendations offered by the F1000Prime platform, could help with this issue .
Holmberg notes that more research is needed, especially on understanding what different altmetric indicators really tell us. Haustein stresses that putting the numbers in context is needed: “Currently we know how much attention is gained from different sources. Now we need more qualitative analysis on what the different user groups in these channels are and why and how they are discussing research online.”
Many other questions also remain unanswered. How to define research impact? What does impact mean for different stakeholders? Which of the several altmetrics could be used and for what purposes?
What comes next?
Even if proper definitions for different altmetric measurements are missing, some standards for the field are already being sketched. The National Information Standards Organization (NISO) has proposed norms for altmetrics data quality. A code of conduct on this issue is awaiting public comments and will be published in the summer. The proposal demands altmetric data comply with three standards: transparency, replicability and accuracy .
Being a co-chair in the NISO data quality working group, Haustein supports the project. Various altmetric data providers were included in the process. Altmetric.com is supporting the code and Haustein hopes that many other data aggregators will follow suit to improve the quality of underlying data.
Availability of big data with numerous information sources makes assessing metrics much easier. Data quality increases and there are better tools than ever before to analyse what is happening when scholarly information spreads around the web.
Holmberg is optimistic about the information that altmetrics can provide. Haustein also notes good developments:
“Slowly people are moving away from using only one metric. Even if it makes things more complex, altmetrics do give us more context and help us to capture impact beyond citations.”
1 Bornmann, L. 2014. Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. Journal of Informetrics, 8, 895-903.
2 Haustein, S. 2016. Grand challenges in altmetrics: heterogeneity, data quality and dependencies. Scientometrics.
3 For info on manipulation, see e.g. the article Haustein, S., Bowman, T. D., Holmberg, K., Tsou, A., Sugimoto, C. R., & Larivière, V. 2016. Tweets as impact indicators: Examining the implications of automated “bot” accounts on Twitter. Journal of the Association for Information Science and Technology, 67(1), 232-238.
4 Wiley blog. 2015. Sugimoto, C. “Attention is not impact” and other challenges for altmetrics.
5 Bornmann, L. 2015. Alternative metrics in scientometrics: a meta-analysis of research into three altmetrics. Scientometrics. 103(3), 1123-1144.
6 NISO Alternative assesment metrics iniative.
Pasi Ikonen is a Project Secretary at the Department of Communication, University of Jyväskylä, home of Journalism Research News.
Pictures: Measuring Beauty by Iaogooll, licence CC BY-SA 2.0 & Made to Measure by Ian Muttoo, licence CC BY-SA 2.0, cropped, adjusted lightness