Here is a brief overview of Altmetric, just in case you haven’t heard about it. Altmetric LLP is a UK-based company with <50 employees that provides collects “impression” data to evaluate academic studies and factor it into a single score, which is then presented in a colorful circle which Altmetric calls a doughnut. Unlike other evaluations matrices that require active inquiry to evaluate a paper, Altmetric employs search bots that find all the mentions of any study online and rank it following a simple formula explained here. Briefly: News=8, blogs=5, Twitter=1 and there are few more restrictions to prevent cheating. Studies with only 20 doughnut points are already at the top 5% of most-read or most-interesting papers. This only shows how bad we all are in communicating our findings to the public. Starting from 2013, Altmetric began publishing the 100 papers that received the top scores.
I am no stranger to Altmetric. In 2013, our article that criticized ENCODE reached #73 in that list with 673 doughnut points (on the left).
In 2014, our GPS paper missed the list by only 10 places. Later that year, I actually became aware of the existence of Altmetric and was happy to learn that almost all my population genetic papers are ranked in the top percentile of Altmetric.
So how would you get the desired Altmetric doughnuts? Call every newspaper in the world? Wrong! There is a finite number of newspapers in the world, but practically infinite number of users on Twitter and “tweeting”/“retweeting” is far easier than to get someone to write a news report about your paper. Although worth only one point, tweets accumulate fast.
With that wisdom in mind, I tweeted my latest paper on the origin of Yiddish a few times and indeed, despite my modest number of followers, the paper went viral. My Altmetric doughnuts rose from 17 to 50 and from 127 to 253. Within less than two weeks they popped up to 388 fresh and juicy doughnuts! Based on the score of past winners I estimated that 500 would get our paper to the top 100 list, but fate stood in the way…
Who moved my doughnut?
One night I went to sleep hogging 392 doughnuts, but woke up with only 220, not enough doughnuts to feed even the private police force in New Orleans. What happened? Altmetric sliced my twitter score by half. “Maybe they made a mistake in counting before” I told myself and kept following the score. I also noticed that Altmetric listed less than 20% of all the media outlets that reported my study and decided to do a small experiment. Altmetric allows submitting websites that they missed to I submitted one blog. Altmetric added this blog under “Misc,” which doesn’t get score. Few days later Altmetric exacted its revenge on questioning its formula and took away 30 of my precious doughnuts. On the positive end, the rising interest of the Twitter community gained me again 335 points from Tweets alone. It was time for a second experiment. This time, I sent Altmetric a list of 10 blogs that were missed. Once again I learned the cost of vigilance. Altmetric confiscated 50 of my doughnuts, wiping more twitter points and pushing me way below the 300 line. I cheered myself again that maybe the original Twitter count was incorrect, but I contact the tech support and asked why Altmetric lists less than 20% of the websites that mentioned my study? Altmetric tech support explained that the algorithm is going through some upgrades and agreed to look at the list of missing websites. By that point I was again the proud owner of 304 doughnuts just before the hungry vengeful Altmetric machine consumed 40 of them without adding a single website. As before, Altmetric suggested that tweets were erased assuming that I have no way to check that, except that I did.
Caught red handed
I recalled that one of my Twitter friends had 124 followers who retweeted about my study, as you can see below, this is nearly TWICE than what Altmetric reports that I currently have. Sloppy Altmetric didn’t bother to fix that number and only some of those 124 tweets were considered towards the 73 count
Recently, homolog.us tweeted this and wrote “dedicated to the ‘altmetric’ God,” suggesting that social media impression do not mean truth. I think he nailed it. Altmetric is playing God evaluating our hard academic work, however this is a false God, a vengeful God, and a petty God that like all false, vengeful, and petty gods it is controlled by a small group of people sitting in some office who uprank or downrank papers based on their agenda, interests, and politics, not by the actual exposure of the paper.
Let us stop pretending that this Altmetric is an objective measure that deserves to be cited on websites and grant applications. Another valuable lesson from this experience is that we should continue improving the way in which we communicate our science to the public.
Prior to the publication of this piece, I sent Altmetric a draft and invited them to respond. Altmetric founder, Mr. Euan Adie, was very quick to respond:
Sorry you’ve had a bad experience with Altmetric. We don’t deduct anything from the score for answering support queries, it’s an automatic process that follows this process.
The score calculation certainly is subjective and it sounds like you might have been affected by a bug at some point too, but actually most of the problems come from us trying to apply that calculation objectively. This is why some things go into the misc tab etc. which I know causes frustration.
Mr. Adie prompt response is much appreciated, however it is difficult to see how his explanation fits with my recent experience with Altmetric where each interaction with Altmetric resulted in quick deduction of points without actually fixing the problem. It is also difficult to believe that a bug would affect only my most recent paper and none of my other papers. Nonetheless, I wish to extend my best wishes to the Altmetric sick bots and hope for quick recovery from the bug.