On the fallacies of Altmetric OR how I fell asleep in Sheffield and woke up in Louisiana

73.png

Here is a brief overview of Altmetric, just in case you haven’t heard about it. Altmetric LLP is a UK-based company with <50 employees that provides collects “impression” data to evaluate academic studies and factor it into a single score, which is then presented in a colorful circle which Altmetric calls a doughnut. Unlike other evaluations matrices that require active inquiry to evaluate a paper, Altmetric employs search bots that find all the mentions of any study online and rank it following a simple formula explained here. Briefly: News=8, blogs=5, Twitter=1 and there are few more restrictions to prevent cheating. Studies with only 20 doughnut points are already at the top 5% of most-read or most-interesting papers. This only shows how bad we all are in communicating our findings to the public. Starting from 2013, Altmetric began publishing the 100 papers that received the top scores.

 

I am no stranger to Altmetric. In 2013, our article that criticized ENCODE reached #73 in that list with 673 doughnut points (on the left).

 

In 2014, our GPS paper missed the list by only 10 places. Later that year, I actually became aware of the existence of Altmetric and was happy to learn that almost all my population genetic papers are ranked in the top percentile of Altmetric.

 

Scoring doughnuts

So how would you get the desired Altmetric doughnuts? Call every newspaper in the world? Wrong! There is a finite number of newspapers in the world, but practically infinite number of users on Twitter and “tweeting”/“retweeting” is far easier than to get someone to write a news report about your paper. Although worth only one point, tweets accumulate fast.

With that wisdom in mind, I tweeted my latest paper on the origin of Yiddish a few times and indeed, despite my modest number of followers, the paper went viral. My Altmetric doughnuts rose from 17 to 50 and from 127 to 253. Within less than two weeks they popped up to 388 fresh and juicy doughnuts! Based on the score of past winners I estimated that 500 would get our paper to the top 100 list, but fate stood in the way…

Who moved my doughnut?

One night I went to sleep hogging 392 doughnuts, but woke up with only 220, not enough doughnuts to feed even the private police force in New Orleans. What happened? Altmetric sliced my twitter score by half. “Maybe they made a mistake in counting before” I told myself and kept following the score. I also noticed that Altmetric listed less than 20% of all the media outlets that reported my study and decided to do a small experiment. Altmetric allows submitting websites that they missed to I submitted one blog. Altmetric added this blog under “Misc,” which doesn’t get score. Few days later Altmetric exacted its revenge on questioning its formula and took away 30 of my precious doughnuts. On the positive end, the rising interest of the Twitter community gained me again 335 points from Tweets alone. It was time for a second experiment. This time, I sent Altmetric a list of 10 blogs that were missed. Once again I learned the cost of vigilance. Altmetric confiscated 50 of my doughnuts, wiping more twitter points and pushing me way below the 300 line. I cheered myself again that maybe the original Twitter count was incorrect, but I contact the tech support and asked why Altmetric lists less than 20% of the websites that mentioned my study? Altmetric tech support explained that the algorithm is going through some upgrades and agreed to look at the list of missing websites. By that point I was again the proud owner of 304 doughnuts just before the hungry vengeful Altmetric machine consumed 40 of them without adding a single website. As before, Altmetric suggested that tweets were erased assuming that I have no way to check that, except that I did.

Caught red handed

I recalled that one of my Twitter friends had 124 followers who retweeted about my study, as you can see below, this is nearly TWICE than what Altmetric reports that I currently have. Sloppy Altmetric didn’t bother to fix that number and only some of those 124 tweets were considered towards the 73 countTweets.png

In summary

Recently, homolog.us tweeted this and wrote “dedicated to the ‘altmetric’ God,” suggesting that social media impression do not mean truth. I think he nailed it. Altmetric is playing God evaluating our hard academic work, however this is a false God, a vengeful God, and a petty God that like all false, vengeful, and petty gods it is controlled by a small group of people sitting in some office who uprank or downrank papers based on their agenda, interests, and politics, not by the actual exposure of the paper.

Let us stop pretending that this Altmetric is an objective measure that deserves to be cited on websites and grant applications. Another valuable lesson from this experience is that we should continue improving the way in which we communicate our science to the public.

Altmetric’s response

Prior to the publication of this piece, I sent Altmetric a draft and invited them to respond. Altmetric founder, Mr. Euan Adie, was very quick to respond:

Sorry you’ve had a bad experience with Altmetric. We don’t deduct anything from the score for answering support queries, it’s an automatic process that follows this process.

The score calculation certainly is subjective and it sounds like you might have been affected by a bug at some point too, but actually most of the problems come from us trying to apply that calculation objectively. This is why some things go into the misc tab etc. which I know causes frustration.

Mr. Adie prompt response is much appreciated, however it is difficult to see how his explanation fits with my recent experience with Altmetric where each interaction with Altmetric resulted in quick deduction of points without actually fixing the problem. It is also difficult to believe that a bug would affect only my most recent paper and none of my other papers. Nonetheless, I wish to extend my best wishes to the Altmetric sick bots and hope for quick recovery from the bug.

Advertisements
This entry was posted in BLOG and tagged , , , , , , , . Bookmark the permalink.

5 Responses to On the fallacies of Altmetric OR how I fell asleep in Sheffield and woke up in Louisiana

  1. Euan Adie says:

    Hi Eran,

    Apologies again that you’ve had a bad experience. We certainly don’t change anybody’s scores to penalize them for writing in.

    In the case of this article:

    https://www.altmetric.com/details/6048538

    one issue is that significant numbers of the tweets are being removed by Twitter or the users in question. This is why you’re seeing a drop in the number of both tweets and the score.

    In total, I can see in our database that there have been 212 tweets / retweets about the article. However, we’ve received a “delete” notice for 115 of them from Twitter.

    In the majority of cases this has happened because Twitter has suspended the tweeter’s account, presumably because they were bots or violated T&Cs. Here’s a selection:

    https://twitter.com/SchrempMariette
    https://twitter.com/758Aytun
    https://twitter.com/731Altan
    https://twitter.com/725Dicle
    https://twitter.com/532Ataberk
    https://twitter.com/528Uzerli

    All of these accounts retweeted the tweet you’ve included the screenshot of.

    Another issue you might be hitting giving the subject area is private tweets: we don’t track tweets from protected accounts, only public ones. This is why you can sometimes see a higher number of retweets than are reported by Altmetric.

    Hope this helps clear things up. Happy to answer any questions here or by email.

    Thanks,

    Euan

    • eelhaik says:

      Dear Euan,

      Thank you for looking into the problem. Originally you have indicated that the low score
      is due to a bug. You now assert that about HALF the tweets were from accounts that were shut down as quickly as they tweeted about my study. I am in no position to validate or invalidate your claim, but this is a very concerning argument that only supports my concerns about the risks to the freedom of speech. Actually, one may argue that Altmetric should at least uphold the score those individuals achieved at the cost of losing their Twitter account.

      Even if this explanation is reasonable, it does not explain the periodicity in which Altmetric deduced my score upon contact nor why less than 20% of the news outlets mentioning my study (see full list in http://eranelhaik.staff.shef.ac.uk/Press.html) were considered even after Altmetric were provided with the missing links. I thereby maintain that Altmetric is a biased measure that does not properly reflect the attention a study has received. I have no intention of following this score anymore. However, feel free to email me when you feel that you have corrected all the bugs and that the Des et al. (2016) has a score that represents the attention it generated. I will be happy to publish an updated post.

      Thank you again for your prompt reply.

      • Euan Adie says:

        Hi Eran,

        Mmm, I thought there was a bug involved – but having looked at the underlying data now I don’t. I’m sorry we didn’t do this sooner.

        > I am in no position to validate or invalidate your claim, but this is a very concerning argument that only supports my concerns about the risks to the freedom of speech.

        Actually you can validate it somewhat. I noticed you subscribed to the tracking emails: do you still have any? You can see some of the deleted accounts there.

        e.g.
        26th Mar 2016 (“Ahmet Reyiz Yılmaz” and “Hülya” now suspended)
        23rd Apr 2016 (“Kaz — Diamond Dogs” now suspended)

        You can also search Twitter for the link to your paper: if you find any public ones that don’t appear on the Altmetric details page / aren’t in the count then definitely let me know (that really would be a bug).

        re: freedom of speech I can’t comment as I don’t know why the tweets or accounts were removed by Twitter. There are some guidelines here:

        https://support.twitter.com/articles/15790

        To reiterate, though, we don’t delete / filter out any data on our side. These are questions for Twitter, not us.

        > Actually, one may argue that Altmetric should at least uphold the score those individuals achieved at the cost of losing their Twitter account.

        You could make that argument – but with the original tweet and tweeter deleted, users can’t tell if the account was a paid for follower, a spam bot or a real person, or even validate that the tweet existed at all. Our primary rule for the score is that all of the data behind it has to be auditable.

        Best wishes,

        Euan

  2. eelhaik says:

    Ahmet Reyiz Yılmaz’s account (https://twitter.com/ARY_YILMAZ) is most certainly not suspended and never was. His last tweet was 37 minutes ago. I do not believe you are using/getting accurate information from Twitter.

  3. Pingback: Was Google Really Censoring Elhaik’s Khazar Research in 2013? « Homolog.us – Bioinformatics

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s