#plum analytics

[plumanalytics]

From Data to Application: New Metrics & the Research Workflow

Robin Champieux, the Scholarly Communications Librarian at OHSU, gave an excellent presentation at the ER&L Conference in Austin, TX a couple weeks ago. We’ve been getting a lot of requests for Robin’s presentation, and the slides are now available here. 

[plumanalytics]

Librarian Entrepreneur—Oxymoron?

Recently I was invited to speak at the University of Pittsburgh iSchool’s TEC Conference. “TEC” stands for “Technology. Entrepreneurship. Creativity.” My presentation, entitled “Librarian Entrepreneur—Oxymoron?” was a fun opportunity not only to discuss my experience co-founding Plum Analytics, but also to look back further to my start at the University of Washington iSchool and subsequent projects and products along the way.

I started by talking about my involvement in a couple of start-ups while getting my MLS degree during what became the dot com bubble. I received my degree in June 2000 just 2 months after the bubble burst and the investment dollars dried up. One of the start-ups I was involved with didn't receive the expected round of funding money so didn't get off the ground at all; the other went out of business in the 30-day period between the time I completed the website taxonomy project I worked on and when I was to be paid. I at least got to keep the branded pen!

Putting my start-up dreams on hold, I was hired by the late Lee Dirks to work in the Microsoft corporate library as Collections Management Librarian. Working for Linda Shaw for the succeeding five years, I helped manage the print and electronic collections at a time when the industry was undergoing really interesting change including: scholarly publishing moving to online access; new ways to collect, manage, and measure electronic resources; and the birth of ebooks. In my talk I showed what the Microsoft home page looked like on the week I started working there (h/t Wayback Machine):

After Google announced the plans to digitize books from 5 of the largest libraries in the world at the end of 2004 and launched Google Scholar a year later, Thiru Anandanpillai, a Product Planner at Microsoft, set in motion plans for Microsoft to compete with these products. Because of my experience working with scholarly publishers, Thiru reached out to me for help in understanding the landscape. A few months later I joined as the third member and only librarian to the just-formed team. We launched Microsoft Academic Search in 2006 and moved quickly to set up the scanning of books from publishers as well as important libraries such as the University of California System, the University of Toronto, and the British Library. This was my first “intrepreneurship” and it was an incredibly fun and exciting time. Unfortunately Microsoft abandoned the effort in May 2008. Seeing the writing on the wall, I left eleven months beforehand, but I am still reminded of that time when I see the continued scanning efforts at the University of Toronto using the 13 scanners that Microsoft donated:

Those scans, which are free from copyright restrictions, still flow to the Internet Archive and Hathi Trust.

And the British Library launched their 19th Century Collection which was the result of the scanning set up there at that time:

In 2009 Lee Dirks and Alex Wade revived Academic Search under the banner of Microsoft Research where they showcased some innovative clustering and author disambiguation features.

My second “intrepreneurship” started in 2008 when John Law tapped me as the Product Manager for what became the Summon Discovery Service. This was another exciting time building a new product from scratch with a great team. It was here that I met Andrea Michalek, who was the Director of Technology for Summon. Andrea and I left ProQuest in the last half of 2011 and we were interested in exploring the "entrepreneur" path next.

After we came up with the idea for Plum Analytics and gathered feedback from some well-respected librarians at the ALA Midwinter Conference in Dallas, Andrea and I founded Plum Analytics in January 2012. At the TEC conference I talked about how we came up with the name and logo for the company as well as building and selling the PlumX product that ultimately resulted in the acquisition of Plum Analytics by EBSCO in January 2014. I also walked through what we used to pitch potential funders to illustrate what investors are looking for in evaluating a start-up.

Many thanks to Mike Depew, the i3 Project Director at the iSchool, and Sheila Corral, the Chair of the Library and Information Science Program, for inviting me and hosting the event.

[plumanalytics]

New Research Comparing Different Altmetrics Tools

The term altmetrics means different things to different people. At Plum Analytics it means any metrics about any type of research output. It is common knowledge that the ubiquitous use of the Internet has made sharing everything, including research, simple. With all of the different ways to share output come different ways to understand the impact and stories of research. In other words, since research is no longer limited to print articles in print journals, we have new ways of looking at it to discover the stories and impact of research. This is where altmetrics plays a role.

Altmetrics challenge old ways of thinking; with this challenge come research opportunities. Recently, a group of researchers from Leibniz Institute for Science and Mathematics Education, University of St. Gallen, Leibniz Information Centre for Economics, Institute for the World Economy, and Leibniz Information Centre for Life Sciences studied the different altmetrics tools available (Jobmann et al., 2014).

The title of their report is “Altmetrics for large, multidisciplinary research groups: Comparison of current tools,” and it can be found here. To date, most of the altmetrics research has compared altmetrics to citation counts. What is interesting about this new research article is that it acknowledges that not all altmetrics are created equal and it compares the altmetrics providers to each other.

Here is the abstract:

Most altmetric studies compare how often a publication has been cited or mentioned on the Web. Yet, a closer look at altmetric analyses reveals that the altmetric tools employed and the social media platforms considered may have a significant effect on the available information and ensuing interpretation. Therefore, it is indicated to investigate and compare the various tools currently available for altmetric analyses and the social media platforms they draw upon. This paper will present results from a comparative altmetric analysis conducted employing four well-established altmetric services based on a broad, multidisciplinary sample of scientific publications. Our study reveals that for several data sources the coverage of findable publications on social media platforms and metric counts (impact) can vary across altmetric data providers. (Jobmann et al., 2014)

Background

This study looked at research articles from a variety of disciplines represented by the five institutes of the Leibniz Association. These are:

Humanities and Educational Research

Social Sciences/Spatial Research

Life Sciences

Natural Sciences/Engineering

Environmental Sciences

The study looked at four altmetrics providers:

ImpactStory

Altmetric Explorer

Plum Analytics

Webometric Analyst

They evaluated the metrics from these four providers across those five disciplines to determine how the providers compared with each other.

The researchers selected research articles published in 2011 and 2012 from the five Leibniz Association research institutes. After some manual validation and duplicate removal, the study covered 1,740 DOIs. The reason they selected to use DOIs is because all four altmetrics services support them.

Using the Tools

The article also commented on the ease of use of the altmetrics tools they used to conduct the research. The researchers had a variety of problems trying to use ImpactStory, including the unreliability of the delete functionality and capacity restrictions in uploading the DOIs. Webometric Analyst required a Microsoft Azure Account key, and it only ran on Microsoft systems. It, too, had capacity problems in that it restricts data download from Mendeley to 500 items per hour.

With respect to Altmetric Explorer, the researchers commented:

Altmetric Explorer itself can only process 50 identifiers at a time. Identifiers have to be numeric, thus, the link format, e.g., http://dx.doi.org, has to be removed from the data before analyses. If searching for articles with DOIs altmetric data has to be immediately exported, because the results cannot be saved. Only the search process with the respective settings can be saved under /my Workflow/. Therefore, Altmetric Explorer is only suitable for an immediate analysis with real-time data. (Jobmann et al., 2014)

The researchers did not have problems with Plum Analytics:

Plum Analytics offers interested institutions a free trial account. In addition, trial users can refer to an EBSCO customer service representative in case of questions. Plum allows for the upload of an extensive list of DOIs and generates downloadable search results in various data formats. No difficulties were encountered in the use of the service. (Jobmann et al., 2014)

Results

The researchers found a lot of variability for what kind of metrics different services covered, with Plum Analytics covering 26 different metrics and ImpactStory and Altmetrics Explorer each covering 12. Webometric Analyst has two versions: one that gets all of their data from Mendeley, and one that gets all of their data from Altmetric Explorer -- and these are mutually exclusive. The metrics from the version of Webometric Analyst that gets their data from Altmetric Explorer are identical to that service. And, as one would expect, the metrics from the version of Webometric Analyst that gets their data from Mendeley, has only Mendeley metrics and no others. Because of these factors, Webometric Analyst is only included in the study to a limited degree.

The researchers described their results:

Plum Analytics (total of DOIs found: 1,452) and Webometric Analyst (1,445 DOIs) appear to provide the best coverage of Mendeley, followed by ImpactStory (1,067 DOIs) (see Figure 3). The Altmetric Explorer only identifies a fraction of the DOIs identified by the other services (247 DOIs). Interestingly, Plum Analytics and ImpactStory generate quite similar data for publications from Sections D (Mathematics, Natural Sciences, Engineering) and E (Environmental Sciences), while the other sections appear underrepresented by ImpactStory. Twitter is best covered by Altmetric Explorer and ImpactStory, while Plum Analytics only generates data for one DOI (see Figure 4). Facebook, instead is best covered by Plum Analytics, with only occasional results by ImpactStory (see Figure 5). Wikipedia is equally covered by Plum Analytics and ImpactStory, but not represented by the other services.  (Jobmann et al., 2014)

Figure 3. Coverage of DOIs on Mendeley for each section found by the data providers. The Leibniz sections are a) Humanities and Educational Research, b) Economics, Social Sciences, Spatial Research, c) Life Sciences, d) Mathematics, Natural Sciences, Engineering, and e) Environmental Sciences (n=1,717)

Figure 4. Coverage of DOIs on Twitter for each section found by the data providers. The Leibniz sections are a) Humanities and Educational Research, b) Economics, Social Sciences, Spatial Research, c) Life Sciences, d) Mathematics, Natural Sciences, Engineering, and e) Environmental Sciences (n=1,717).

The study notes that Plum Analytics draws their Twitter from Topsy.com. After the acquisition of Topsy by Apple, the Twitter feed to Plum Analytics was turned off. Fortunately, Plum Analytics has now replaced their Twitter data and have a robust Twitter solution. (Read more about it here.) Today, the Twitter results for Plum Analytics for this set of articles would look very different.

Figure 5. Coverage of DOIs on Facebook for each section found by the data providers. The Leibniz sections are a) Humanities and Educational Research, b) Economics, Social Sciences, Spatial Research, c) Life Sciences, d) Mathematics, Natural Sciences, Engineering, and e) Environmental Sciences (n=1,717).

Conclusion

One of the things the researchers discovered was that for the same data sources the coverage varies in ways that are not always easily understood.

First, the data providers fetch altmetric data from different social media platforms and with varying detailedness (e.g., Plum Analytics retrieves Facebook likes, comments, and shares, whereas ImpactStory only collects number of shares). Out of the studied data providers, Plum Analytics registers the most metrics for the most platforms. (Jobmann et al., 2014)

Even when all services appear to be using the same metrics from the same supplier, they can still vary.

Second, the data providers differ in the number of DOIs they find on the social media platforms. Especially in the case of Mendeley, Altmetric Explorer performs worst in retrieving readership information for DOIs…Third, the altmetric impact values provided by Plum Analytics, ImpactStory, Altmetric Explorer, and Webometric Analyst for the publications in the data set also considerably deviate. Oftentimes it is the Altmetric Explorer search which results in lower impact values. (Jobmann et al., 2014)

Third, the altmetric impact values provided by Plum Analytics, ImpactStory, Altmetric Explorer, and Webometric Analyst for the publications in the data set also considerably deviate. Oftentimes it is the Altmetric Explorer search which results in lower impact values.

We encourage you to read this article to see even more results and analysis.

References

Jobmann, A., Hoffmann, C.P., Künne, S., Peters, I., Schmitz, J. & Wollnik-Korn, G. (2014). Altmetrics for large, multidisciplinary research groups: Comparison of current tools. Bibliometrie - Praxis und Forschung, 3, http://www.bibliometrie-pf.de/article/viewFile/205/258 [Accessed on February 12, 2015]

[plumanalytics]

New Capture Metrics Launched

PlumX tracks 5 categories of metrics:  usage, mentions, social media, citations, and captures.  Metrics that we categorize as captures are when an individual is interacting with research online and “captures” it for later.  Examples of capture metrics that are already a part of PlumX are:

Bookmarking the page on Delicious

Adding it to Elsevier’s Mendeley as a reader

Favoriting a slide presentation on Slideshare

Favoriting a video on YouTube

Subscribing to a channel on Vimeo or YouTube

Followers, forks, or watchers of source code on GitHub

All of these actions indicate that a user might want to later come back to this research.  For example, a researcher finds an article and adds it to their Mendeley reader or bookmarks it in a bookmark manager such as Delicious so they can go back and read it later.

In the scholarly space, studies have shown that capture metrics are a good leading indicator of future citations.  For example, In the research article When are readership counts as useful as citation counts? Scopus versus Mendeley for LIS journals  (Open access version here), the researchers Maflahi and Thelwall introduce their research study as:

This article investigates the influence of time on the number of Mendeley readers of an article through a theoretical discussion and an investigation into the relationship between counts of readers of, and citations to, four general Library and Information Science (LIS) journals. For this discipline, it takes about seven years for articles to attract as many Scopus citations as Mendeley readers and after this the Spearman correlation between readers and citers is stable at about 0.6 for all years. This suggests that Mendeley readership counts may be useful impact indicators for both newer and older articles.

This week, we added new capture metrics to PlumX that continue to expand this important category to include data from EBSCO’s EDS discovery, EBSCOhost, and eBook platforms.  These new captures are calculated per artifact, and measure when a user emails, exports, prints, or saves an artifact’s citations.  More specially these actions are defined below:

Emails: The number of times an artifact’s citation, abstract and full text (if available) have been emailed

Exports: The number of times an artifact’s citation has been exported direct to bibliographic management tools or as file downloads

Print-outs: The number of times an artifact’s citation/abstract and HTML full text (if available) have been printed

Saves: The number of times an artifact’s citation/abstract and HTML full text (if available) have been saved

Some of these metrics comprise a part of the “hidden web.”  Being able to calculate the number of times users on a platform email or print out a citation, is a glimpse into what work is being saved for later more than others.

The metrics we can collect are of course only a proxy for ALL places and times that research has been saved for later.  We’ve expanded these new capture types using data from EBSCO, and will expand our sources over time as others become available to us.

Given the scale of this data we have just exposed, some PlumX customers will see dramatic increases in capture counts in their profiles. This is one reason why it's so challenging to distill alternative research metrics down to a single number: what happens when new metrics sources start to get collected?  Our approach is now, as it always has been, to collect as comprehensive a collection of indicators as possible.  Using the data should always be done in context and through benchmarking against other items of interest.

[plumanalytics]

New Artifact Types Supported in PlumX

We recently updated the types of artifacts that we support to better reflect the diversity of research documents that our customers want to track. Giving customers flexibility in how they manage their research output collection is a hallmark of PlumX, and is critical to being able to do an accurate assessment. For example, newsletters could have far different metrics profiles than blog posts, so being able to classify them into their own artifact groups ensures a true apples-to-apples comparison between the artifacts within each one.

The new artifact types are:

Blog post

Bibliography

Abstract

Review

Reference

Web Resource

Newsletter

You can see a complete list of all PlumX metrics and artifact types on the PlumX metrics page. To request more information about Plum Analytics go here.

[plumanalytics]

PlumX Review in JMLA

Robin Champieux, Scholarly Communication Librarian, Oregon Health & Science University (OHSU), recently reviewed PlumX for the Journal of the Medical Library Association (JMLA).

In this review, Robin does an excellent job of describing PlumX and what it does. Her conclusion is:

PlumX is a powerful tool: both the data and the user interface can support multiple use cases and research questions. For example, Oregon Health & Science University (OHSU) Library is exploring how to use PlumX to help individual researchers uncover the full impact of their work, especially among different populations. OHSU wants to enable researchers to tell more nuanced stories about their science, especially in the context of funding proposals and the new biosketch format that the National Institutes of Health is piloting.

To read the whole review go here.

To request more information about PlumX go here.

[plumanalytics]

Come See Plum Analytics in Person

ALA Midwinter Meeting

If you are going to Chicago this weekend for ALA Midwinter, here is where you can see Mike Buschman, Co-Founder of Plum Analytics:

"Moving Beyond Altmetrics - Measuring Research Impact in the Modern Era." Mike  will look at how to use a variety of new metrics to tell the stories and answer the questions about your research. The session will take place in Jackson Park/CC 10AB at the Hyatt Regency McCormick. Saturday 8:30 am

EBSCO Booth #2211. See an overview of PlumX. Sunday 9:30 am

Or, drop by booth #2211 any time for a PlumX demo or to talk about modern metrics.

ALIA Information Online

For our readers in Australia, come meet the newest Plum Analytics Specialist, Mike MacKinnon, in the EBSCO booth #23 & 24.

[plumanalytics]

The Charleston Advisor Interview with Mike Buschman

Mike Buschman, Co-Founder and Chief Product Office of Plum Analytics was recently interviewed for The Charleston Advisor.

Some highlights from the interview include:

"It is true that traditionally the term “altmetrics” refers to Social Media… As such, Plum Analytics has never fit perfectly into this category definition… which is why you see people expanding the definition of altmetrics to come closer to our notion of ‘not altmetrics—ALL metrics.’"

"The metrics and analytics uncover the qualitative stories about what is happening with research output. Revealing the over-hyped or ‘popular’ research is another example of a story that the metrics allude to. For instance, if you look at a PlumX report on the output of a researcher and see that one article stands out in its social media interactions, you do not just take the number and say that that is necessarily good—you look behind the numbers and see that the paper was controversial, over-hyped, slammed by colleagues, etc."

To read the entire interview go here.

To request more information about Plum Analytics go here.