Plum Analytics - Tell the Story.

Stay up to date on the latest at Plum Analytics

[plumanalytics]

CHUC Using PlumX to Measure and Analyze Research Impact

The Centro Hospitalar e Universitário de Coimbra/Coimbra Hospital and University Centre (CHUC) in Portugal is the first hospital in southern Europe to implement PlumX to help measure and analyze the impact of their medical research. CHUC is the largest Portuguese hospital of its kind, featuring a comprehensive research, teaching and patient-centered care approach. .

Dr. Helena Donato, Documentation Service Director at CHUC, said that the hospital has sought ways to track research impact other than just ‘traditional’ research metrics. She envisions using PlumX to gauge mentions and usage of CHUC research in social media and other online avenues – as they happen. “We want to monitor in real time the online activity around our scientific research by tracking metrics such as downloads, number of readers, discussion and comments in social networks,” Dr. Donato said, noting that PlumX is the best tool with which to evaluate the scientific impact of CHUC’s hospital research center without the delays associated with other research impact tools. “Measuring only traditional citations and the impact factor doesn’t give a complete picture of the impact of clinical medical research.”

Dr. Donato and her colleagues see great value in the ability of PlumX metrics to raise the international profile of CHUC and afford exposure to its research output quickly, due to the instantaneous feedback the metrics provide for research articles. “We want to achieve greater international visibility and recognition by our peers. Using PlumX metrics, we expect to disseminate the impact of our clinical research more quickly and effectively.”

To read the press release go here.

[plumanalytics]

Announcing the PlumX Grant Widget

We work hard to make sure that our customers can use the metrics we gather about their research where they want to see them. Our newest addition to our growing list of widgets is the Grant Widget. This widget allows grantmakers and other funders and foundations to embed important information about grants that is contained in PlumX into their web pages and systems.

Below is an example of an grant from Autism Speaks. The name of the grant is “Identifying Genetic Variants on the Y Chromosome of Males with Autism.” This grant has the following research output associated with it:

25 Articles

3 News Stories

1 Blog

Here is the widget:

Notice that in the list of research output (artifacts) you can see the Plum Print for each one so at a quick glance you can tell something about what is happening with this research. 

From the widget, you can click the “see details” link to see the Grant Detail page in PlumX.

If you scroll down this page, it also has a table of the output. Below is an example of that.

You can click on any of the titles of the output to see the detailed artifact page.

This grant widget joins our growing list of widgets:

Artifact Widget (Plum Print)

Researcher Widget

Group Widget

To read about our other widgets go to this previous blog post.

[plumanalytics]

‘Impact’, the Library and Managing Research Excellence in UK Higher Education

Presenting a guest post from Steve Giannoni, Director of Sales (Academic & Govt) UK & Ireland, EBSCO Information Services, on his thoughts on the library's role with the research office.

Having spent almost 10 years of my EBSCO-life working predominantly with Academic Libraries, last week I found myself at a conference attended by over 700 delegates with only 3 people containing the word ‘Librarian’ or ‘Library’ in their job title. I was at my inaugural, and the 25th anniversary, ARMA conference which amongst other things, brings together those responsible for managing and administering the research output of Higher Education Institutions (HEIs) in the UK.

It’s been a long time since I have been exposed to a world beyond library-centric discourse, where topics such as discovery, eBooks, journals, databases and other end-user primed services acquired by information professionals, consumed by students, researchers and academics were set aside in favour of discussions about ‘research output’, ‘quantitative metrics’, ‘grant-funding compliance’ and the ‘REF’ (Research Excellence Framework).

Although the conference was attended by directors, managers and administrators of the research office, the overwhelming agreement from the dozens of people I met with at the event, was that Libraries and Library professionals absolutely have a part to play in the support of research and the data-wrangling necessary to help the research office evidence what attention, engagement and impact the research output of the institution has on the outside world.

Libraries have an inherent expertise in curating data in this digital information age and I believe this competence contributes to the reason why we see more and more ‘research support’ librarian roles coming to the fore. Libraries are increasingly being recognised as important campus partners, aiding the institution’s operational response to the requirements of national assessment processes, not least for example (in the UK), in preparation for the next REF (perhaps in 2020?).

Under the shadow of OA policies which internally and externally mandate newly accepted articles be deposited in the Institutional Repository, the Library increasingly sees its efforts being cross pollinated with the research offices’ need to manage (compliantly) the published output in order to meet funder’s requirements. In Library circles, discussions about the relationship between OA publishing and research impact are set to increase and I expect Libraries to be more involved in the research office’s activities as they attempt to showcase their competencies and enhance their presence within the institution. Fledging OA University Press initiatives in the UK and North America are drawing on library knowledge and experience to take them forward – further evidence of a changing role.

Back at the ARMA Conference, there were many parallel sessions spanning the 2-day event which focused on impact, metrics and research funding. A common theme emerged in that as competition grows in the pursuit of grants and funding, there appears to be an ever increasing interest in applying organisations to look beyond Citation metrics as a measure of the past success of an institution’s research output. Has the research world suddenly moved away from Citations as a means of impact measurement and replaced the ‘Gold-Standard’ with a newer, hipper alternative?

Well, no. This is where PlumX comes in. Referred to as more of an ‘All-metrics’ rather than Alt-metrics tool, PlumX provides both Citation metrics alongside other quantitative ways of tracking the output of a researcher, a department an institution or alternatively, research that has been produced as a direct result of funding from an external source, such as a research council.  It’s no wonder that the funders of research are themselves investing in PlumX, so that they can track, measure and analyse the various ways in which the research they are funding is being interacted with. An example of this is Autism Speaks who in July 2014, began using PlumX for this very reason.

PlumX can be used as supporting evidence within a narrative (e.g. case studies) when applying for future research grants now that social, economic and political engagement with researcher’s outputs can be interpreted in ways not before possible when looking at Citations alone.

I for one am excited about the direction research assessment and the measurement of research outputs in the UK are taking and I expect as OA takes hold in 2016, we’ll be seeing more and more reference to All-metrics, rather than conversations surrounding Alt-metrics (as alternatives to Citations) as has been the case in previous years.

[plumanalytics]

Integrating PlumX Metrics in Your Repository: An Open Repositories Poster

We are at the Open Repositories meeting in Indianapolis, IN this week and are talking to many institutions about their repositories and integrating modern metrics into them.

We participated in the poster session on Tuesday evening with our poster, “Widget Integration in Open Repositories:  Real World Experiences with the PlumX Widget.”

This poster contains five stories about PlumX and integration:

Get Started

Researcher Profiles

Artifact Identity Resolution

Analyze

Integrate PlumX Where You Need It

Get Started

This section depicts our ability to start with your repository and mirror the structure in PlumX so you can see your metrics in the way that you want. A good example of this is the way that PlumX uses DSpace communities. You can read more about that in this blog post. Below is an example of structures that might be in PlumX.

Researcher Profiles

Here we are talking about automatically creating researcher profiles from any existing system. This includes open systems such as ORCID or Harvard Profile System as well as many other researcher profiling systems.

Artifact Identity Resolution

This section refers to the logic PlumX uses to find all relevant versions of a research article or artifact. For example, you might integrate a DOI from your repository into PlumX. PlumX will take the DOI and find other associated IDs such as a PMID so that metrics for multiple versions of the article can be found. PlumX can even start with a URL to and find DOIs and other IDs.

Analyze

Once all of the versions of the artifacts are found and the metrics are gathered, PlumX creates a variety of analytic reports. These reports are downloadable as images or into PDFs.

Integrate PlumX Where You Need It

This is one of the most exciting parts of how PlumX integrates. PlumX widgets are a way to bring metrics into other important systems including repositories. PlumX supports multiple kinds of widgets, most noticeably the Plum Print. The Plum Print is the article or artifact level widget that is a visual representation of the metrics. Below is an example.

There are several versions of this Plum Print widget as well as author and group widgets. You can read more about these widgets in this blog post.

To view the poster in its entirety please go here.

[plumanalytics]

PlumX Leverages DSpace Communities for Simple Integration

The integration between PlumX and DSpace makes it simple for DSpace customers to get started quickly to have PlumX metrics, dashboards and analytic reports.

When we work with new customers to implement PlumX we always start with the best place for them. One good example is their Institutional Repository. When a customer has already gathered their research output into an IR we work with them to create the best integration for both initial setup and ongoing maintenance.

DSpace offers a function known as communities that allows users to divide their research into groups and hierarchies. These communities usually reflect their institution, for example, schools and departments. PlumX mirrors the communities to automatically create PlumX research impact dashboards for each DSpace community.

Below is an example from Tomas Bata University in Zlin in the Czech Republic. In their DSpace implementation they have four top communities, each with several communities underneath in the hierarchy. You can see this in the picture below.

In PlumX, the top four communities are mirrored.

When you click on any of these in PlumX you drill down to the next level in the community hierarchy.

For each level you have all of the features and functions of a PlumX Dashboard, including:

List of research output or artifacts

Table of metrics for all artifacts

Analytic reports

Ability to export the data

Data filters 

Once the community and hierarchies are mirrored in PlumX, PlumX uses the OAI protocol for metadata harvesting to get all of the articles and other research output from the DSpace repository. Then, PlumX gathers the metrics and creates the reports. The research output does not need a DOI in order for PlumX to find metrics.

Once the metrics are gathered you can embed the PlumX metrics widget into the DSpace repository. Some PlumX customers have reported that by providing all of these metrics within the IR, authors have seen a benefit to depositing their research because they can go back to the IR to see the reach and impact of their work. We call this the “carrot” approach to IR compliance. We wrote a paper about it that you can access from this blog post.

Below is an example of an article in the University of Manitoba DSpace repository. At the bottom of this article, you can see PlumX metrics in one of the available widgets (there are several).

A closer look at the PlumX metrics:

Notice that you can see metrics for different versions of the article, not just the one in the DSpace repository.

From DSpace you can link to the record in PlumX where you have access to more features such as transparent detail behind the metric numbers and the list of tweets and blog posts about the article or artifact.

We love working with customers and their institutional repositories. We have implemented a variety of IRs including:

DSpace

Fedora

ePrints

bepress Digital Commons and Selected Works

Custom IRs

We work closely with all of our new customers and are constantly learning new things that add to our repertoire of tools for IR integration. 

Starting with the IR is an easy way to get started with modern metrics and can provide an incentive for your researchers to deposit research. 

[plumanalytics]

Panel Talks About New Ways to Measure Scholarly Impact

The Association to Advance Collegiate Schools of Business (AACSB) recently held their International Conference & Annual Meeting (ICAM). Plum Analytics co-founder, Mike Buschman, was invited to be a panelist on a panel titled, “Enabling Impactful Scholarly Research” with Eric Bradlow, Vice Dean and Director, Wharton Doctoral Programs, The Wharton School, University of Pennsylvania and Gregory J. Gordon, President, Social Science Research Network. The panel was moderated by Dan LeClair, Executive Vice President and Chief Operating Officer, AACSB International.

According to LeClair, this was one of the most popular sessions at the conference. LeClair summarized the session. One of the things he said in his summary:

As research outputs take on new forms, including pure-online journals, recorded virtual conferences, and software code, and as people begin to use research in different ways, technology has been enabling us to measure usage, influence, and impact accordingly. With help from new companies, we are finding ways, for example, to track book utilization by chapter, document when a piece of software is used or built on, and register each time an article appears in a doctoral seminar reading list. These alternative metrics will help business schools to demonstrate that their research makes a difference by providing a richer set of tools beyond citation analysis, and to look beyond the current view that refereed journal publications are the ultimate “end game.”

To read LeClair’s entire summary go here.

[plumanalytics]

New Easy to Use Artifact-Level Metrics Page

We are happy to announce a new artifact-level metrics page. An artifact is any piece of research output. Some of the most popular artifacts include:

Article

Blog Post

Book

Source Code

Theses/Dissertation

Video

Below is the new Artifact-Level Metrics page. This is the largest redesign of PlumX to date. Since we launched PlumX three years ago, we have interacted with 100s of institutions in dozens of countries. We took all of their feedback to heart to enhance our product.  

This page has several improvements to help you uncover and tell the stories of your research output. 

First, we are now summarizing the metrics in each category. Much like the Plum Print, this will make it easier for you to understand at a glance what is happening in each category.

For example, in this article you can see a metric for the total usage, with a breakdown of each sub-category. This is an easier way to digest these numbers. For full transparency, you can still easily expand the category to see all of the detail and where possible access the underlying data.

Another important new feature of this design is the exposure of the latest tweet and blog post about the artifact. 

Above is an example of this article with the most recent tweet highlighted. You can easily view all of the tweets about this article with a click of the button.

We do a similar thing with blog posts.

Go here to read more about how we handle Tweets, and here for Blog Posts.

From the beginning we knew that seeing metrics for individual articles would never be enough for institutions to be able to really uncover the stories about their research. So, PlumX was built with the ability for customers to define and import their own groups and hierarchies. Here are a few examples:

On the new PlumX artifact page these groups and hierarchies are better exposed.

In the example below this researcher from the University of Pittsburgh is part of the faculty in the School of Medicine. This research is in the areas of Cell Biology and Molecular Physiology and Pediatrics. You can then easily navigate to these other areas from this article to see related research and researchers.

We are proud of our new design. Let us know what you think by emailing us or sending us a tweet. 

[plumanalytics]

PlumX Expands “Mentions” Category by Tracking Blogs

Blogs are an important part of today's scholarly communication ecosystem. Researchers often write about their own or others’ research, share data and carry on conversations that push their field forward. This is why we’ve been working hard to expand our coverage of blog mentions in PlumX. 

PlumX tracks five categories of metrics -- usage, captures, mentions, social media and citations (go here for more information). Mentions is the category where we uncover and track the conversations about the research. Blogs are a new source of Mentions metrics. And, because Mentions are buried in thousands of disparate sites that may not be focused on research topics all the time, these can be difficult metrics to find.

When a blog post refers to specific research output we will collect information about the post, including a snippet, to give you an easy way to see the blog posts in context within PlumX. We will also provide a link to the post itself.

Here is an example of an article we track in the field of economics.

You can see that this article has been referenced in two different blogs. By clicking on the hyperlinked number you are taken to a PlumX page with the title and snippet of each post.

From here you can click on the title in PlumX to see the blog post itself. 

Another interesting Mention that PlumX reveals about this article is in a Wikipedia article about Agency Cost.

PlumX exposes the places this article was referenced and used, helping to uncover the stories of the research.

[plumanalytics]

Dublin Business School Implements PlumX

The Dublin Business School (DBS), Ireland’s largest independent third level college, is the first institution in Ireland to take advantage of the modern metrics provided by PlumX to help tell the stories of their faculty’s research output and promote its value to current and prospective students.

DBS Head librarian, Marie O’Neill, described the reason for the selection, “PlumX provides DBS with the opportunity to publicly showcase the research output and impact of the College. Demonstrating the research activity of faculty is an integral aspect of progamme accreditation and validation, and PlumX offers transparency to quality assurance agencies in terms of showcasing - on open access - the research output and impact of an institution.”

PlumX is an integral part of the institutional repository initiative at DBS. They plan to integrate PlumX into the IR so that the impact of the research output in the repository can be measured. “The software provides a wider range of research impact metrics than is available from institutional repositories alone, such as references to scholarly work in social media,” O’Neill said.

DBS also plans to use PlumX to assist in marketing and accreditation for the College. O’Neill stated, “PlumX has huge marketing value for the institution. It will also be critical in forging collaborative relationships and in the context of institutional review and programme review by Quality and Qualifications Ireland.”

To read the full press release go here.

[plumanalytics]

CABI and PlumX Partner in Pilot Program

Modern Metrics Will Measure CABI Authors’ Research Output in Innovative Ways

We are excited about our new partnership with CABI. CABI is an international not-for-profit organization that improves people’s lives by providing information and applying scientific expertise to solve problems in agriculture and the environment. 

CABI will use PlumX to showcase the recent work of their authors on CAB Direct, their science research portal. The portal will utilize the Plum Print, so users can see clear feedback about how articles and books are consumed at a glance.

Caption: An example of the Plum Print

To read the press release go here.

[plumanalytics]

Metrics Interest Growing - A Glimpse into a Week of Plum Analytics

It was just over three years ago that we formed Plum Analytics with the dream of helping the world better understand what was going on with research, particularly recent research. At the time there was just a small community interested in this. 

Over the last three years we’ve been well aware that this has been changing, but, it wasn’t until I sat down to make sure that we are prepared for next week that I realized how much the world is interested in utilizing modern metrics. We are involved in five different events with nine different speaking topics.

We are really excited to share the possibilities of what can happen when you broaden your view of the impact of research with modern metrics with such diverse groups of people.

Here is a list of where we will be speaking. If you are attending any of these conferences please reach out to the Plum Analytics speaker.

Advancing Research Communication & Scholarship

“Using Modern Metrics to Tell the Stories of Research” A workshop Sunday, 4/26, 3:00 - 5:00 pm EDT Andrea Michalek

“New Metrics: Funding Agencies & Academic Research Institutions Working Together” A presentation Monday, 4/27, 3:40 - 5:50 pm EDT Ed Clayton & Andrea Michalek

“Community Development of Standards for Alternative Assessment Metrics” A panel discussion Monday, 4/27, 5:15 - 6:40 pm EDT Andrea Michalek, Todd Carpenter (NISO), Jamie R. Liu (ACS) and Sara Rouhi (Altmetric)

“Tell the Story: Use Modern Metrics to Discover the Hidden Stories of Your Research” A 24x7 talk Tuesday, 4/28, 1:10 pm EDT Andrea Michalek

National Organization of Research Development Professionals

“Get the Grants You Deserve: Using Modern Metrics in Pre-Award Workflows” A presentation & demonstration Thursday, 4/30, 3:45 pm EDT Great Falls Room Mike Buschman

“Get the Grants You Deserve: PlumX and Modern Metrics” A poster

International Society for Medical Publication Professionals

“Innovation in Metrics: Capturing the Full Impact of Publications” A panel discussion Wednesday, 4/29, 8:00 am EDT Mike Buschman, Bhakti Kshatryi (Novartis) and Tom Rees (PAREXEL International), Sarah L. Feeny (Complete Medical Communications)

Association to Advance Collegiate Schools of Business

“Enabling Impactful Scholarly Research” A panel discussion Monday, 4/27, 1:45 pm EDT Mike Buschman, Eric Bradlow (The Wharton School, University of Pennsylvania) and Gregory J. Gordon (SSRN), Dan LeClair (AACSB)

National Information Standards Organization Virtual Conference

“NISO Altmetrics Project: Update from 3 Working Groups” Wednesday, 4/27, 4:00 pm EDT Mike Showalter (Plum Analytics), Mike Taylor (Elsevier), Martin Fenner (PLOS), Todd Carpenter (NISO)

Computers in Libraries

“Altmetrics: Meaningful Metrics in Management” A panel discussion Wednesday, 4/27, 2:30 pm EDT Maryellen Sims (Plum Analytics), Richard P. Hulser (Natural History Museum of LA County), Dr. Sharon Q. Yang (Rider University), Patricia H. Dawson (Rider University)

[plumanalytics]

Visions of the Future for Academic Publishing

This was the title of a recent blog post on the BioMedCentral blog. The authors, talk about the history of academic publishing, the massive changes that have taken place in the industry, and what the future may hold. 

What we found particularly interesting was the part of the blog post called “Measuring the impact of research, not journals and articles.” 

Measuring the impact of research, not journals and articles Diana Marshall, Senior Managing Editor

Here is the excerpt:

People like measuring things, establishing order, categorising and ranking. The Impact Factor is an established journal-level measure of citations. Every year journals, editors, authors and institutes prepare to receive new Impact Factors which might affect their journal strategy or where they publish.

There is debate about the value of Impact Factors and whether other measures better reflect the impact of a piece of research. Should factors such as public engagement, reproducibility or the impact on policies be considered when measuring the impact of a piece of research?

Article-level and alternative metrics such as Altmetrics are now available and widely used, as well as platforms like Impact Story and Plum Analytics, which allow authors and institutions to look at the impact of their research in a new way. In future, metrics will increasingly take into account more than simply citations but other measures such as how the data in an article is used and cited, reproducibility, applications of the work in places such as policies and software, and engagement or social media impact.

With increasing attention to the impact of the original article – and indeed the impact of the research itself rather than its published summary, could journal-level metrics in future be entirely redundant? Publishers have an important role in facilitating better metrics and enabling authors to look at metrics across all their outputs.

The author also asks this question:

Should factors such as public engagement, reproducibility or the impact on policies be considered when measuring the impact of a piece of research?

You can read the whole blog post here.

[plumanalytics]

From Data to Application: New Metrics & the Research Workflow

Robin Champieux, the Scholarly Communications Librarian at OHSU, gave an excellent presentation at the ER&L Conference in Austin, TX a couple weeks ago. We’ve been getting a lot of requests for Robin’s presentation, and the slides are now available here. 

[plumanalytics]

Librarian Entrepreneur—Oxymoron?

Recently I was invited to speak at the University of Pittsburgh iSchool’s TEC Conference. “TEC” stands for “Technology. Entrepreneurship. Creativity.” My presentation, entitled “Librarian Entrepreneur—Oxymoron?” was a fun opportunity not only to discuss my experience co-founding Plum Analytics, but also to look back further to my start at the University of Washington iSchool and subsequent projects and products along the way.

I started by talking about my involvement in a couple of start-ups while getting my MLS degree during what became the dot com bubble. I received my degree in June 2000 just 2 months after the bubble burst and the investment dollars dried up. One of the start-ups I was involved with didn't receive the expected round of funding money so didn't get off the ground at all; the other went out of business in the 30-day period between the time I completed the website taxonomy project I worked on and when I was to be paid. I at least got to keep the branded pen!

Putting my start-up dreams on hold, I was hired by the late Lee Dirks to work in the Microsoft corporate library as Collections Management Librarian. Working for Linda Shaw for the succeeding five years, I helped manage the print and electronic collections at a time when the industry was undergoing really interesting change including: scholarly publishing moving to online access; new ways to collect, manage, and measure electronic resources; and the birth of ebooks. In my talk I showed what the Microsoft home page looked like on the week I started working there (h/t Wayback Machine):

After Google announced the plans to digitize books from 5 of the largest libraries in the world at the end of 2004 and launched Google Scholar a year later, Thiru Anandanpillai, a Product Planner at Microsoft, set in motion plans for Microsoft to compete with these products. Because of my experience working with scholarly publishers, Thiru reached out to me for help in understanding the landscape. A few months later I joined as the third member and only librarian to the just-formed team. We launched Microsoft Academic Search in 2006 and moved quickly to set up the scanning of books from publishers as well as important libraries such as the University of California System, the University of Toronto, and the British Library. This was my first “intrepreneurship” and it was an incredibly fun and exciting time. Unfortunately Microsoft abandoned the effort in May 2008. Seeing the writing on the wall, I left eleven months beforehand, but I am still reminded of that time when I see the continued scanning efforts at the University of Toronto using the 13 scanners that Microsoft donated:

Those scans, which are free from copyright restrictions, still flow to the Internet Archive and Hathi Trust.

And the British Library launched their 19th Century Collection which was the result of the scanning set up there at that time:

In 2009 Lee Dirks and Alex Wade revived Academic Search under the banner of Microsoft Research where they showcased some innovative clustering and author disambiguation features.

My second “intrepreneurship” started in 2008 when John Law tapped me as the Product Manager for what became the Summon Discovery Service. This was another exciting time building a new product from scratch with a great team. It was here that I met Andrea Michalek, who was the Director of Technology for Summon. Andrea and I left ProQuest in the last half of 2011 and we were interested in exploring the "entrepreneur" path next.

After we came up with the idea for Plum Analytics and gathered feedback from some well-respected librarians at the ALA Midwinter Conference in Dallas, Andrea and I founded Plum Analytics in January 2012. At the TEC conference I talked about how we came up with the name and logo for the company as well as building and selling the PlumX product that ultimately resulted in the acquisition of Plum Analytics by EBSCO in January 2014. I also walked through what we used to pitch potential funders to illustrate what investors are looking for in evaluating a start-up.

Many thanks to Mike Depew, the i3 Project Director at the iSchool, and Sheila Corral, the Chair of the Library and Information Science Program, for inviting me and hosting the event.

[plumanalytics]

New Research Comparing Different Altmetrics Tools

The term altmetrics means different things to different people. At Plum Analytics it means any metrics about any type of research output. It is common knowledge that the ubiquitous use of the Internet has made sharing everything, including research, simple. With all of the different ways to share output come different ways to understand the impact and stories of research. In other words, since research is no longer limited to print articles in print journals, we have new ways of looking at it to discover the stories and impact of research. This is where altmetrics plays a role.

Altmetrics challenge old ways of thinking; with this challenge come research opportunities. Recently, a group of researchers from Leibniz Institute for Science and Mathematics Education, University of St. Gallen, Leibniz Information Centre for Economics, Institute for the World Economy, and Leibniz Information Centre for Life Sciences studied the different altmetrics tools available (Jobmann et al., 2014).

The title of their report is “Altmetrics for large, multidisciplinary research groups: Comparison of current tools,” and it can be found here. To date, most of the altmetrics research has compared altmetrics to citation counts. What is interesting about this new research article is that it acknowledges that not all altmetrics are created equal and it compares the altmetrics providers to each other.

Here is the abstract:

Most altmetric studies compare how often a publication has been cited or mentioned on the Web. Yet, a closer look at altmetric analyses reveals that the altmetric tools employed and the social media platforms considered may have a significant effect on the available information and ensuing interpretation. Therefore, it is indicated to investigate and compare the various tools currently available for altmetric analyses and the social media platforms they draw upon. This paper will present results from a comparative altmetric analysis conducted employing four well-established altmetric services based on a broad, multidisciplinary sample of scientific publications. Our study reveals that for several data sources the coverage of findable publications on social media platforms and metric counts (impact) can vary across altmetric data providers. (Jobmann et al., 2014)

Background

This study looked at research articles from a variety of disciplines represented by the five institutes of the Leibniz Association. These are:

Humanities and Educational Research

Social Sciences/Spatial Research

Life Sciences

Natural Sciences/Engineering

Environmental Sciences

The study looked at four altmetrics providers:

ImpactStory

Altmetric Explorer

Plum Analytics

Webometric Analyst

They evaluated the metrics from these four providers across those five disciplines to determine how the providers compared with each other.

The researchers selected research articles published in 2011 and 2012 from the five Leibniz Association research institutes. After some manual validation and duplicate removal, the study covered 1,740 DOIs. The reason they selected to use DOIs is because all four altmetrics services support them.

Using the Tools

The article also commented on the ease of use of the altmetrics tools they used to conduct the research. The researchers had a variety of problems trying to use ImpactStory, including the unreliability of the delete functionality and capacity restrictions in uploading the DOIs. Webometric Analyst required a Microsoft Azure Account key, and it only ran on Microsoft systems. It, too, had capacity problems in that it restricts data download from Mendeley to 500 items per hour.

With respect to Altmetric Explorer, the researchers commented:

Altmetric Explorer itself can only process 50 identifiers at a time. Identifiers have to be numeric, thus, the link format, e.g., http://dx.doi.org, has to be removed from the data before analyses. If searching for articles with DOIs altmetric data has to be immediately exported, because the results cannot be saved. Only the search process with the respective settings can be saved under /my Workflow/. Therefore, Altmetric Explorer is only suitable for an immediate analysis with real-time data. (Jobmann et al., 2014)

The researchers did not have problems with Plum Analytics:

Plum Analytics offers interested institutions a free trial account. In addition, trial users can refer to an EBSCO customer service representative in case of questions. Plum allows for the upload of an extensive list of DOIs and generates downloadable search results in various data formats. No difficulties were encountered in the use of the service. (Jobmann et al., 2014)

Results

The researchers found a lot of variability for what kind of metrics different services covered, with Plum Analytics covering 26 different metrics and ImpactStory and Altmetrics Explorer each covering 12. Webometric Analyst has two versions: one that gets all of their data from Mendeley, and one that gets all of their data from Altmetric Explorer -- and these are mutually exclusive. The metrics from the version of Webometric Analyst that gets their data from Altmetric Explorer are identical to that service. And, as one would expect, the metrics from the version of Webometric Analyst that gets their data from Mendeley, has only Mendeley metrics and no others. Because of these factors, Webometric Analyst is only included in the study to a limited degree.

The researchers described their results:

Plum Analytics (total of DOIs found: 1,452) and Webometric Analyst (1,445 DOIs) appear to provide the best coverage of Mendeley, followed by ImpactStory (1,067 DOIs) (see Figure 3). The Altmetric Explorer only identifies a fraction of the DOIs identified by the other services (247 DOIs). Interestingly, Plum Analytics and ImpactStory generate quite similar data for publications from Sections D (Mathematics, Natural Sciences, Engineering) and E (Environmental Sciences), while the other sections appear underrepresented by ImpactStory. Twitter is best covered by Altmetric Explorer and ImpactStory, while Plum Analytics only generates data for one DOI (see Figure 4). Facebook, instead is best covered by Plum Analytics, with only occasional results by ImpactStory (see Figure 5). Wikipedia is equally covered by Plum Analytics and ImpactStory, but not represented by the other services.  (Jobmann et al., 2014)

Figure 3. Coverage of DOIs on Mendeley for each section found by the data providers. The Leibniz sections are a) Humanities and Educational Research, b) Economics, Social Sciences, Spatial Research, c) Life Sciences, d) Mathematics, Natural Sciences, Engineering, and e) Environmental Sciences (n=1,717)

Figure 4. Coverage of DOIs on Twitter for each section found by the data providers. The Leibniz sections are a) Humanities and Educational Research, b) Economics, Social Sciences, Spatial Research, c) Life Sciences, d) Mathematics, Natural Sciences, Engineering, and e) Environmental Sciences (n=1,717).

The study notes that Plum Analytics draws their Twitter from Topsy.com. After the acquisition of Topsy by Apple, the Twitter feed to Plum Analytics was turned off. Fortunately, Plum Analytics has now replaced their Twitter data and have a robust Twitter solution. (Read more about it here.) Today, the Twitter results for Plum Analytics for this set of articles would look very different.

Figure 5. Coverage of DOIs on Facebook for each section found by the data providers. The Leibniz sections are a) Humanities and Educational Research, b) Economics, Social Sciences, Spatial Research, c) Life Sciences, d) Mathematics, Natural Sciences, Engineering, and e) Environmental Sciences (n=1,717).

Conclusion

One of the things the researchers discovered was that for the same data sources the coverage varies in ways that are not always easily understood.

First, the data providers fetch altmetric data from different social media platforms and with varying detailedness (e.g., Plum Analytics retrieves Facebook likes, comments, and shares, whereas ImpactStory only collects number of shares). Out of the studied data providers, Plum Analytics registers the most metrics for the most platforms. (Jobmann et al., 2014)

Even when all services appear to be using the same metrics from the same supplier, they can still vary.

Second, the data providers differ in the number of DOIs they find on the social media platforms. Especially in the case of Mendeley, Altmetric Explorer performs worst in retrieving readership information for DOIs…Third, the altmetric impact values provided by Plum Analytics, ImpactStory, Altmetric Explorer, and Webometric Analyst for the publications in the data set also considerably deviate. Oftentimes it is the Altmetric Explorer search which results in lower impact values. (Jobmann et al., 2014)

Third, the altmetric impact values provided by Plum Analytics, ImpactStory, Altmetric Explorer, and Webometric Analyst for the publications in the data set also considerably deviate. Oftentimes it is the Altmetric Explorer search which results in lower impact values.

We encourage you to read this article to see even more results and analysis.

References

Jobmann, A., Hoffmann, C.P., Künne, S., Peters, I., Schmitz, J. & Wollnik-Korn, G. (2014). Altmetrics for large, multidisciplinary research groups: Comparison of current tools. Bibliometrie - Praxis und Forschung, 3, http://www.bibliometrie-pf.de/article/viewFile/205/258 [Accessed on February 12, 2015]

[plumanalytics]

New Capture Metrics Launched

PlumX tracks 5 categories of metrics:  usage, mentions, social media, citations, and captures.  Metrics that we categorize as captures are when an individual is interacting with research online and “captures” it for later.  Examples of capture metrics that are already a part of PlumX are:

Bookmarking the page on Delicious

Adding it to Elsevier’s Mendeley as a reader

Favoriting a slide presentation on Slideshare

Favoriting a video on YouTube

Subscribing to a channel on Vimeo or YouTube

Followers, forks, or watchers of source code on GitHub

All of these actions indicate that a user might want to later come back to this research.  For example, a researcher finds an article and adds it to their Mendeley reader or bookmarks it in a bookmark manager such as Delicious so they can go back and read it later.

In the scholarly space, studies have shown that capture metrics are a good leading indicator of future citations.  For example, In the research article When are readership counts as useful as citation counts? Scopus versus Mendeley for LIS journals  (Open access version here), the researchers Maflahi and Thelwall introduce their research study as:

This article investigates the influence of time on the number of Mendeley readers of an article through a theoretical discussion and an investigation into the relationship between counts of readers of, and citations to, four general Library and Information Science (LIS) journals. For this discipline, it takes about seven years for articles to attract as many Scopus citations as Mendeley readers and after this the Spearman correlation between readers and citers is stable at about 0.6 for all years. This suggests that Mendeley readership counts may be useful impact indicators for both newer and older articles.

This week, we added new capture metrics to PlumX that continue to expand this important category to include data from EBSCO’s EDS discovery, EBSCOhost, and eBook platforms.  These new captures are calculated per artifact, and measure when a user emails, exports, prints, or saves an artifact’s citations.  More specially these actions are defined below:

Emails: The number of times an artifact’s citation, abstract and full text (if available) have been emailed

Exports: The number of times an artifact’s citation has been exported direct to bibliographic management tools or as file downloads

Print-outs: The number of times an artifact’s citation/abstract and HTML full text (if available) have been printed

Saves: The number of times an artifact’s citation/abstract and HTML full text (if available) have been saved

Some of these metrics comprise a part of the “hidden web.”  Being able to calculate the number of times users on a platform email or print out a citation, is a glimpse into what work is being saved for later more than others.

The metrics we can collect are of course only a proxy for ALL places and times that research has been saved for later.  We’ve expanded these new capture types using data from EBSCO, and will expand our sources over time as others become available to us.

Given the scale of this data we have just exposed, some PlumX customers will see dramatic increases in capture counts in their profiles. This is one reason why it's so challenging to distill alternative research metrics down to a single number: what happens when new metrics sources start to get collected?  Our approach is now, as it always has been, to collect as comprehensive a collection of indicators as possible.  Using the data should always be done in context and through benchmarking against other items of interest.

[plumanalytics]

New Artifact Types Supported in PlumX

We recently updated the types of artifacts that we support to better reflect the diversity of research documents that our customers want to track. Giving customers flexibility in how they manage their research output collection is a hallmark of PlumX, and is critical to being able to do an accurate assessment. For example, newsletters could have far different metrics profiles than blog posts, so being able to classify them into their own artifact groups ensures a true apples-to-apples comparison between the artifacts within each one.

The new artifact types are:

Blog post

Bibliography

Abstract

Review

Reference

Web Resource

Newsletter

You can see a complete list of all PlumX metrics and artifact types on the PlumX metrics page. To request more information about Plum Analytics go here.