Research Impact: Measuring Impact

Most research will be best represented using a combination of quantitative and qualitative approaches to measure impact, although this can differ vastly by field.

The following is a broad overview of medicine and health 

Medicine and Life Sciences

• Feature wider collaboration and a more rapid rate of publications - can expect higher citation rates than other disciplines such as mathematics.
• Publish frequently in journals but also produce conference papers, books and book chapters and reports, generating a vast quantity of literature that cites within itself
• Impact in these areas tends to be more quantitative and metric focused, often measured in relation to the number of citations the research generates

Citation Analysis:

• Citations Count
• Citation patterns
• Citer Analysis
• h-index
• Journal Metrics

Other forms of Impact Analysis:

• Altmetrics
• Influencing Practice, Policy and Programs

Citation Analysis is an indication of the quality of the article.

• Keep in mind to only compare citation counts of articles that are published in the same field.
• Though these metrics indicate quality in a general manner it should be treated carefully. For example an article could have great impact and a large citation count, not because it represents great quality but because the science is poor and the impact has been catastrophic.
• Citation counts and patterns can be found in most databases including PubMed and Google Scholar.

Citation Counts

Citation counts represent the number of times a scholarly document has been cited by other researchers. It is the basic tool of citation analysis. 

Citation Patterns

Citation patterns are dependent on citation counts. They can be useful in showing how your research performs over time.

• Citation items in each year shows peaks and troughs which can reflect the research cycle. This can help you show your productivity

• Citations in each year can show that your work is continuing to grow in influence e.g.

. 

Citer Analysis

Citer analysis provides qualitative measures of your research impact by providing information about the citations of your work, including:

• who is citing your research
• where they are
• what institution they're from
• in which publications they have published
• in which discipline they have published

Finding out who is citing you and where they come from can help you show:

• significance of impact
• research trends
• evidence that your work is influential across multiple disciplines

Scopus and Web of Science provide graphic and analytic tools to help gather this information. If you do not have access to these resources through an Academic institution, Google Scholar can provide similar information but requires more manual effort.

Altmetrics for tracking scholarly communication on the Web.

Altmetric scores take into account the number of times an article is read or mentioned and is weighted by the importance of the mentioner and by the source. For example, an article mentioned in a newspaper would be weighted more heavily than a mention in a tweet by an individual.

Many journals include some altmetrics on their websites that include views, downloads and social media mention

Mentions of articles are tracked in the following source types:

• mainstream media e.g. newspapers
• tweets
• blogs
• policy documents
• multimedia
• online reference managers
• social media
• manual additions

Altmetrics:This is a metric commonly used to track scholarly communication on the Web. This tool has been integrated into many databases such as Science Direct and also journal websites such as the British Medical Journal.

Research impact on policies, programs and practice can be illustrated by the number of mentions in Government and Non-Government Organization documents such as white papers, parliamentary papers, reports, program evaluations etc.

• Tracking and measuring research impact on policies, programs and practice can be difficult and will need a variety of web based tools, from Google alerts to database searches, tailored specifically to your needs. Please see your Librarian for help. 

Primary Health Care Research Information Services (PHCRIS) provides researchers with practical guides, tips, videos, fact sheets, on how to influence policy development.

• Maximizing the Influence of Your Research on Policy
• Introduction to Knowledge Exchange
• Introduction to Research Impact and Policy
• How to Communicate your Research to Decision Makers

H-Index, also called Hirsch index or Hirsch number, is an author level metric.The author is rewarded by having a range of papers that have been cited numerous times, rather than having one or two papers with extremely high citation rates.To accurately interpret the h-index, it's important to understand the way your field publishes and cites so that the h-index becomes meaningful. 

Definition: An author with an index of h has published h papers each of which has been cited in other papers at least h times.

Example: An Author will have an h-index of 12, if 12 of the author’s papers have been cited at least 12 times by other articles.

Image source: Wikimedia Commons http://en.wikipedia.org/wiki/File:H-index-en.svg

Moore, G., Redman, S., Haines, M., & Todd, A. (2011). What Works to Increase the Use of Research in Population Health Policy and Programmes: A Review. Evidence & Policy: A Journal Of Research, Debate And Practice, 7(3), 277-305.

Abstract:

Policy agencies are implementing strategies to increase the use of research in policy decisions. This paper examines the evidence about the effectiveness of these strategies. We conducted an extensive search focused on population health policy and programmes. We classified 106 papers meeting study criteria into research type (conceptual, descriptive and intervention). We examined the descriptive studies to identify commonly nominated potential intervention strategies. We examined the intervention studies to evaluate the impact of the tested strategies in increasing the use of research in policy decisions. There is little evidence about which strategies increase the use of evidence in population health policy and programmes.