Extracting Evidence

The process of extracting evidence from the literature is one that can probably never follow any set rules. Different researchers will find different ways of maximising their efficiency. Here we describe one approach that has been successfully used in our research projects.

1. Select papers for extraction

An initial assessment of whether the paper contains evidence is usually possible from reading the abstract. In our experience 70% of hits from literature database searches are irrelevant to the questions we are asking.

2. Read abstracts to filter 'relevant' literature

Read carefully to make sure the study is related to the question you are asking, and seems to contain evidence (be prepared to be wrong occasionally).

3. Print and highlight, or annotate on-screen

For papers retained after reading the abstract, I prefer to print them out (being as efficient as possible with paper). I need to be able to highlight and annotate the paper as I read.

4. A focused approach to reading

Despite the effort that goes into writing it, the Introduction is not going to tell you much about what the researchers found. It can help you to understand what question they were asking. Concentrate on the Methods and Results, and to a lesser extent the Discussion. This is quite a different way of reading a paper to what most of us are used to. We specifically address the following sections.

Introduction

What question/s were the researchers asking?

Methods

Information on experimental design, number of replicates and statistical analysis used. These allow us to assess study quality.

Results

Information on effects seen, including the effect size, dose-response, p-values from statistical tests, etc. This is the bulk of the evidence.

Discussion

Helpful for assessing coherence of the results. Do they make sense in terms of existing literature / theory?

5. Annotate

I use different coloured highlighters to identify different pieces of evidence in a study – there is often more than one. One colour is used for general information (e.g. study design, replication, climate, location, spatial scale), and then individual colours used for each piece of evidence I find in the paper (usually concentrated in results and discussion, but occasionally in methods as well). Highlighting different pieces of evidence in different colours is invaluable when it comes to entering that evidence into the database.

6. Enter extracted evidence into database

I do all the pieces of evidence at the same time. I find it easier to highlight all the evidence in the paper and enter it later, rather than to attempt to enter it while reading. If entered while reading, there is a greater chance of having to back track to change misinterpretations, and there is always the possibility of getting timed out of the database and losing all changes made to the page to that point.

When a paper contains multiple pieces of evidence, they are usually related. Using the 'duplicate' button on the View Evidence page can greatly reduce the amount of work needed to input several pieces of evidence, and also make the language used more consistent. Be careful to go through all fields in the new record to assess which ones do and do not need to be changed.

And lastly, we recommend FILLING OUT EVERYTHING in the evidence record, unless there really isn’t any information (e.g. it's a factorial experiment so 'dose-response' is not relevant). It is easy to leave out information, particularly the Question being asked, the various Detail fields, and page details for the evidence. But doing this makes it less likely that a future user will be able to make sense of your evidence record, and will have to repeat the extraction.