Table of Contents page

                            The Process of EBL: Critical Appraisal of the Evidence

Tutorial goals applicable to this lesson: 

• Evaluate the applicability of evidence-based practice for your setting.

Objectives met by this lesson:  

• Critically appraise resource documents and data for their validity and applicability to your setting.
• Discuss barriers to implementation of evidence-based practice for your setting.

Total time required: 30 minutes for prerequisite reading; 20 minutes for reading through the module, and 90 minutes for the hands-on component.

Reading assignments prior to module:  Booth & Brice Ch. 9: Appraising the Evidence

Greenhalgh T, Taylor R. How to read a paper. BMJ series. Assessing the methodological quality of published papers  http://bmj.bmjjournals.com/cgi/content/full/315/7103/305 

NOTE: This series, available entirely online (see the EBM section of the tutorial’s Bibliography page for the series URL) may help you to consider and evaluate how the LIS question sets were created.  You will be asked later to reflect on how well the questions provided here work. In addition to evaluating the usefulness of the question sets, you are requested to reflect on how such questions might help to evaluate the sources  we use frequently, such as surveys on listserves and ‘how we done it good’ articles that are not scholarly.

 

---

 

Take a minute to write about a recent experience you have had (or are aware of) where you or others used published LIS literature to support decision-making.  How were you made aware of the information? Was the information directly applicable to your own situation?  (short essay question box here)

This module is all about finding and then evaluating what you find, in order to avoid ‘reinventing the wheel’ with library projects.  Why not use what you can of others’ experiences?  We do that all the time anyhow, sharing experiences and lessons learned at conferences, on listserves, and (less often) more formally, through published literature. 

By the time you finish the module, you’ll be able to critically appraise published LIS research for its validity and applicability.  Following your hands-on practice of these skills, you will reflect on your experiences and assess how well the proposed evaluative questions function with LIS literature. You’ll also have an opportunity to to assess their use with information available through other, frequently-used channels, such as LIS listserves.

Remember the 5-step process of EBL as it has been described:

1. Formulate a clearly-stated, answerable question 
2. Track down the best evidence
3. Critically appraise the evidence
4. Apply to clinical practice
5. Evaluate performance

Recall that in the hands-on component of the previous module, you constructed a search grid, proposed search terms and phrases, and ran a quick search in CSA LISA to see what your search strategy would retrieve. You also evaluated the performance of the resource (LISA), reflecting on the experience, how the ‘well built question’ functioned to assist in the process, and how you might improve on your retrieval.

In this module, we’ll build directly on the previous hands-on experience, adding in the next step: critical evaluation of what you find.

Unlike the field of medicine, the literature with potential application to LIS practices is not overwhelmingly vast. As discussed in the previous module, it is more likely to be scattered among the resources of disparate disciplines than to be concentrated in a few, highly-regarded repositories. No matter where it's located, its application to practice must be preceded by a critical appraisal process – or we risk accepting biased information, potentially wasting time and resources.

The usefulness of any evidence is directly related to its validity, reliability, and applicability. Validity concerns the amount of freedom from bias. Reliability is about the 'trustworthiness' of a study's results - in other words, whether the study would obtain the same (or similar) results if the same conditions were replicated (reproducibility). Questions of applicability ask whether the intervention being tested has made a measurable difference for the chosen population and in the particular setting. Often, this difference is expressed statistically.

An important thing to note, especially if little evidence can be found, is that even a poorly designed or less than rigorously conducted research study may still provide useful information. Being aware of a study's weaknesses (and most or all studies have them!) is just part of the process involved in assessment. Of course, a major 'weakness' if it can be called that, is that research findings from other locations can never perfectly match your own setting, and so must be appraised with the important question in mind about how the findings from another setting may be applied to your own.

Checklists for critical appraisal have been created in every field doing evidence-based practice, and are a key component of the model. The existence of such carefully-created criteria for evaluation assists the reader in a number of ways, including helping to remember important criteria, reducing uncertainty, and providing ways to measure or evaluate a body of data using uniform standards. In fact, previously established methods for appraisal have led to the creation of guidelines and standards, as is the case in EBM, where uniform critical appraisals of sets of research findings (for example, systematic reviews or even meta-analyses) have helped to establish "gold standards" for diagnosis and treatment. Perhaps you can understand, reading this, why it would be difficult to overstate the importance of critical appraisal!

This tutorial does not address methods of evaluating statistical data, though future changes may include it.

The checklist included here is from Evidence-based Practice for Information Professionals: a handbook, chapter 9 (Booth & Brice, 2004). It's most suitable to the evaluation of a user study.

A second checklist is intended for use in evaluating a needs analysis (from Booth & Brice, 2004).

A third set, adapted from Trisha Greenhalgh’s critical evaluation article paper on assessing clinical systematic reviews, is still under development for LIS studies:

This hands-on experience is intended to build directly on the hands-on exercise you did for the last module, where you built a search grid, proposed terms and phrases, and conducted several short searches in order to evaluate that part of the process.  Now, you are asked to select two of the articles you found during your search, and – using one or more of the evaluative question sets you just saw in the main part of this module – critically evaluate them.  Finally, you will once again reflect on your experiences, sharing those thoughts with others through the class bulletin board.

Instructions for this assignment

Your responses should be brief but comprehensive, covering each of the questions asked by the question set you decide to use.  Use a word processing program to complete this assignment.  When you’re done, post your document to the bulletin board (link).

What should be included in your response:

• Looking back at module 3, determine what type of research study would be most appropriate to the well-built question:  Explain your choice.
• Provide the citation and abstract for your chosen article.
• List the evaluative question set you consider appropriate.
• Using a critical appraisal checklist, evaluate the research study.  It will help with this process if you copy and paste your chosen checklist to a Word document.
• What is your conclusion about the validity, reliability, and applicability of this research article in answering your question? Please support your answer with direct references tothe text of the article when possible and appropriate.
• How well do you think the proposed evaluative questions function with LIS literature?
• Lastly, use what you have practiced in evaluating published LIS literature by considering one of these documents, each the result of surveys conducted on the MedLib-L listserv:

1. http://listserv.buffalo.edu/cgi-bin/wa?A2=ind0103B&L=MEDLIB-L&P=R5865&I=-3

2. http://listserv.buffalo.edu/cgi-bin/wa?A2=ind0506A&L=MEDLIB-L&P=R3408&I=-3

3. http://listserv.buffalo.edu/cgi-bin/wa?A2=ind0201B&L=MEDLIB-L&P=R607&I=-3

·         Summarize your thoughts about the process of evaluation for the information in these survey response documents. Be sure to include the URL of your chosen document with your summary.  Answer the following questions:

o Could you answer all the questions from any of the evaluative question lists?

o What is your evaluation of the evidence provided?

o How would you use the evidence provided?

Your work will be assessed based on its content, organization, and clarity.   I have provided the form I’ll be using to perform that assessment, below.   

Possible points: 100  

A      86-100 points

B       70-85 points

C      58-69 points

D      50-57 points

Note: This tutorial is intended to encourage thought and discussion, so there is no ‘grade’ assigned except to allow you to self-assess your learning.  If you are not satisfied with your performance, you may wish to review the concepts we’ve covered in the pertinent areas. 

Assessment Criteria