Of Verbs and Nouns, Harvesting and Fireworks: the Danger of a Single Study

Want create site? Find Free WordPress Themes and plugins.

By Laurie Less

Renowned African literary laureate, Chimamanda Adichie Ngozi’s words ring so true to me, “The danger of a single story… Our lives, our cultures are composed of many overlapping stories, says Chimamanda who warns that if we hear only a single story, we risk a critical misunderstanding”, colloquially we may refer to this as having a limited viewpoint and this may effectively skew our thinking. For evaluators, the single story pose a number of risks in that although the evidence of a single evaluation may not be wrong…. it is however incomplete – ‘the whole is greater than the sum of its parts’ – and that is the fundamental difference Evidence Synthesis seek to remedy, in that it produces new knowledge and patterns; addresses complexity; and leads to policy making in support of positive social outcomes.

So as I sat enthralled, listening to the speakers Dr E Obuku and Laurenz Langer of the African Centre for Evidence (ACE), engage their audience on the topic, “Producing Evidence Synthesis in SA”, the single message that resonated throughout the lecture by Laurenz was, “…the danger of a single study”.

We all synthesise evidence, everyday….

In our personal lives we make use of a range of pieces of opinions (evidence) to make decisions on a daily basis. This is no different when we try to feed evidence into the policy or practice process. Main principle is the combined results of many pieces of evidence (studies) are more reliable than that of a single piece. We call the process of combining, i.e. adding up and arranging, different pieces of evidence ‘synthesis’.

In our personal lives we make use of a range of pieces of opinions (evidence) to make decisions on a daily basis. This is no different when we try to feed evidence into the policy or practice process. Main principle is the combined results of many pieces of evidence (studies) are more reliable than that of a single piece. We call the process of combining, i.e. adding up and arranging, different pieces of evidence ‘synthesis’.

We are often drawn into controversial epistemological debates, as there are lots of criticisms about Evidence Synthesis but the counterfactual: if we don’t synthesise evidence it’s like watching a fireworks display – everyone publishes individual studies but no one sees the bigger picture (analysis). “Each evaluator seeks attention – each evaluation look great in their own right, shiny, important, flashy – occasionally we have a sense of the overall picture, but we can never be sure because we can only sense it for a second and then the next primary study pops up demanding attention. So we go home feeling good and happy, but do we really understand the overal picture?” Langer used a beautiful harvesting metaphor throughout (see attached picture), saying that it was like growing crops without harvesting them (thankful that it was not yet another rugby or sporting metaphor). There are many primary studies as there are many crops in the world, primary studies are important, they are the foundation of our knowledge but we can get lost in them easily if we don’t harvest and structure the combined knowledge coming out of them. Harvesting and structuring, this is what research synthesis is about.

There are different ways of doing this:

  • Rapid/Responsive Evidence Assessments (REAs);
  • Literature reviews: but there is the danger that experts cherry pick the studies or knowledge they like most that confirms their view point example the Ugandan study on – homosexuality);
  • Systematic reviews: this is a more comprehensive and careful form to review all evidence in a structured and transparent way and not steamroll past details and context;
  • Evidence maps; and
  • Evidence gap maps.

However, one way is not the right way. All evidence synthesis follows a rigorous and transparent research process because evidence is socially constructed.

The evaluator decides what relevant evidence is; Finds evidence; Categorises the evidence; Appraises the evidence; Organises, unpacks and aggregates the evidence.

Laurenz used a metaphor of Synthesis Reviews as both VERB and NOUN. Do we advocate for the use of the method or the use of the output? He explained that as a verb – Synthesis Review is the methodology employed or the action. As a noun it is the actual product as a result of the process in other words the actual review published journal article, or policy paper. Whilst Langer stated that he advocated for the method not the product, he went on to state that the “product” if legitimate can lead to a change in policy thus influencing development. Conversely in a country like Uganda said Dr Obuku, where a survey (product) showing 96% of the population was against homosexuality[1], led to President Museveni signing a highly controversial anti-gay bill into law. So, contextual and political relevance is crucial when using evidence synthesis as a methodology and evaluators must be mindful of typical lacunas such as cherry picking to confirm a result as decisions could be made instrumentally and symbolically.

Currently in SA it is mostly health organisations which use Evaluation Synthesis. In my opinion this poses a challenge to the social sciences to advocate for its use. What kind of knowledge should be included in SA, as an example, should we include indigenous knowledge and who should coordinate the evidence of using evidence synthesis?

Examples of the use of evidence syntheses from South Africa:

  • Traditional methods used by PRICELESS-SA and ICT in education.
  • The more recent example using non-traditional methods, was the panel of SA experts to review the minimum wage. A number of specialists with a broad range of experience gathered evidence and knowledge that was known and synthesized this.
  • More recently I’ve discovered that the SA government through the DPME used evidence maps in a study, “The Synthesis Evaluation of Asset Creation through Government Subsidy Housing” which ultimately served in Cabinet. This evaluation was instrumental in the review and drafting of the new White Paper on Human Settlements.

Evidence Synthesis Review methods arise out of the health sector and the challenge is to deepen it beyond this sector. As the field progresses, this will result in greater uptake and interest. There is a need to strengthen studies, research, academics, research councils and civil society to produce synthesis reviews in SA. An opportunity exists for more academic institutions to include syntheses reviews as a methodology in their post graduate studies in the field of M&E.

A few questions to ponder:

  • How does evidence synthesis relate to Social Economic Impact Assessment Systems (SEIAS)?
  • What is the role of civil society organisation in evidence synthesis?
  • Are we advocating for the use of the method or the use of the output?
  • How can we increase academic interest to teach and or apply synthesis review?

Examples of seminal evidence syntheses from South Africa: Education Meta- Analysis: Investor and key stakeholder response to review of education evidence in SA Besharati N and Tsotsotso K (2017)[2]; A systematic review of the impact of Micro-finance; Task shifting from doctors to non-doctors for initiation and maintenance of antiretroviral therapy; E-learning of evidence-based health care (EBHC) to increase EBHC competencies in healthcare professionals: a systematic review by Tarryn Young, April 2017 (PRICELESS-SA) Wits University.

[1] Pew Research 2013

[2] CLEAR AA knowledge product.

 

 

 

Did you find apk for android? You can find new Free Android Games and apps.

Leave a Reply

Your email address will not be published. Required fields are marked *