Inward-looking evaluation: Gauteng Stories and Metaphor emergent process

Get Complete Project Material File(s) Now! »

Background

Africa in the twenty first century remains the forgotten continent (Chimere-Dan, 1999). This is the continent where poverty is most widespread, nations are least economically productive, food security is most compromised and HIV/AIDS is most destructive (Moyo, 2009). Africa seems stubbornly depressed. Development thinkers grapple with the forces behind the continent’s chronic lassitude, in an era of explosive global progress.

The funders’ case

Large development funding agencies operate at scales of millions in currency, thousands of people, hundreds of projects in dozens of countries (Organisation for Economic Cooperation and Development (OECD), 2010). Funding agencies face risk of being charged with massive scale corruption, or with reallocation of their budgets, and therefore their jobs, unless they can demonstrate their own worth. If their finances are squandered or stolen, they are held accountable to their own power structures. Some monitoring of the destination of their spending, and the outcomes of their interventions is necessary and critical to their own learning and management (O’ Dwyer & Unerman, 2008; OECD, 2008). Conventional, logic-based evaluation is designed primarily to meet these needs (Ebrahim, 2005; Kilby, 2006; Gray, et al., 2006). Any alternative system for evaluation must renegotiate and meet funders’ needs for accountability.

Research objectives

To identify viable evaluation process elements and principles for assessing the outcomes of CBO efforts in building a community-based response to the impact of HIV, which:
i) Support CBO self-determination and development as organisations;
ii) Encourage responsive project planning and organisational learning;
iii) Respond to the accountability needs of funding agencies.

Ontology

Ontology: “Philosophy: The branch of metaphysics that deals with the nature of being. Logic: The set of entities presupposed by a theory.”1 Ontology refers to a view of reality. It asks us to consider what assumptions, or pre-suppositions, underpin our theory of reality. It describes the world view from which a researcher takes her perspective. Quinn-Patton (2002, p.134) considers ontology to refer to a belief in a single, verifiable truth, as opposed to socially constructed multiple realities2. This fits well with the definition. If we assume that there is a truth, and that it can be described and determined, our theory reflects this. If, however, we assume that truth is relative, and can only be described as a vantage point, then theory must be quite different.

ACKNOWLEDGEMENTS
ABSTRACT
CHAPTER 1 INTRODUCTION
1.1 Rationale
1.1.1 Background
1.1.2 Development as power
1.1.3 The culture of the CBO-service contractor
1.1.4 The funders’ case
1.2 Problem statement
1.3 Research objectives
1.4 Ontology
1.5 Epistemology
1.6 Delineation and limitations
1.7 Definitions of key terms and concepts
1.7.1 Community-based organisations
1.7.2 Evaluation
1.7.3 Participatory
1.7.4 Development
1.7.5 Assessment, as compared with evaluation
1.8 Underlying assumptions
1.9 Contribution of the study
1.10 Brief chapter overview
1.10.1 Chapter 1 Introduction
1.10.2 Chapter 2 Literature review: Situation context
1.10.3 Chapter 3 Methods: Research approach in brief
1.10.4 Chapter 4 Results
1.10.5 Chapter 5 Discussion
1.10.6 Chapter 6 Conclusion
1.11 Ethics
1.12 Additional institutional requirements
CHAPTER 2 LITERATURE REVIEW AND THEORETICAL FRAMEWORK
2.1 Introduction
2.2 HIV, development, civil society and accountability
2.2.1 South African scenarios for the future: The position of civil society in the
institutional fabric
2.2.2 History: Civil Society in post-apartheid South Africa
2.2.3 The “third sector”: Defining civil society
2.2.4 Tensions and interests: The roles of civil society
2.2.4.1 The third sector
2.2.4.2 The public-private-civil services niche
2.2.4.3 Agents of democracy?
2.2.4.4 So aren’t NGOs and CBOs actually private sector?
2.2.4.5 Service providers to the poor
2.2.4.6 The role of CBOs
2.2.4.7 Sustained developmental impact?
2.2.4.8 CBOs in the HIV and AIDS response
2.2.5 Size of the NGO / CBO sector
2.2.5.1 In money
2.2.5.2 In numbers
2.2.5.3 In people
2.2.6 Organisational behavior and organisational relationships:
2.2.6.1 Power
2.2.6.2 Donor relationships
2.2.6.3 Downward accountability: Constituents
2.2.6.4 Inward accountability: Staff and volunteers
2.2.6.5 Upward accountability: Funding sources
2.2.6.6 Holding the powerful to account
2.2.7 Monitoring and evaluation (M&E)
2.2.7.1 Conventional, ‘logical’ evaluation methods for M&E
2.2.7.2 The impact of funding and evaluation on organisations
2.2.8 Capacity building
2.3 Towards alternative principles and practice in evaluation for CBOs
2.3.1 Organisational Learning: Moulding organisational behaviour
2.3.2 Principles of developmental M&E
2.3.3 Complex dynamic emergent systems
2.3.4 Emergence
2.4 Conclusions of the literature review
CHAPTER 3 RESEARCH DESIGN
3.1 Introduction.
3.2 Overarching theoretical framework: evaluation and meta-evaluation.
3.2.1 Grounded theory
3.2.1.1 Grounded theory in brief
3.2.1.2 The grounded theory debate
3.2.1.3 Grounded theory
3.2.1.4 Grounded theory method
3.2.1.5 Constructivist grounded theory
3.2.2 Critical change theory and process use
3.3 Research structure: Three worlds and two legs
3.4 Research approach
3.4.1 Meta-methodology : Key concepts in reality-based methods development
3.4.1.1 Exploratory research
3.4.1.2 Action Research for methods development
3.4.2 Evaluation: Key concepts in alternative, participatory, developmental processes
3.4.2.1 Action Learning or Participatory Action Research
3.4.2.2 Narrative in evaluation
3.4.2.3 Metaphor
3.4.2.4 Stories of Most Significant Change
3.4.2.5 Qualitative evaluation
3.5 Research setting
3.5.1 Informal settlements
3.5.2 Low-income suburbs
3.5.3 Rural village
3.6 Sampling
3.6.1 Sampling strategy
3.6.2 Sample population
3.6.3 Sample size
3.6.3.1 Gauteng Stories and Metaphor process
3.6.3.2 North West MSC
3.6.4 Case Studies
3.7 Research process
3.7.1 Gauteng: Stories and Metaphor
3.7.2 North West: Stories of Most Significant Change
3.8 Data recording
3.9 Data analysis
3.9.1 Analysis in action research and constructivist grounded theory
3.9.2 Participant analysis
3.9.3 Mentorship and peer review as collective analysis
3.9.4 Case Study analysis
3.9.5 Criteria for analysis
3.9.6 Deductive and inductive analysis
3.9.7 Coding, themes and patterns
3.10 Dissemination and Proceduralisation
3.11 Ensuring quality
3.11.1 Rigour and trustworthiness
3.11.2 Boundaries, challenges and possible sources of error
3.11.3 Ethics
3.12 Conclusion to the methods chapter
CHAPTER 4 RESULTS
4.1 Introduction
4.2 Chapter structure
4.2.1 Within the cases
4.2.1.1 Non-empirical study: Action research cycle from data to theory
4.2.1.2 Empirical study: CBO evaluation from stories to learning
4.2.1.3 Content analysis using Theory of Change
4.2.2 Between the cases
4.2.3 Closing the phases
4.3 Inward-looking evaluation: Gauteng Stories and Metaphor emergent process
4.3.1 Case Study 1: TT
4.3.1.1 Diagram of process
4.3.1.2 Description, reflection, learning and conclusions
4.3.1.3 Exhibits from TT
4.3.1.4 Action and questions leading into Case Study
4.3.2 Case Study 2: JJ & JD
4.3.2.1 Diagram of process
4.3.2.2 Description, reflection, learning and conclusions
4.3.2.3 Exhibits from JD
4.3.2.4 Reflections with mentor
4.3.2.5 Action and questions into Case Study
4.3.3 Case Study 3: QN
4.3.3.1 Diagram of process
4.3.3.2 Description, reflection, learning and conclusions
4.3.3.3 Exhibits from QN
4.3.3.4 Reflections with mentor
4.3.3.5 Action and questions leading into Case Study
4.3.4 Case Study 4: DG
4.3.4.1 Diagram of process
4.3.4.2 Description, reflection, learning and conclusions
4.3.4.3 Exhibits from DG
4.3.4.4 Reflections with mentor
4.3.4.5 Action and questions leading into Case Study
4.3.5 Case Study 5: BN
4.3.5.1 Diagram of process
4.3.5.2 Description, reflection, learning and conclusions
4.3.5.3 Exhibits from BN.
4.3.5.4 Action and questions leading into Case Study
4.3.6 Case Study 6: CL
4.3.6.1 Diagram of process
4.3.6.2 Description, reflection, learning and conclusions
4.3.6.3 Exhibits from CL
4.3.6.4 Action and questions leading into Case Study
4.3.7 Concluding the Gauteng Stories and Metaphor process
4.4 Outward-looking evaluation: Applying Most Significant Change methodology in community development setting 2
4.4.1 Research setting and context
4.4.2 Diagram of process
4.4.2.1 STEP 1 Preparation and sensitisation
4.4.2.2 STEP 1b Recruiting the team
4.4.2.3 STEP 1c Training the researchers
4.4.2.4 STEP 2 Defining the domains of change
4.4.2.5 STEP 3 Defining the reporting period
4.4.2.6 STEP 4 Collecting Most Significant Change stories
4.4.2.7 STEP 5 Analysis: Selecting the story of most significant change
4.4.2.8 STEP 6 Feeding back the results
4.4.2.9 STEP 7 Verification of stories
4.4.2.10 STEP 8 Quantification
4.4.2.11 STEP 10 Revising the system: Recommendations.
4.4.2.12 Exhibits for the Mabeskraal Most Significant Change process
4.4.3 Concluding the MSC Phase (MSC STEP 9)
4.4.4 Gaps: What the method does not achieve
4.5 Conclusion to the results chapter
CHAPTER 5 DISCUSSION
5.1 The practice: Towards alternative methodologies for evaluation of CBOs
5.1.1 Inward and outward looking evaluation
5.1.2 Stories
5.1.2.1 Impact is meaning
5.1.2.2 Story collection
5.1.2.3 Collectively analysing narrative
5.1.2.4 Stories as grounded evaluation
5.1.3 Metaphor in evaluative analysis
5.1.4 Facilitating participatory evaluation
5.1.4.1 Who holds the pen?
5.1.4.2 Community researchers: participatory learning in action
5.1.4.3 The facilitator
5.1.4.4 Bias and subjectivity
5.1.5 Diversity as an evaluation concern
5.1.5.1 Interpreting unfamiliar behaviour
5.1.5.2 Familiar behaviour
5.1.6 Ethics
5.2 The principles: Making evaluation developmental
5.2.1 Power in evaluation
5.2.1.1 Literacy as a vessel for power
5.2.1.2 Language games in the evaluation profession
5.2.2 Appreciative inquiry
5.2.3 ‘Holding’ the organisation: Evaluator responsibility
5.2.4 Evaluating for inward accountability
5.2.5 Evaluating in complex systems: Realist approaches
5.2.5.1 Learning the language: standardising and quantifying criteria
5.2.5.2 Alternative assumptions: Theory of Change
5.2.5.3 Grounding evaluation criteria
5.2.5.4 Quantifying outcomes: measuring grounded indicators
5.2.5.5 Ownership: whose evaluation, whose criteria?
5.2.5.6 Funders’ criteria checklists
5.2.6 Funding relations
5.2.6.1 Into the funding game
5.2.6.2 Community service entrepreneurs
5.2.6.3 What if there was no CBO donor funding at all?
5.2.6.4 Supply and demand: The funder dilemma
5.2.6.5 Development evaluation: an oxymoron
5.2.6.6 Funding review and evaluation: not the same thing
5.2.7 What about capacity building?
5.2.8 Shadow: the poltergeist of organisation dynamics
5.3 Development, power and CBO character in metaphor
5.3.1 The Knights
5.3.2 The Saints
5.3.3 The Snakes
5.3.4 The Sheep
5.4 Conclusion to the discussion
CHAPTER 6 CONCLUSION
6.1 Introduction
6.2 Summary of findings and associated recommendations
6.2.1 Theoretical contribution
6.2.1.1 Complex dynamic theory
6.2.1.2 Emergence
6.2.1.3 Grounded theory
6.2.2 Meta-evaluation: Methodological contribution
6.2.2.1 Action research
6.2.2.2 Iterative, cumulative coding
6.2.3 Developmental evaluation for CBOs: Practical contribution
6.2.3.1 Organisation-centred, visual and verbal communication and evaluation formats
6.2.3.2 Appropriate M&E technology
6.2.3.3 Intangible, complex, systemic thinking
6.2.3.4 Alternatives to predictive planning and evaluation
6.2.3.5 Responsive, pragmatic, organisation relevant evaluation
6.2.3.6 Purpose prevails over method
6.2.3.7 Be appreciative
6.2.3.8 Facilitation, more than evaluation
6.2.3.9 Participation
6.2.3.10 Evaluation and organisation development
6.2.3.11 Internal accountability
6.2.3.12 Capacity building
6.2.3.13 Ethics
6.3 Conundrums and unanswered questions
6.3.1 Subjectivity
6.3.2 Exploitation or volunteerism
6.3.3 Funding relationships
6.3.3.1 More, smaller, easier funding relationships
6.3.3.2 Funding review and learning evaluation
6.3.3.3 A culture of engagement
6.3.3.4 The power of money
6.3.4 Power as a development resource
6.3.5 Development and colonialism: dare we ask?
6.4 Returning to the research question: achievements and limitations of the study
6.4.1 Problem statement and research objectives
6.4.2 Thesis outline
6.4.3 Limitations and unmet potential
6.5 Suggestions for further research
6.5.1 Further theoretical research
6.5.2 Methodological research
6.5.3 Suggestions for practical research
6.6 Potential significance
6.7 In closure
REFERENCES