Prioritising Competing M&E Needs and Demands in an Adaptive Programme: 7 Takeaways

In this blog post, Gloria Sikustahili, Julie Adkins, Japhet Makongo & Simon Milligan discuss seven key lessons about effective monitoring and evaluation from their experiences of implementing a 'iterative programming and adaptive management' approach in the Institutions for Inclusive Development programme in Tanzania.

We’ve all been there. We’ve drowned in the weight of programme documentation; the need to capture everything, to report everything, to be seen to be held accountable for all our actions or inactions. Yet on other occasions we’ve all sighed with exasperation that the programme we’re tasked with supporting has very little to help us understand what’s happened along the way, and why decisions were made.

So how do we strike the right balance? Whose needs are we meeting? And how do we better negotiate and prioritise the needs of different documentation users?

The DFID and IrishAid-funded Institutions for Inclusive Development programme in Tanzania – a “test tube baby” on iterative programming and adaptive management – explores new ways of tackling wicked problems in a variety of areas, such as solid waste management, inclusive education, and menstrual health. As a programme designed to be agile and opportunistic, it needs to have fast feedback loops so that staff and partners can make informed decisions about if and how to adjust strategy and tactics during implementation. It must also be accountable for its performance and generate lessons about how change occurs.

So, as the programme enters its final months of implementation, what lessons have we learned about M&E in an adaptive programme in terms of what, when and how to document and for whom? Here are our seven takeaways:

1. All parties benefit when genuine attempt is made to understand not only the programme theory of change, but also the basis for certain actions during the adaptive process. It pays to build rapport with donors and reviewers; to understand their interests, needs, concerns and motivations. While this might not necessarily lead to fewer requests, it can lead to more streamlined responses; responses that ‘tick the box’ first time around. Annual reviewers and mid-term evaluators also need to understand the interests of implementers and the programme managers rather than focusing only on donors’ interests. This means reviewers should take time to understand the programme’s journey to date and future trajectory, while balancing the interests of both parties is a necessity. All too often we, as human beings, find ourselves believing certain things of our partners, be they donors, reviewers, implementers, partner agencies or others. We can project our own insecurities, values, beliefs, assumptions and prejudices onto others, and can react negatively when faced with “another silly demand for yet more documentation”.

2. Utilisation is key and this needs to shape decision-making about what and when to document, and for whom. Documentation efforts should be clear-headed and purposeful. In much the same way as Michael Patton champions Utilization-Focused Evaluation, programmes should adopt a similar principle, i.e. that the need to document should be judged on its likely usefulness to its intended users, including donors, reviewers, managers, implementers and partners. Without agreement and clarity about purpose, intended user and relative costs, programmes end up documenting for documentation’s sake. But with actions come consequences. Excessive time spent ‘feeding the beast’ is time spent away from working towards an intended outcome.

3. Documentation needs and demands need to be negotiated and balanced: there is an ever-present risk of prioritising the documentation needs of funders and reviewers at the expense of managers and implementers, and doing so can be counterproductive. Don’t get us wrong: we aren’t saying there is a binary choice between the needs of funders and reviewers on the one hand, and implementers on the other. Yet, the reality is that those working hands on, at the front line, make decisions in real time and do so against a backdrop of uncertainty and incomplete or implicit understanding. By contrast, managers and implementers are often involved in slower, more structured processes. This meeting of two related, yet different realities creates an environment in which different needs, expectations and demands must be successfully navigated, even negotiated. Senior managers, such as team leaders, often find themselves acting as a buffer or intermediary with the donor. Inevitably, unclear, excessive or competing expectations about what must be documented creates uncertainty, bias and inefficiencies. All information users – donors, reviewers, implementers and partners – should think through what must be documented and reported, from what might be documented and reported and from what they would like to see rather than love to see documented and reported in an ideal world (thanks to outcome mapping for this prompt). As a general rule of thumb, key information should be synthesised and summarised as headlines. Brevity in documentation forces clarity of thought and aids the production of ‘formal’ reporting (annual & semi-annual reports, case studies, and even blogs!) when required.

4. To aid programme performance, donors must make choices and recognise the consequences of their signals. We acknowledge that all parties (funders, managers and implementers) require a certain level of documentation for accountability and sense-making purposes. OK, let’s get the “Yeah, obviously” observation out of the way first. Programmes have many constituencies or stakeholders, not least the donor agencies, and each party have their own needs and interests in documentation. Of course, where programmes are under-performing, close scrutiny is expected and necessary. However, staff within donor agencies hold positions of power. General queries or requests for information can be construed as demands – demands that often require the time-consuming compilation of documentation without a clear rationale and can result in airbrushed content which overlooks the messy realities. Unfiltered lists of questions, requests and comments from a variety of donor staff to a report, for example, can tie up implementers as they seek to make sense of, justify and explain, resulting in a ping pong to and fro which might have better resolved out over a cup of coffee. A confident programme with confident, capable and experienced staff can push back but this takes time and trust, and neither come over night. 

5. Reflection and documentation are two related, important, yet different things. I4ID, like many agile programmes, is built on experimentation. Staff value reflection but the process must be shaped by a desire to improve performance. To borrow from Graham Teskey’s recent blog post, purposeful reflection allows implementers to reach a decision about specific workstreams (i.e. this is what we are going to do from here); which  means that implementers will necessarily prioritise capture of what they need to know to move forward, building on tacit knowledge and shared understanding. And, that reflection-and-capture need not translate to extensive documentation. Yet, challenges arise when donors and reviewers seek to make sense of often messy realities, a number of months after the event. This can lead to a situation in which events and decisions are fully documented not because it is valuable to implementation but to cover off possible future need and demands. The answer? Understanding, keeping line of sight on what matters, and accepting ‘good enough’. The most successful staff capture-and-reflection platforms are those founded on real-time discussion and action. Early efforts in I4ID to document key events on a weekly basis using a Word-based template and again on a monthly basis using an Excel-based dashboard stagnated within 12 months. By contrast, those platforms that thrived at I4ID - the Monday morning meetings, staff WhatsApp groups, and the Quarterly Strategic Reviews by the programme team, donors and some invitees – were founded on real-time exchange of reflections and ideas. This suggests that staff respond more favourably to live, interactive platforms, not an impersonal capture and storage. These platforms should be well organised and managed to avoid biases and defensiveness.

6. Synthesis occurs most readily where discussions are well structured. Discussion is great, yet it must lead to something. For that to occur, use should be made of three key questions in key events (e.g. weekly meetings, strategic reviews) and associated minutes: What? So what? Now what? For example, what has happened in the operating environment over the last month, what programme effects have we seen, and what lessons have we learned about how change occurs? So, what does that mean for us and specifically, if/how our programme can best support the reform agenda? Now what should be done and by whom, not least before we next meet? Objective discussions require good facilitation to avoid bias and defensive reactions. 

7. Donors should consolidate and localise oversight functions wherever possible. The decision to have multiple levels of oversight – the donor agencies themselves, the annual reviewers appointed by the donors to verify the claims made by a programme, and the external evaluators appointed by the donors to capture lessons identified by the programme – creates a living organism that has many needs and expectations. Sometimes these are aligned, other times not. So, it is right to ask, at what point can oversight functions (e.g. an external Results and Challenge team that produces annual reviews, and a separate external Mid-Term and End-of-Programme-Evaluation team) be consolidated or streamlined? Donors should ensure that local institutions are included in the evaluation and reviewer teams to get local perspective on the process and results, build capacity to local stakeholders and promote adaptive programming locally.