Monitoring is not Evaluation
Monitoring asks what was done: how many people took part, how many trainings were delivered, how many enterprises were reached, how much of the budget was used, whether activities finished on time.
Evaluation asks a different question: what did those activities change? Did the participant’s knowledge increase? Did behaviour change? Did the probability of employment rise? Did income grow? Did access to services improve?
And impact evaluation goes one step further: did that change actually come from the programme?
From Activities to Impact
A good M&E system does not count outputs. It builds the logic of the programme as a whole.
In a youth employment programme: input is budget, experts and training material; activity is the vocational training; output is the number of young people who completed it; outcome is the share who found work; impact is the net increase in employment caused by the programme.
Without that distinction, successful implementation and successful impact get confused with one another.
- Inputs
- Activities
- Outputs
- Outcomes
- Impact
Theory of Change
We usually start an evaluation before any data is collected, by questioning the logic of the programme itself: what problem is it trying to solve, through which mechanism is change expected, what are the intermediate stages, and which assumptions have to hold.
From there a theory of change and a results chain are built, and indicators are attached to that structure. Monitoring without choosing the right indicator first only produces a large quantity of numbers.
Indicator Framework
Not every indicator is meaningful. A good indicator is measurable, interpretable, trackable over time, connected to the programme logic and usable for a decision.
Output, outcome, impact, process and quality indicators are designed together as needed. For each one, the definition, numerator and denominator, data source, measurement frequency, owner and target value are stated explicitly.
A baseline story
Baseline: Showing Success Is Easy
At the end of a programme, showing "success" is easy. The real matter is knowing the starting point.
Suppose 32% of young people were in employment before the programme started and 45% at endline. At first glance that is a 13-point increase. But we immediately ask what happened to young people outside the programme over the same period.
If the comparison group moved from 31% to 42%, general economic conditions may have played a significant role. Evaluation is the work of separating those effects.
Baseline–Midline–Endline
For some projects a start and a final measurement are not enough. Baseline shows the starting position, midline the direction of change while the programme is running, and endline the result at the close.
This structure matters particularly in long-running social policy, education, health and employment programmes: the intermediate measurement shows not only the outcome but whether the programme is moving in the right direction.
An evaluation story
"How Many People Got a Job?" Is Not Enough
One of the studies ID took part in under ILO/İŞKUR focused on the impact evaluation of youth employment programmes; it appears in our corporate portfolio directly as a youth employment impact evaluation.
In work of this kind it is not enough to count those who found work after the programme. If 300 of 600 participants found a job, the success rate might look like 50%. But the real evaluation question is: how many of those 300 would have found work anyway?
This is where the comparison or control group comes in. In the KİGEP impact evaluation, programme beneficiaries and a comparison group were followed together. That difference is the difference between a result and an impact.
Counterfactual
At the centre of impact evaluation sits a state we cannot observe: what would have happened to the same person had they not joined the programme?
That is the counterfactual. We cannot observe the same person as both participant and non-participant, so we construct a comparable group through appropriate methods — a randomised control group, a matched comparison group, a similar region, a similar institution or similar individuals.
RCT & Experimental Evaluation
Where conditions allow, a randomised controlled trial is one of the strongest designs available. Participants are assigned at random to intervention and control groups, which strengthens the case that the two groups were comparable to begin with.
ID’s portfolio includes an RCT implementation of interventions to prevent child marriage under UNICEF, with multiple target groups, panel follow-up and back-check processes.
In designs like these, randomisation is not the only critical element. Panel attrition, contamination between groups, implementation fidelity and data quality matter just as much.
Quasi-Experimental Evaluation
If randomisation is not possible, evaluation does not stop. In the real world not every programme can be randomised: it may already have started, participants may already have been selected, or ethical and operational reasons may rule out a control group.
Quasi-experimental methods can then be used: difference-in-differences, propensity score matching, regression discontinuity, interrupted time series and other comparative approaches.
Our governing principle in choosing a method is simple: we do not make a stronger causal claim than the data permits.
An evaluation story
Is a 20% Increase Really Success?
Say average employment in the firms of an entrepreneurship programme rose from 5 people to 6. At first glance: a 20% increase.
But if average employment in non-participating firms in the same sector also rose, from 5 to 5.8, the programme’s observable net contribution looks rather different. And if firms with high growth potential were selected into the programme in the first place, selection bias may be at work.
This is why we do not settle for the before-and-after difference. We test the alternative explanations.
Process Evaluation
Implementation matters as much as the result. If a programme did not produce the expected effect, there are two main possibilities: the programme theory was wrong, or a well-designed programme was poorly implemented.
Process evaluation helps separate the two. We look at whether the intended group was reached, whether participants were genuinely exposed to the programme, whether implementers followed the protocol, whether the intensity of activities was sufficient and whether local conditions affected delivery.
That way, instead of saying "it did not work", we can understand why it did not.
Mixed Methods Evaluation
Numbers can tell us what happened; people can tell us why. We find it powerful to use quantitative and qualitative methods together in evaluation.
Quantitative analysis may show that women’s continuation rate in a programme is lower than men’s. Understanding why may need in-depth interviews, focus groups and stakeholder interviews. The problem may not be the programme content at all — it may be transport, care burden, class times or employer behaviour.
Mixed-method evaluation brings "what happened?" and "why did it happen?" into one research frame.
KAP & Behavioural Change
In some programmes the target is not employment or income but a change in knowledge, attitude and practice. KAP indicators can be used in that kind of work.
ID’s UNICEF / Ministry of National Education early childhood education KAP study covered access and service quality across 12 provinces.
In studies of this kind, baseline–endline comparisons can be built on knowledge scores, attitude scores and behavioural indicators. Operationalising KAP indicators, developing scores and indices and computing baseline–endline change are part of our M&E capacity.
Disaggregation
An average does not represent everyone. A programme’s overall result may be positive while the picture in subgroups is very different.
We therefore disaggregate evaluation results by sex, age, region, nationality, disability, income group and vulnerability status as needed.
A programme may show an overall improvement of +8 points — while the figure is +14 for women, +3 for men and zero in rural areas. That information can change the programme design entirely.
Monitoring Systems & Dashboards
Monitoring is not an annual report. A good monitoring system makes decisions possible while the project is still running.
KPI systems, target-versus-actual tables, early warning indicators, field progress dashboards and Power BI reports can be developed as needed.
But the purpose of a dashboard is not to produce colourful charts. It is to see a deviation early enough to act on it.
Evaluation Data Quality
You cannot run a good evaluation on bad data. Because M&E measurements are compared over time, data quality carries extra weight here.
Asking a question one way at baseline and another way at endline looks like a small change. It is enough to make it unclear whether an observed difference came from the programme or from the instrument.
We therefore pay particular attention to baseline–endline comparability, identical indicator definitions, standard collection protocols, panel tracking, back-checks and verification. Designing measurement instruments consistent with the KAP indicator framework, and preserving baseline–endline comparability, are explicit parts of our technical approach.
Evaluation is for Learning
Evaluation is a learning instrument, not an audit. Its purpose is not only to separate successful projects from unsuccessful ones. The more important question is what we learned.
Which approach worked? In which group was the result stronger? Where did the bottleneck form? Which assumption turned out to be wrong? What should we change next time?
Our M&E approach is therefore learning-oriented: at the end of an evaluation we try to produce not only findings but actionable recommendations.
