Comments

Captivating blog. Thanks Lindy.

Thank you for this honest and inspiring article.

The article pretty much explained the current situation and I believe the expenditure has gone passed the revenue generated in PNG. 'MP fund' is one of the area that experienced relative increase especially during the boom period and I guess is continuing today. What does 'MP fund' mean? I tend to understand that MP fund refer the funds that is discretional to the MP, hence the relative increase has seen more discretional funds to the MPs.

Your reminder that these impact evaluations at best tell us "what worked" rather than "what works" is a good one. We should be looking to draw evidence from a many evaluations rather than a very small number. That is why 18 (plus or minus a few) for PNG and the Pacific isn't going to tell us very much.

At least one paper in that ACIAR series uses the term impact evaluation in a way that does not seem consistent with 3ie's criteria; other papers in the series are about "impact assessments", which also do not seem consistent 3ie's criteria. From limited sampling, it is likely there is an issue with the methodology from a 3ie perspective, although I'm not saying there is any wrong with the methodology based on the objectives of the ACIAR papers. On the other hand, the ACIAR series simply may not meet 3ie's criteria on publication.

it's great this conversation is opened up. i come from background of 25 years of communicating about aid (as a fundraiser).lessons learned: 1. don't talk about aid, talk about people 2. 'hero' is not Australian Government or DFAT or formerly AusAID, it is 'Australia' or 'Australian people' 3. go to where audience is 'at', not where we'd wish they would be 4. simple, emotional, first person 5. show problems 'fixed' 6.. and be patient 🙂

Hi, thank you very much for the information.

The table doesn't seem to capture any of ACIAR's impact evaluations - is there a reason for that? Are they not considered to be methodologically sound? There are quite a few for the Pacific - see <a href="http://aciar.gov.au/publication/term/25" rel="nofollow">here</a>

Ashlee - Your arguments rest on the assumption that DFAT management cares about aid.

It is not for me to tell 3ie how to categorize things, but as an author of several of those 18 papers I wouldn't consider many of them as "impact evaluations". By my definition, perhaps only the financial literacy study, which had explicit investigator-driven randomization. We would hope that all econometric studies that hope to uncover causal effects would be using good research designs that are not (too) prone to omitted variable bias (aka selection effects). Thus, I would think that the work we did on transport access and poverty in PNG using an IV strategy would qualify despite 3ie omitting it. When there is randomization available, as with the migration lotteries, it is good to use it. But it is hardly the silver bullet that too many development people think it is ("what works" should be more properly called "what worked, in that particular context when implemented by those particular people" because absent a theory as to why it worked there is no guarantee that it will work elsewhere - e.g. Rozelle et al Stanford REAP have new results on deworming that show no effect on school performance in rural Western China, contrary to the very publicized effects in Kenya).

For those interested, <a href="https://dfat.gov.au/aid/how-we-measure-performance/ode/Documents/impact-evaluation-discussion-paper.pdf" rel="nofollow">here is a discussion paper</a> on impact evaluations from AusAID / ODE. The paper is from 2012 and shows that greater use of impact evaluations was on the radar. Perhaps the embers of the Impact Evaluation Working Group are still warm?

Thanks, Ash, for this great policy brief and blog series. As far as I can tell, DFAT (and other aid agencies) have almost nothing to lose and everything to gain from communicating more, and more openly, about what Australian aid does and the difference that it makes in people's lives. I think your recommendation on increasing engagement mainstream media is a particularly important one; I imagine many of the people who go to the aid pages of the DFAT website or follow it on Twitter are already well-informed about Australian aid -- the challenge is reaching the broader public.