Comments

Thanks for this interesting blog and discussion paper. A resurgence in oil prices will probably wreak havoc again on many of the small island economies, sad to say. The 2008 fuel crisis was a real wake up call for the Marshalls, as you mention. The silver lining on that cloud was that it forced leaders to focus more attention on reforms. The major electric utility was already developing a reform plan when the crisis hit, forcing it to fast-track the process (it was aleady insolvent prior to the crisis!). The crisis also resulted in RMI developing its first comprehensive national energy policy and action plan, which emphasizes energy efficiency, renewables, and other initiatives. Australia, ADB, and other partners responded with good support. Since 2008 we've seen a wide range of energy projects and initiatives get moving. But.... much more could have been done to strengthen the country's resilience, both in terms of its energy security and its fiscal position. As they say, history repeats itself... because we don't always learn our lesson the first time around!

The issue of publishing evaluation is who goes first. Dinuk seems to suggest the NGO should start and publish all of their evaluations (the definition of 'all' becomes interesting here), but many do publish program overviews ( more than only CARE) which are meta studies of all their evaluations and key lessons learnt. Partner program evaluations are also published on the web, for example the We Can End Violence against Women program in South Asia (supported by multiple donors) AusAID does not publish all of it evaluations (it is fairly selective), nor does the World Bank. In line with Chris and Garth comments most useful evaluation are those to the partner community, country, or NGO, depending on who the partner is. I have done a Third Party Assessment for the partner government of a large decade long World Bank program. I suspect neither the partner government of the World Bank would like it published, but it was useful to the partner government as it gave a clear steer on how future loans may be directed in that sector, and how it might direct is own investment. Likewise evaluations I undertake for NGOs are largely directed to the partner as they are the ones that it is most useful to guide them in the negotiations with donor NGOs. I am not sure AusAID encourages recipient government to review the AusAID programs but that would be a much greater step forward in transparency than mere publication. Most published evaluations focus too much on outputs than outcomes, and the authors are very mindful of who is paying the bills. Perhaps greater attention could be to academic studies of development effectiveness at local level which look at these issues particularly from the recipient point of view, as these programs invariably have multiple donors. For example I have looked as effectiveness in a sector across 15 small and medium NGO in India. Published studies like these may yield more that published evaluations with their focus on short term results.

The ‘Capability Trap’ papers highlight the importance of mapping the local organisation context as a precursor to selecting and lifting administrative capability. Organisations come with their own histories and operate in specific contexts. I have found that two factors impact on the goals, methodologies and markers negotiated to define measureable success. I currently work in Timor-Leste and a feature of donor literature is a tendency to define the development of civil service expertise as a post-independence phenomenon. The scorched earth response, which included mass murder, has contributed to the narrative of a ‘phoenix’ nation, rising new and untested from adversity. The Indonesian civil service decamped, taking with them managerial and technical expertise, leaving the United Nations Transitional Authority in East Timor (UNTAET), with the responsibility for establishing and ‘capacity building’ a cadre of civil servants across the entire spectrum of government with an unskilled workforce. The work continues today with a low capacity workforce...The human resources reality is however, more complex. At one stage I conducted interviews with all staff in three Directorates (63 people) and was surprised to find that 48% of people had been previously employed in similar jobs in the Indonesian civil service for an average of 6.3 years. While few had held managerial or technical roles, one could not characterise staff as being novices to administrative work. This brings into focus Pritchett and Weijer’s point about measuring system performance and developing staff ‘buy in’ to any change process by acknowledging and valuing prior experience. As stated in the article, ‘training’ should take advantage of existing staff skills and knowledge. The context of their former work also needs to be taken into account because it shapes staff perceptions regarding organisational purpose and indicators of success. In the case of Timor-Leste, Indonesian, donor and current staff capacity building/professional development have relied on significant subsidies which unfortunately foster ‘isomorphic mimicry’. I would suggest that the focus of external assistance has to be the work of the organisation and the people who perform that work. If, as advocated by Pritchett et al., that donor agencies move away from concentrating on significant organisational targets towards individuals and their ability to perform their key work tasks, then these tasks need to be clearly defined and measured. Richard Curtain has commented, more than once, that AusAID appears to be averse to actually defining and measuring capacity development in practical terms. There is a tendency for some capacity building approaches, such as the Staged Capacity Building Model (AusAID 2006) and ‘enabling environment’, to value high level outcomes or products over a systematic approach which takes into account the contribution of all members, at all levels of a department to achieving agency objectives. As the articles note, it is possible to have an apparently satisfactory outcome, but one which is so heavily reliant on outside funding, expertise and systems support that it is not only unsustainable, but undermines the ability of recipient governments to develop their own, more modest but effective systems.

Joel, I haven't heard any Australian names come up yet - for the IMF job Paul Keating and Peter Costello were mentioned! There have been some interesting suggestions made and those worth exploring would include: Luisa Diogo, Michelle Bachelet, Ngozi Okonjo-Iweala, Sri Mulyani Indrawati, Trevor Manuel, Jairam Ramesh and Kemal Derviş. Your comment though points to a big challenge - getting most of the countries outside the G8 to stand behind a single candidate. Even if this were to occur they would struggle to achieve the simple majority of votes required could consensus not be reached, thanks to the weighted voting system. On Summers, I could have referred to his more recent comments on women's lack of aptitude for science and maths and taken with the pollution comments these suggest that Summers may not have the diplomatic skills required to lead an international institution. However, his comments about pollution and promotion of financial deregulation in the US are even more pertinent as they give some insight into his view of what development is and how to achieve it. Susan

Thank you Chris and Garth for your comments. I agree with your comments that ideally, the poor in our communities should have a say relating to the effectiveness of our aid program. Such actions would reduce the incentive imbalances between the aid agencies and the poor. My suggestion, that we make public disclosure of evaluations a pre-requisite for NGO accreditation (and therefore funding), is one of the first steps that can be made to improve transparency. This is unlikely to be too difficult for AusAID to implement and administer. A next step could be ensuring that publically available NGO evaluations are up to a particular standard and that those NGOs receiving core funding from AusAID are evaluated by someone independent of both AusAID and the NGOs. These actions would allow any interested Australian to view whether their taxes are being spent appropriately – as arguably the Australian taxpayer is the most important (albeit on average the least interested) stakeholder. But these are not the only steps – other steps to improve transparency are as you suggest by “implementing, sharing and publicising innovations” and making activities accountable to the poor.

I think Chris has the right emphasis here. Evaluations are important (especially if we can get people to learn from them and apply the lessons - not often achieved), but shortening and strengthening the lines of accountablility to poor communities is more important. Dinuk cites profit as a critical motivator in the commercial area. Profit is not generally generated by the assessments of expert evaluators of the commercial transaction, but by the powerful choices of people on the ground. While there have been many improvements in AusAID processes recently, making activities accountable to the poor, seems to me, to need a lot more focus and work by AusAID management.

Thanks Stephen for this useful focus on AusAID's activity rating system. Yes - making as much data available as possible is necessary for contestibility and is likely to provide useful information for AusAID at the same time. This sort of rating scheme provides an important summary of activity performance but is not sufficient. It is good to see that AusAID's planned performance framework is going to include a lot more clear measures of output (eg x thousand children immunized) and outcome (eg achieve a y% reduction in child mortality). We've had too many years of satisfactory scores for activity performance but few clear and concrete measures of poverty reduction.

Susan, Thanks for this blog. Who might non-US candidates be? If the world rallies around a different candidate who has a great reputation, then there might be an interesting challenge. Maybe Kevin Rudd could be farmed out to lead the Bank...! And I'll throw in my two cents on Summers. I'm not a fan of his but I think bringing up a 1991 memo (that he claims he didn't actually write) to discredit him is not a very compelling argument. There are lots of things he has done that deserve critique but basing our reservations predominantly on that memo says more about the quality and relevance of pure economic thinking than it does about Summers himself. Joel

I certainly I agree with Dinuk that greater transparency amongst International NGOs is important. As the 'Promoting Voice and Choice' report (http://bit.ly/x3rUvT) - which I authored for ACFID in 2010 - noted there have been some improvements in this area but they were not sufficient. Much of the innovation in this area has focused as much on NGOs becoming more accountable and transparent to the people and communities they work with, as on transparency to the Australian public. However these experiences are inadequately shared and often done in a piecemeal fashion. A truly strategic approach to transparency and accountability would see more collective efforts to not only publish reviews and evaluations - which is starting to happen and is important - but also have a focus on implementing, sharing and publicising innovations which allow people living in poverty to be holding the powerful - including International NGOs - to account

Well I really enjoyed reading it. This tip provided by you is very useful for good planning.

Many thanks for your post. In relation to comparisons between Fiji and Mauritius, I would like to highlight the importance of trade preferences for the development of industry in both nations. Subramanium and Roy (2003) found that export manufacturing was the foremost reason for the economic success in Mauritius and that furthermore, preferential access to OECD markets for garment manufacturers in Mauritius (particularly relative to Asian producers with obvious competitive advantages in terms of economies of scale and lower-cost production) was important for growth of the sector. Whilst the multi-fibre agreement, which had been important for Mauritus, ended in 2004, it had allowed Mauritius to develop an industry that could shift to more complex, and more value-added, manufacturing tasks. I won't go into the full details here (perhaps another blog piece is in order) but the preferential access to Australian and New Zealand markets for Pacific manufacturing exports, provided under SPARTECA, were also absolutely vital for the explosive growth of garment exports from Fiji from 1987 onwards (though Rabuka's incentives for foreign investors in the form of tax-free factories probably helped too). While maligned by many as an industry that was ultimately uncompetitive, by the late 90's 18,000 people worked in Fiji's garment export industry and it had overtaken sugar exports as the single greatest source of export earnings. Whilst the sector contracted significantly following the removal of subsidies for Australian fabric exporters in 2000 (which had been a key incentive for exporting fabric to often-Australian-owned 'cut, make and trim' operations in Fiji, which would in turn re-export finished articles back to Australia under SPARTECA preferences), and the expiry of the MFA (which had guaranteed a quota of Fiji exports to the US) garment exports from Fiji are still worth around FJD 90 million per annum, and the sector could yet remain a longer term prospect (particularly if the restrictive rules of origin requirements of SPARTECA are reviewed). Sorry to bang on, but the overall point is that meaningful access (and in many cases preferential access) to metropolitan markets has clearly been important for economic growth in both Pacific island countries and the African small island states. A lesson to be drawn is that Australia can do a lot more to improve market access for Pacific exporters.

Hi Stephen, Great blog - thanks. Perhaps you or colleagues have addressed this elsewhere but who is the ODE? It's internal to AusAID, right? So it's not independent assessment? Is that something you think AusAID needs or do you think an independent review every 5 years is sufficient. Most aid NGOs have independent evaluations for their projects - should AusAID have the same? Having AusAID staff evaluating AusAID projects for AusAID managers that they might work for in the future sounds a bit tricky... Would be interested in your thoughts. Joel