When "parachuting in" isn’t an option
Exploring value with integrity across languages, continents and time zones
Julian King and Adrian Field
In recent years we've all become more familiar with videoconferencing technology, relying on it for everything from meetings to professional development workshops and even program evaluations. The COVID-19 pandemic accelerated the growth of videoconferencing. This was no blip, but the acceleration of an ongoing trend towards online engagement. By 2032, the market for this technology is projected to reach $22 billion.
While the convenience of remote engagement has revolutionised communication, it also raises questions about its potential impacts on the quality and safety of our work. Are we fostering genuine connection, or are we merely exacerbating the problem of consultants “parachuting in”?
In a recent presentation at the Australian Evaluation Society (AES) Conference in Melbourne (September 2024), we explored these questions through the lens of our evaluation of Urban 95, an international initiative to transform cities for young children and their caregivers by considering urban spaces from the height of a three-year-old: 95 cm. Our focus was the municipality of Jundiaí, Brazil, applying the Value for Investment (VfI) system to explore the value of initiatives led in partnership with Ateliê Navio, a design consultancy. In late October, Adrian Field and Thomaz Chianca reflected further on this project at the American Evaluation Association conference in Portland, Oregon. This was part of a panel presentation about applying rubrics in diverse ways and across diverse contexts.
Our evaluation was conducted almost entirely remotely, and this brought some challenges and opportunities worth sharing.
Here we’re going to look at three key aspects of our approach:
Addressing the "parachuting in" challenge;
Developing a theory of value creation; and
Using rubrics to facilitate collaboration.
The “Children’s Factory” in in Jundiaí - a repurposed former factory space
Urban95 - seeing the world through the eyes of someone 95cm high - a three-year old
Innovative street design in in Jundiaí
“If you could experience the city from an elevation of 95cm – the height of a 3 year old – what would you change?” - VanLeer Foundation.
The “parachuting in” challenge: bridging cultures remotely
This evaluation involved working across three time zones:
Jundiaí, Brazil, the municipality where the investment took place;
Auckland, New Zealand, the home of Dovetail Consulting, the evaluators; and
Den Haag, Netherlands where the Van Leer Foundation is headquartered.
The evaluation engaged with a mix of Portuguese and English speakers.
Evaluations conducted remotely can sometimes feel as if consultants are "parachuting in" - engaging superficially without the deep context needed for authentic understanding. We did not want to be those consultants! Avoiding this problem was a priority for us. Our guiding approach was the Value for Investment system, providing a structured, participatory framework that invites diverse perspectives and emphasises collaboration. Despite the remote setting, we ensured participation from all stakeholders through a series of workshops, which included developing evaluation questions, building a theory of value creation, developing rubrics, and collectively making sense of evidence.
We prioritised some specific principles to bridge cultural gaps. The concept of contra proferentem, originating from contract law, reminded us that ambiguity should never disadvantage participants. This helped guide our evaluation design and interactions to ensure fairness and clarity, especially for stakeholders unfamiliar with evaluation language. Another guiding principle was manaakitanga, a Māori term from Aotearoa-New Zealand emphasising hospitality, generosity and care for others - values that helped us create a welcoming and supportive evaluation process.
A crucial factor supporting an inclusive, responsive, contextually viable and meaningful evaluation was a Babel fish. What’s that? According to The Hitchhiker’s Guide to the Galaxy, a superbly funny book written by Douglas Adams in 1978, a Babel fish was a small, yellow fish that could be inserted into a person’s ear to provide instant translation of any language, effectively eliminating language barriers. (More recently, the term has been revived in products like Google’s Pixel Buds, which aim to provide real-time translation via earbuds connected to smartphones).
In reality, we didn’t have either sort of Babel fish. But what we did have was Thomaz Chianca, a Brazilian evaluator and native Portuguese speaker with a PhD in evaluation. Thomaz played an invaluable role, not just in translating between English and Portuguese, but crucially by explaining the meaning of evaluation concepts that require more than literal translation, and bridging cultural understandings between the principally New Zealand-based evaluation team and stakeholders. He, along with Caren Mattiello, who assisted with translation from New Zealand, ensured that evaluation materials were accessible in both languages. Together, we scripted our presentations ahead of time to facilitate smooth transitions between English and Portuguese, making the process well-choreographed and inclusive.
Despite the challenges, we learned that genuine engagement can be achieved remotely. However, in some cases, face-to-face interaction is still necessary - particularly for community stakeholders, who need to speak with and within their communities. Building relationships remains at the heart of evaluation, and meaningful conversations take time and cannot be rushed. Ultimately, Caren conducted some face-to-face meetings to supplement the substantial online engagement. The quality of this engagement and the continuous adaptation to participants' needs strengthened the credibility of our findings.
Theory of value creation: thinking beyond impact
When evaluating Urban 95, we needed to understand not just how the initiative intended to bring about change, but also the value it aimed to generate. While a theory of change explains how a program or policy is expected to lead to intended outcomes, it typically doesn’t specify the value the program will generate (i.e., the merit, worth or significance that key stakeholders place on Urban 95 and its impacts).
To supplement the theory of change, we developed a theory of value creation, detailing the value proposition of Urban 95 and how the resources invested could be used efficiently and effectively to generate equitable social value. The idea of building capital assets was central to this theory. Urban 95 aimed to build physical capital by improving urban infrastructure and create spaces that are safe, fun, and accessible for children. This in turn was expected to foster social capital through stronger connections between children, caregivers, and community networks.
The significance of capital assets is that they last across multiple accounting periods and generate ongoing returns – but in order to get those returns, the capital assets first have to be grown, and once grown they require ongoing nurturing. We posited that if Urban 95 managed resources efficiently and equitably, it would build physical and social capital over time - leaving legacies that would contribute, ultimately, to positive outcomes for children and their communities. This holistic approach allowed us to understand not just whether resources were used efficiently, but whether they created sufficient value to justify the investment – where the value proposition was tied substantively to capital-building with relatively less emphasis on longer-term, harder to measure and attribute social returns.
Portuguese version
English version
Rubrics for collaboration and transparency
To support collaborative evaluation design and transparency in evaluative judgements, the evaluation featured rubrics co-created and used with stakeholders. The rubrics identified how well the Urban 95 investment was expected to perform against an agreed set of criteria (key aspects of the theory of value creation) and standards (levels of performance, ranging from "just good enough" to "excellent”). During workshops with implementation partners, we worked together to define these performance levels.
Although these rubrics only explicitly defined two levels of performance, these levels allowed for interpolation to four levels in practice, providing a clear understanding of progress and helping identify areas that needed improvement. Anything that did not meet standards for “just good enough” would by definition not be meeting expectations. Similarly, anything that is above “just good enough” but below “excellent” would be judged as being on a pathway to excellence.
By making evaluation criteria explicit, we facilitated honest, constructive conversations about the value being generated.
For example, here are the rubrics for building physical and social capital as described in the theory of value creation:
A note on economic analysis
While economic analysis often forms part of the VfI approach, the evaluation of Urban 95 did not include a social cost-benefit analysis (CBA). Assigning monetary values to intangible benefits, such as positive impacts on children's lives that could be attributed to the intervention, proved challenging. The complexity of the intervention, alongside the lack of robust evidence for monetising social impacts, made CBA infeasible in this case. Instead, VfI served as a viable alternative, allowing us to consider multiple forms of value through the use of rubrics and mixed methods, without monetary valuation.
Final reflections
The Urban 95 evaluation in Jundiaí demonstrated that meaningful, remote engagement is possible, though it requires careful attention to cultural and linguistic nuances, genuine relationship-building, and an inclusive, collaborative approach. The feedback from one of our evaluation participants reflects the impact of this process:
"This process was really, really important for me and for my team because … we have expertise in cities, …early childhood as an architect and urban planner but we did not have expertise in monitoring. And it was incredible to see this process because we had all these workshops that were really hard. I remember each one in a different country and I remember my head started to burn, like trying to do together the theory of change, all the translation going on and it helped me so much to see what we were doing and to question myself and my work culture. … What's more important is are we changing really what we are doing? Like are we having value for children in Jundiaí? So it was really beautiful to see all this process and make me see also how can we improve that in our day-to-day work.” (Key Stakeholder)
Our experience with Urban 95 shows that videoconferencing can support evaluations that are inclusive and impactful, even across distances. Our approach intentionally sought to share power throughout the evaluation. By embracing collaborative principles, respecting cultural contexts, and prioritising relationships, we can conduct evaluations that not only meet but exceed expectations, creating real value for those involved.
Here's the report if you’d like to take a peek:
Field, A., King, J., Moss, M., Parslow, G., Schiff, A., Chianca, T.K., Chianca, G., Mattiello, C. (2024). Urban 95/Ateliê Navio Value for Investment evaluation. Report for Van Leer Foundation. Dovetail Consulting Limited, Auckland. https://vanleerfoundation.org/publications-reports/value-for-investment-study-urban95-in-jundiai/
We would like to acknowledge all participants in the evaluation, including the Urban95/Ateliê Navio and municipality leadership, families, caregivers and community leaders. All gave generously of their time, thoughts and reflections. Your critical input to this evaluation is gratefully received.
Our grateful thanks are extended to our Evaluation Advisory Group of leaders from Jundiaí Municipality, Van Leer Foundation and Ateliê Navio for their guidance and reflection, particularly during the design of the evaluation. We also wish to acknowledge the provision of data for analysis by Jundiaí Municipality. This Value for Investment evaluation was funded by the Van Leer Foundation.
And finally, thanks to Thomaz Chianca for peer reviewing this article and for your essential support and guidance throughout the evaluation.
