Mapping the impact of key city-wide cultural programs

Featured Story

December 2019

developmental evaluation map

Toronto Arts Council’s vision is to  connect every neighbourhood with the transformative social and economic benefits of the arts.  To that end, TAC regularly evaluates its programs and processes to better understand the outcomes and impact of our work.

One recent such assessment was led by local firm PROCESS, a team of urban planners, designers, and artists who teamed up with social impact firm Openly and Gladki Planning Associates to undertake a developmental evaluation of five City of Toronto and Toronto Arts Council cultural programs working with artists, community organizations, and residents outside Toronto’s downtown core.

"The evaluation was a review of a number of art and cultural programs across the city in order to understand the collective impact and opportunities for cross-learning about the strengths and gaps of the various programs," explains Nadia Galati, co-principal at PROCESS, who helped lead the project alongside fellow co-principal Sara Udow (who also provided insights for this story).

The evaluation – which began in spring 2018, with a final report delivered earlier this year – examined five key programs, including Cultural Hotspot, StreetARToronto, Arts in the Parks, Animating Historic Sites, and Artists in the Library.

As part of the review, PROCESS worked with Openly to utilize a developmental evaluation approach grounded in collaboration, with a focus on mutual learning and growth rather than a simple success-or-failure appraisal. PROCESS also applied a user-experience, human-centered approach to the evaluation, illustrating the findings from five user perspectives: the public, artists and cultural producers, community partners, staff, and management.

To gather their evaluation data, PROCESS conducted site visits, interviews, surveys, focus groups, and regular meetings with the steering committee. Between July and October 2018, they engaged about 400 people who participated in the five cultural programs to learn more about their experiences.

"Focus groups allowed us to receive in-depth feedback from a few, while the surveys provided opportunity to reach more people," Galati notes. "Interviews and site visits provided an opportunity to meet people where they are, and also allowed us to experience the programs for ourselves."

This hands-on approach yielded insights a more straightforward research process may not have revealed, Galati points out.

"On one of the site visits in a park in Scarborough, we saw intergenerational and multicultural community members sitting around a campfire enjoying food and song," she recalls. "There were many youth volunteers at this event, now in its third year. One youth's parents said their kid eagerly waits for the program all year. A survey may only give one layer of information – actually experiencing the programming and talking to those engaged in a program offers a richness to the information that quantitative data alone cannot provide."

Even the report PROCESS delivered at the end of the year-long evaluation was far from conventional – the 'visual report' features large-scale infographic-style User Experience (UX) maps, a tool that visualizes different user groups/audiences’ experiences with a program to help understand the gaps, successes, and challenges experienced throughout their journey.

"This wasn’t actually the plan," Galati says. "Once we began to speak to members of the public, artists, etc., we realized that sharing the data visually by user groups was the most useful way to illustrate where alignments and differences exist and communicate the complexity of experience (not everyone experiences the same program in the same way – who someone is impacts their experience) in a digestible manner that is interactive and has the ability to tell multiple stories simultaneously."

This visual tool helped City of Toronto and TAC staff to review the findings and immediately see where alignments or differences arise, but the maps can also be used by decision-makers and community members as a resource to better understand how decisions are made.

Click HERE to view the Cultural Programs Developmental Evaluation maps.

"I think the maps are useful for everyone to read and experience," Galati says, offering some tips for how to best understand the data presented in the visual report:

  • The purpose of mapping the user experiences and perspectives from the five user groups is to ensure that diverse voices are considered and included in the recommendations for sustainable and meaningful improvements to the cultural programs.
  • Similarly, each UX map represents a 'snapshot' of experiences. It is not meant to represent one user journey in one specific program, but instead serves as an overview of a collection of experiences.
  • The five maps highlight users' journeys throughout the different cultural programs, illustrated in a flowchart-type of graphic. The x-axis represents a timeline, illustrating experiences leading up to a cultural program project (Pre-Program), during a project (During-Program) and user perspectives of impact after the project occurs (Post-Program).
  • Each UX map also includes a y-axis that acts as an emotion line expressing positive to negative experiences and perspectives. The y-axis also includes user experience profiles for each user group. This helps to better understand a diversity of users’ needs, experiences and perspectives, acknowledging that not all people in one user group have the same experiences.
  • Key improvements/recommendations are included at the bottom of the map (which are also included in a written report {link?}).

Ultimately, the program data showed the five cultural programs are achieving their goal of increasing activity and investment in artists and organizations delivering programming to neighbourhoods outside Toronto’s downtown core, but the evaluation also indicated specific areas for improvement.

"The majority of users find the programs to be successful," Galati says. "The five programs should work together to cross-promote and leverage the great work the other is doing, and could better incorporate outreach methods and guides for artists, cultural producers, and staff to ensure they’re on the same page."

Other takeaways include the need to simplify the application process and offering various ways to apply in order to increase the diversity of participating artists, and to develop anti-oppression/anti-racism training for the arts organizations and City staff involved.

"This process has been helpful to evaluate whether or not the programs are meeting the priorities overall,” Galati adds. “It helps to see where experiences align and where they diverge."