by Mayur Patel

Facilitator: Mayur Patel, John S. and James L. Knight Foundation

Scribe: LuAnn Lovlin, The Winnipeg Foundation

Information and Digital Media

Citizens require relevant, contextual and authentic information to run their lives and to manage their communities. Information can also be a creative and powerful tool for achieving social change. But how do we evaluate whether our media and communications projects make a difference in the world, how do we make our projects more effective and how can we learn lessons for the future? These questions formed the main part of the discussion during the breakout session.

Information and communications projects cover a wide variety of activities, including TV, radio and film, telecommunications-based projects, broadcast campaigns and web-based projects.

In evaluating the progress and impact of these activities, our approach should be guided with the end user in mind (ordinary citizens, communities, organizations or policymakers); be inclusive of all participants and contain credible and accurate information.

Our Challenge

There are several practical difficulties involved in evaluating the impact of information and communications projects.

• The goal of these projects, which often includes outcomes such as greater public awareness, increased dialogue and empowerment are difficult to measure objectively, especially when compared to projects that provide direct services, such as education, health and housing.
• It can be difficult to define a specific target audience for a media initiative (some projects are geographically defined, whereas others involve wider forms of broadcasting).
• Social change often happens slowly, which makes it hard to capture a project’s impact over a short period of time.

Understanding Social Change

During the breakout session, participants discussed how to adapt traditional logic models to the area of information and digital media, namely: a) how to conceptualise the linkages between outputs, intermediate and final outcomes, and b) what leading indicators reveal whether a project is on track to achieve success. Depending on the particular project these might include:

• Outputs: the launch of a website; adoption and utilization rates; levels and depth of distribution within a community; the establishment of strategic partnerships with other organizations, e.g. newspaper, community centers etc.
• Intermediate Outcomes: increased dialogue on a particular issue within a community; analyzing site traffic peaks during times when local issues are discussed; greater collaboration between different members of a community; higher levels of deliberation.
• Final Outcomes: changes in public awareness of local issues; changes in individual attitudes or behaviour; greater social inclusion; the formation of new advocacy related efforts or organizational practices; changes in local state policies.

Tracking Online Behavior

The discussion about outputs and activities focused on the need to capture online patterns of behavior. Increasingly media projects make use of web-based technologies and social media tools, which provide us with the opportunity to track a variety of indicators of people’s behavior. These include:

• The number of unique visitors, frequency of visits, time spent on site, depth of visits; referring URLs; natural search results; number of registered users; responses from the blogging community; technorati authority ranking; number of RSS subscribers; conversions of visitors to contributors on the site etc.

Organizations can use several free online tools and statistical packages to capture and analyse these indicators, including Google Analytics, Woopra (especially useful for less trafficked websites, such as blogs) and CrazyEgg, among others. However, measuring clickstream data does not capture people’s levels of engagement and does not provide insights into why members of a community are using a website, are they completing what they set out to do and are they satisfied? For this qualitative data is important. Here a few tools have emerged, which allow organizations to survey their website visitors, e.g. 4Q. The next step, understanding whether, and how, levels of online engagement lead to offline action in communities remains an ongoing challenge.

What have we learned so far?

At the close of the session, four principles were highlighted to guide evaluations of information and communications projects:

1. Capture real time feedback – increasingly information projects have an online component, involving the use of social media tools and web-based technologies. This provides us with an opportunity to analyze the formation of new relationships, conversations and networks, provided that we’re actively looking out for these activities.
2. Recognise the importance of outputs­­ – social change takes a long time, and so we should be sure to value a project’s ability to achieve particular milestones. These outputs should be thought of as set of leading indicators of a project’s potential success in the long run. In setting these outputs, think about what would reveal whether a project was on track to have the greatest possible chance of success.
3. Flexibility is key – given the rapid pace of technological change, the field of information creation, sharing and gathering is constantly evolving. This makes it difficult for projects to anticipate potential obstacles. Organizations and foundations should be cognisant of these challenges.