The following, written by Knight’s Vice President of Strategy and Assessment Mayur Patel, is cross-posted from the Global Forum for Media Development. Above: Finding a Foothold: How Nonprofit News Ventures Seek Sustainability from Knight Foundation.

Basic numbers tell an incomplete story about the effects of journalism, according to Mayur Patel, the Knight Foundation’s vice president of strategy and assessment.

If you were philanthropist Pierre Omidyar, how would you measure the impact of your soon-to-be-launched new journalism venture? Omidyar, the founder of eBay and the venture philanthropy network that shares his name, recently committed $250 million to starting a new mass media organization with Glenn Greenwald, the former Guardian reporter who broke a series of stories on national security surveillance. 

You might suggest Omidyar track the number of stories his journalists publish, the quality of the reporting, and the number of people who read their articles each month. Are these the right metrics? They don’t seem to get at real impact, something Omidyar has put at the heart of his philanthropy. Maybe the focus should be on public policy changes and shifts in legislation. But what happens if the journalism produced isn’t so narrowly defined? Welcome to the media measurement merry-go-round.

Those of us who work in media development believe that democracy would be poorer without good journalism. Journalism itself must have some democratic and social value, then. The problem today is that the measures we commonly use have been shaped by thinking about media’s economic value to advertisers. As Jonathan Stray noted, a series of public media impact summits summed it up best: the “usefulness of tools in this arena is limited by their focus on delivering audiences to advertisers.”

The primary metrics – uniques, times on site and page views – are really about tracking eyeballs and traffic. And although they’re fairly faulty and can be easily gamed, we still rely on them. Even when these metrics do give you a general sense of the number of people who may have encountered your story, they tell you little about who read it and what they did with that information. How do we get beyond just looking at the reach and frequency of journalism? How do we start tracking the impact of great storytelling and reporting on people’s personal and civic lives? Thankfully, a vibrant discussion on this topic is starting to take hold with variousjournalists and editors pushing for better ways to assess media’s effects on people’s attitudes and behavior.

This is an exciting time to be having this conversation. As our media environment has become more highly immersive and interactive, we’re now surrounded by a wealth of data generated by people’s consumption and production of digital media. Importantly, this data is accessible at the level of an individual story or article, rather than just a whole publication. This is the kind of information we could use to improve how we understand the ways people consume information, and how this leads to greater awareness, knowledge and even changes in behavior. Yet the tools and approaches we have today to measure and analyze this data are underdeveloped.

Some media organizations are making progress in how they track their impact. Their experimentation in this area stems from a commitment to informing audiences and not just publishing content. And in some cases they’re using this feedback to make their journalism more relevant, timely and impactful.

Grist, a nonprofit news organization focused on environmental issues, has developed simple ways to measureengagement with audiences that guide its content strategy. Started more than 15 years ago by journalist Chip Giller, the organization declared its vision is to “make green second nature” for people by slicing in humor and popular content with original reporting on sustainability. Rather than just looking at broad reach metrics, Grist tracks indicators that give it a sense of its churn and retention rates, such as repeat visitors on a monthly basis. In parallel, the organization regularly surveys users on its website and conducts an annual in-depth reader survey to understand how its reporting influences the environmental actions of its core audience.

There isn’t one simple metric or formula we can put into practice that will magically reveal the impact of media. Instead it requires experimentation with different ways of quantitatively and qualitatively gathering insights. ProPublica, a large nonprofit investigative newsroom, for example, uses a suite of qualitative narrative accounts it tracks as part of its Impact Report Card to understand the influence of its stories on broader policy changes (see its white paper on this). In its Dollars for Docs investigative project, ProPublica created a single, comprehensive database that allows patients to search for their doctors or medical centers and see the payments from pharma companies made to them. In tracking the impact of this reporting, ProPublica looked at how its national story about the relationship between corporations and health care providers was localized in more than 100 towns and cities across the United States and tracked the responses of hospitals and universities, such as the Medical College of Georgia, which ended up introducing new restrictions on the payments allowed from pharmaceutical companies. Given the slow pace with which public policy and organizational practices change, ProPublica’s team has made a concerted effort to the follow the long tail of their journalism after the organization’s projects are complete.

For-profit media, such as The New York Times, are also starting to move in the direction of tracking the impact on the issues they cover. The New York Times’ Interactive Editor, Aron Philofer, hired a recent Knight-Mozilla fellow explicitly to work inside the newsroom to help measure the social impact of the newspaper’s reporting. Some of the areas they’ve explored include developing new metrics for gauging people’s interaction with their data applications and using promotional data, such as the time an article stays on an organization’s home page, to better understand the true value of an article’s page views. Through this work, they’re trying to refine their use of traditional online metrics that often track a story’s reach without information about the context in which it is published and promoted by a news organization.

Recognizing the challenges faced in measuring media impact, the Knight and Gates foundations partnered at the start of this year to launch a hub for experimentation with new techniques for collecting and sharing best practices in understanding shifts in people’s knowledge, attitudes and behavior related to their engagement with media. The Media Impact Project, housed at The Norman Lear Center within the USC Annenberg School for Communication and Journalism, brings together a range of interdisciplinary skills (data analytics experts, journalists, and social and behavioral scientists) to help advance how we study the impact of media on society.  As the project progresses, we hope it becomes a key resource that helps shape practices within media organizations. Making progress on these issues will take time, and as the project gathers momentum, we hope other partners and funders will join the effort.

Fixing the way we think about media measurement has the potential to be transformative. It will be a great asset to content creators who want to understand what works and how to stay relevant in an environment of rapid change.