bgm: ac/re

where arts & culture meet research & evaluation

[bgm-ac-re]

Sustainable development: Will we ever know if we’ve achieved it?

The UN has just passed a series of Sustainable Development Goals (SDGs). Seventeen bold goals for the world: No more hunger! Education for all! Watch the video to see what they’ve agreed to work toward. It’s a beautiful picture of what humans could achieve.  

Leave it to Dr. Tom Schwandt of the University of Illinois at Urbana-Champaign to bring a sobering voice to the party. How will we ever measure progress toward those goals? And if we can’t measure them, how will we ever know if we achieved them? 

“If evaluation is a religion,” he says, referencing an earlier comment by Dr. Deborah Rugg, “we might have to do a little praying.” He brings up five issues the UN will face in trying to measure progress. 

1) The SDGs are meant to be universal, but it’s not clear yet what policies and programs will be needed. The challenges to achieving sustainable development are different for low, middle and high income countries. How can we collapse both interventions and their measurement into a global summary, or even a global picture? How can we structure meaningful reporting on a global scale? 

2) There is an extremely strong focus on data, both availability and quality. There’s an entire UN subgroup assigned to coming to agreement on a set of indicators for the 17 SDGs. So far they’ve come to agreement on 159 indicators and are still debating another 20 or so. Big data, citizen-generated data, other types of data, what are the risks of data collection and ownership? The real danger here, Schwandt says, is that a focus on data for indicators leads to focus on monitoring which is not the same thing as evaluation. How do we do evaluation without falling into the monitoring trap? 

3) Schwandt suggests reading a blog post by Caroline Heider of the Independent Evaluation Group at the World Bank. In sum, she points out the 17 SDGs are so complex, integrated and interrelated that standard linear models for measuring and impact will not work. We will have to develop new strategies for incorporating systems thinking and methods of doing systems oriented evaluation.

4) Attribution and contribution: how can we possibly measure which policies and programs and in what dosage work? And in which settings? 

5) Finally, there’s that tricky issue of scale. It’s hard enough to measure some of these outcomes at the country level. How can we roll up findings to the global level, taking into account inter- and intra-regional differences in measurement.

[bgm-ac-re]

I learned evaluation as a science, practiced it as an art, and continue to believe in it as a religion.

Dr. Deborah Rugg, senior advisor to US mission to the UN

[bgm-ac-re]

At the American Evaluation Association poster session I saw dogs, dragons, kittehs and multivariate regression analysis. Watch the video here!

[bgm-ac-re]

The LA Times just did a great dataviz of arts education data from the LA Unified School District. I was curious about how they did it, so I opened up the code to take a look.

When I scrolled down to the very bottom, I found this little Easter egg hiding in the code. Click on the image to get a closer look.

[papertownsy]

THIS IS REBLOG RELEVANT FOR ONLY TODAY IN THE WHOLE OF HUMAN HISTORY AND ITS FUTURE

[artsresearch]

Data: not just numbers; not always big

Pity the poor arts administrator who has drunk the Big Data kool-aid without fully understanding its strengths, limitations and ramifications. 

Who thinks data only means numbers (”But how can I quantify that?”) without knowing the many varied research and evaluation tools available to them. See: An intro to qualitative methods.

Who thinks research always begins with a survey. See: What is a research question?

Who thinks you always have to collect original data yourself. See: What is secondary data? 

Who doesn’t take the time to find out if the biggest questions the organization is asking have already been researched and answered. See: How to write a literature review.

Who doesn’t consider the potential ramifications of null (or negative!) results before launching a study. See: Research on why researchers don’t publish when their study turns up nothing terribly interesting.

Who has fallen for the Big Data Gurus who argue that quantity can overcome non-representative methods. See: A very technical article about overcoming bias in large nonrepresentative internet datasets.

Who asks grantees to “prove” the “impact” of their programs without understanding what it truly takes to measure cause and effect in the social sciences. See: Actual news headlines that mistook correlation for causality.

As an arts administrator, you don’t need big data, you need accurate information. You need to ask smart questions and provide guidance to your research/evaluation staff or contractors about what’s important to you, then let them figure out what the best methods are to answer your questions. Before diving into the research, make sure your question hasn’t already been answered. If not, see if there is secondary data available already to answer it. Good quality research takes time and resources - make sure you have enough of both to commit to the project.

Data is only as powerful as the people who wield it. As you would with any sharp instrument, use it with care and respect and it will serve you well.

[thetinhouse-deactivated20180604]

Happy birthday to Ursula Le Guin! Here’s a great letter penned by Le Guin from Letters of Note in 1987.

[artsresearch]

This is my favorite part of the Broad. It reminds me of the assembly line factory videos on Electric Company.

[artsresearch]

What I learned at breakfast…. 1) When arts are used as an interpretive layer, they can clarify, complicate or make things visible. –Rosten Woo 2) “The only way out is in. Once we take that risk, we will make it out…. Perhaps not unscathed, but certainly all the better for it.” –crystal am Nelson 3) “I grew up when we were embracing this new technology: the automobile.” –Aaron Paley

[artsresearch]

Greater Philadelphia Cultural Alliance issues a report comparing arts and culture in eleven cities nationwide. I’d give you the details, but I’m sworn to secrecy until Oct 26.

[artsresearch]

If you were doing research in the arts, which of these would you select as your top priorities?

[artsresearch]

“Arts, no less than transportation, health, housing and human services, are essential building blocks of community.” –Rip Rapson President and CEO, Kresge Foundation

[artsresearch]

Allow me to recommend Spring for Coffee, just outside the back door, near the registration desk

[bgm-ac-re]

Plenarying at GIA; looking for the unexpected

[bgm-ac-re]

What I learned at the #GIArts plenary: The big idea is welcome --Alison M. De LaCruz We look to the #arts to break borders --Yuval Sharon I want to be philanthropic in the broadest sense possible: I want to love people. --Alexandra Grant

[bgm-ac-re]

I just finished reading Us Conductors by Sean Michaels. Look for by review soon in TNBB Blog .

[penamerican]

Poetry

Now the poems seem to come one after the other.

That can only mean one thing:

Autumn has arrived.

“Poetry” is a poem from Cold Moons, a collection of sparse, minimalistic ecopoetry by Icelandic poet Magnús Sigurðsson that deals with nature’s resilience to the passage of time. You can read a larger excerpt of Cold Moons here. Meg Matich is the recipient of a 2015 PEN/Heim Translation Fund Grant for her translation of Cold Moons. You can read Matich’s essay on translating Cold Moons here.