Evaluation — Actors, Values, and Metrics
An examination of modes, values, methods, and metrics of evaluation with an aim to develop more organic, bottom-up led evaluation practices in arts and culture.
This journey has been nested in a pressing moment for our World, where the overlapping humanitarian, social, and ecological crises accelerated to a point of a global disruption caused by a pandemic of proportions unseen for generations. Life as we know it is changing before our eyes, and we are again reminded of Marx’s famous words: ‘All that is solid melts into air’, and so is the cultural and artistic life that we only recently knew and lived as practitioners. And this most recent in the stream of crises that we’ve seen in the course of the last decade or so, serves as another reminder about how urgent is the need for pivoting essential and radical changes of ways we organise life on Earth in order to provide for survival of Earth’s ecosystems, including sustainability of communities, both human and beyond. An important part of that change must involve artistic and cultural production in its core – we must urgently rethink how cultural and artistic production is being created, curated, mediated, accessed. And equally important: how cultural and artistic production is being assessed and measured, based on what values, by whom, and using which metrics?
As a community of communities scattered all over Europe and the Middle East and North Africa (MENA) region, RESHAPE has been a perfect playground to gather and examine a variety of experiences of cultural actors in relation to the evaluation processes and their different aspects. It has also given us an opportunity for developing a much bigger and much more nuanced picture than would have been possible without RESHAPE. Besides that, working together provided a lot of comfort and solidarity in times of isolation and uncertainty. RESHAPE has proven to be a reflection and action-oriented process where we could gather to critically assess the current state of the art in the artistic and cultural field, as well as imagine and create blueprints for a different, happier common future.
Early on in the process we detected the need to shake the tree of evaluation, using this metaphorical language not only to demonstrate the need to reframe its basic notions, but also to be able to distinguish its various fruits, be they ripe or green, sometimes already sagged and rotten. Our attempt to examine different modes, values, methods, and metrics of evaluation stems from the desire to contribute to the development of organic, bottom-up led evaluation practices in arts and culture. We want our work to result in an exercise for a potential model of evaluation that can be modified and adapted by organisations within RESHAPE but also outside of its immediate reach. Ideally, the model could be tested on RESHAPE itself in the aftermath of activities’ realisation.
Methodology and approach
Avoiding (but not completely eschewing) the usual survey methodologies, we reached out to the tools immanent to the artistic type of research: collective work based on observation, (many) conversations and meditations on the subject, gathered by examining a body of work of practical and academic research about evaluation and the related fields.
One of the important goals for us was to re-examine, and contribute to the transformation of the role of expertise and experts – shifting the evaluation towards processes of learning and knowledge transfer in order to make them empowering for organisations and individuals around them – strengthening their ownership over evaluation processes, instead of facilitating utilitarian and mercantile logics, more often than not imposed in a top-down manner. Equally important for us was to learn about existing tools of evaluation that organisations all around Europe, in the MENA region, and globally are using in their everyday work, to experience what are the organic practices of evaluation developed by practitioners themselves and to gain deeper insight into how knowledge produced in evaluation processes is being used and reproduced. Finally, an important part of this endeavour was the necessity to obtain insight into different cultural landscapes, all parts of a wider cultural ecosystem within RESHAPE’s horizon, in order to develop a deeper understanding of problems related to evaluation practices and the ways to overcome them.
Interviews & questionnaire
Through a series of interviews with practitioners active in different countries of Europe and the MENA region we wanted to accomplish two main objectives: learning about participants’ practices and attitudes with regard to evaluation processes, and learning about the context in which participants operate, including the nuances about the functioning of the cultural ecosystems in different geographies. Interviews were conducted as a semi-structured type of survey with open-ended questions. Practitioners included members of our Trajectory group Fair Governance Models as well as our colleagues operating in different contexts and in different capacities. The questionnaire for the interviews was jointly developed by the team and involved inquiring about: basic organisational structure, organisational attitude and motivation, methodologies and metrics that organisations use, how is evaluation being used/implemented, how is knowledge gained in evaluation processes communicated. As a complementary tool to the interviews a short questionnaire was designed that focused on a small number of key questions tackling values that inform the evaluation processes, usage of knowledge produced by the evaluation, and how participants see the future of evaluation processes. The questionnaire was designed with the primary purpose to reach the RESHAPE community and to scan the prevalent themes and undercurrents related to evaluation.
Learning resources and inspiration
Different resources have been used throughout this journey, as an inspiration. First and foremost, there were the numerous group and personal encounters with fellow Reshapers as well as other practitioners in our field, during our physical meetings in Lublin, Tangier, Cluj, Sofia, as well as remote meetings in Zagreb and Athens. These exchanges not only enabled us to gain insight into the various interpretations of evaluation processes, existing practices and overarching questions, but have also made us aware of the fact that evaluation persists as one of the key questions in the context of cultural governance, especially with regard to pressing needs to develop fair models of governance based on solidarity and redistribution of power and resources. One of the guiding examples that has greatly inspired our team was Bhutan’s Gross National Happiness index, as a philosophy structured around governance based on collective happiness/wellbeing and preservation of life on Earth.
Another great inspiration for our work was provided by the work of Brazilian futurologist Lala Deheinzelin, who developed 4d Fluxonomy, a governance concept founded on the need to elucidate complex connections between cultural, environmental, social, and financial aspects while developing a sustainable future through the application of new economic models.
In addition, the practical application of Fluxonomy principles into an evaluation tool prepared by FARO, a research group of 14 various Ibero-American cultural organisations gave us an incentive to start developing a version of the tool that was suitable for application in the framework of RESHAPE and could be adapted for communities beyond RESHAPE’s scope. Working on a development of a nuanced and intricate evaluation matrix enables these actors to envisage a scenario for a transition period, notwithstanding the speed and magnitude of changes that we are facing in the contemporary world.
The connection between evaluation and governance?
Evaluation is a crucial part of survival of cultural ecosystems, as it enables the creation and systematisation of knowledge and skills necessary to perpetuate practices that are crucial for institutional/organisational existence as well as enabling us to become conscious of practices that are detrimental for organisations and find ways how to unlearn them. On the other hand, evaluations are a part of a larger complex of scrutiny and control mechanisms (together with audits, quality assessments, and so on), applied externally and from above in order to justify the existence of these external actors, especially funding bodies, government agencies, private foundations, and various public and private investors in arts and culture.
These two processes are inevitably connected, albeit very different. A number of practitioners we have been talking to made a distinction between evaluation that organisations initiate themselves as means to build and share knowledge, seeing these processes often as tacit and organic and leading to the organisational empowerment and greater ownership of individuals over activities and crucial processes within the organisation. Opposed to that, many of the individuals we have been talking to marked the evaluation processes organised and led by the funding bodies as a form of control, as highly formalised processes based on pseudo-quantities (Habib – Enquist and Möntmann, 2018).
When we talk about fair governance it’s inevitable to think about the values that inform what fair governance actually is about (or, rather, should be) and about systems set in place designed to measure how exactly values are being put into practice. No less important are questions of who sets the rules of the evaluation as well as who has a say and who participates in which capacity. Evaluation is a complex endeavour, burdened with internal conflicts and contradictions between motives, subjects, and objects who perform the evaluation/the evaluation is performed on, tools, methodologies and metrics used to perform the evaluation, frameworks, narratives, and language in which evaluations are being interpreted. In that sense, it is much more productive to think about multiple evaluations, instead of a single all-encompassing term.
Perhaps one framework that bridges all the inner contradictions of evaluation is the one of establishing sovereignty over interpretation and valuation (Zembylas, 2019). The sovereignty over interpretation, or the right to establish the narratives determining one’s work make up the core ingredient with regard to governance of institutions and organisations regardless of their size, mission, and formal structure.
Formal language and requirements of evaluation processes led by external factors have permeated the organisational structures, often leading towards institutional isomorphism (DiMaggio & Powell, 1983), that is, increased similarity and homogeneity of actors functioning in the same field. Furthermore, by reducing evaluations to their technical aspects while avoiding the tricky questions pertaining to ownership and direction, evaluations can often become a legitimation tool of no-change. Many organisations, also some of the ones we had the opportunity to talk to, actively resist and challenge these practices.
Organisations will often challenge the official narratives of the evaluation by developing alternative routes and methodologies, leaning on affective and community‑oriented models of evaluation. All the colleagues we have been talking to, no matter where they operated, stressed human contact, conversation, and orientation towards the community as key ingredients of bottom-up driven evaluation. At the same time, all the colleagues stress the necessity to dedicate more time and resources in developing different evaluation methodologies in order to gain deeper knowledge about intricate connections between what we do, how we do it and how it corresponds with the context in which we operate. We hope that our work and this report as one of its essential parts make a small contribution towards greater understanding in that sense.
Context of the research
The first part of our research contextualises matters related to evaluation through general description of circumstances in which interviewees operate, drawing the lines of contrast and similarities between the different cultural ecosystems in which their practices are nested. In the process of development of this research, we interviewed our close collaborators, colleagues engaged in the RESHAPE process as well as colleagues with whom we engaged outside of the project’s scope. In total, we conducted eight in-depth interviews with 14 cultural workers operating in Turkey, Croatia, Serbia, Palestine, Bosnia and Herzegovina, UK (Scotland), Switzerland, and Belgium.
The majority of our interviewees were individuals engaged in non-institutional cultural production, that is, independent cultural actors from various countries in Europe and the MENA region. Most interviewees’ work is nested within a collective/organisation, although some of them are active as freelancers. These organisations vary in terms of formal frameworks, disciplines, size, and scope of their activities. They are active across multidisciplinary forms of artistic expression (performing arts, dance, theatre, visual arts, media, and so on) as well as in diverse critically engaged cultural practices. Organisations we tackled also come in various shapes and sizes, varying from small artistic initiatives to veteran institutions, informal collectives to long-established organisations, organisations focusing their work around one or several annual events/projects to organisations engaged in multi-stakeholder structures with complex programmatic dynamics. Despite great differences between organisations, there are many similarities, mainly with regard to values and basic programmatic orientation: a strong inclination towards socially engaged arts/culture, working closely with communities, critical attitudes towards current social and political context, and active engagement with various actors in terms of fostering positive changes.
A diversity of geographical contexts, together with dependent social and political circumstances has marked our endeavour and represents the great value of this research. Our interviewees came from eight different countries and territories: Turkey, Croatia, Serbia, Palestine, Bosnia and Herzegovina, UK (Scotland), Switzerland, and Belgium. Besides being embedded in their local and national contexts, all of the actors that we’ve been talking to are actively engaged in international collaboration. Both bigger and smaller urban as well as rural areas are relevant for our interviewees’ work. Even though national cultural policies as well as international funders greatly shape the organisational field, our interviewees all emphasised the importance of being embedded in a local context, such as the cities, towns and various other local communities, the ones that hold the potential to bridge the gap between complex workings of international bodies and national policy-making institutions that often fail to recognise the needs and the importance of artists and cultural workers in social development:
Independent evaluator, Palestine: ‘I find it easier to have dialogues with city governments so I am a strong advocate for city development in the [MENA] region. Because cities tend to have more elections and less political appointments. I like to work with cities and I have considered them to be more receptive than the Ministry of culture.’
There are great and well-known disparities between the geographical and socio-political contexts in which our interviewees operate, including ones that pertain to institutional and financial sustainability of organisations as well as disparities that are determined by the differences in the organisation of cultural spheres and the overall objectives of cultural policies (and politics in a wider sense). As much as positions of cultural operators in Western European countries differ from those in Turkey, Palestine, or Bosnia and Herzegovina (especially in terms of available resources), our interviewees have much in common in terms of the conditions in which they operate. First of all, they face similar pressures of top-down imposed priorities when it comes to the conditions of production, both in terms of content and in relation to delivery to audiences. Regardless of a specific geographical locale, all of our interviewees have testified having difficulties constantly adjusting to demands of decision-making and/or funding institutions, where tools for evaluation are often seen as a negative type of instrumentalisation of independent cultural actors in a larger scheme of things. In Western European, so-called ‘developed’ countries, these pressures are often expressed through demands for structural rationalisation and adoption of a corporate organisational culture. Up to a certain level, similar tendencies are present in Eastern Europe and the MENA region, with one notable difference. Namely, while in Western Europe pressure often comes from governmental bodies and/or agencies connected to governments, in the MENA region and Eastern Europe the pressure often stems from diversified sources (government and private funding bodies) as the determining need is to obtain finance and other resources necessary for production, which are scarce. This often leads to a situation that can be crudely described as follows: While in Western countries pressures are put primarily on organisational structuring and management and governance of organisations, in the East the pressure is more emphasised through programming priorities that often determine the content of organisations’ work.
In the East, collectives rely heavily on personal enthusiasm of the involved cultural actors, often without formal professionalisation of roles and relations. While in the Western countries, organisations are being coerced into a hyper-structuralised ecosystem that lack potential for experimentation and development of synergistic momentum outside of siloed views on artistic and cultural production:
Cultural operator, Serbia: ‘It’s also that the small organisations here, everything between civil society organisations, small start-ups ... Some things that function as a rock band or whatever, and also a lot of personal, almost family relations. It’s very difficult to formalise. It’s not a professional thing. It doesn’t have professional structure, even though it has the results that are high quality. And organisations like that produce lots of good cultural content in Serbia. It does not apply any kind of professional logic. It’s difficult because they’re not organised. They’re like some friends decided to do something together. And then you have everything in between your relationship.’
Cultural operator, Belgium: ‘We have top-down management being implemented more and more, and a sort of rationalisation happening in this very fragmented art field. If anything, the Flemish art scene is a very hybrid one, with super small organisations, many institutions collaborating in many diverse ways and the government doesn’t really like this uncontrollable situation – big spaghetti that we work in. So, they’re now implementing categorisation and with this categorisation come new rules. ... So we are facing a big rationalisation and categorisation of the field in very strict categories and with these categories come evaluation points which are different but it also creates a total immobility, because if you’re classified as a certain type of organisation, you should do a certain work and you cannot reinvent or redefine yourself or maybe have a different mobility inside of this, because you would not meet the evaluation point.’
Maybe the biggest point of contradiction between positions of actors from Western Europe and the Eastern countries became visible during an interview with a colleague who operates in Turkey. While colleagues in the West struggle with excessive structuring and pressure to adopt business practices, cultural actors in Turkey avoid formal registration in order to escape being pressured by the current political and administrative regime:
‘We use the benefits of being unregistered. We can do anything political, about the government. No one knows about it, and on paper we don’t exist. It’s a plus to defend these rights.’
From these insights it is clear that all organisations we were in touch with need to navigate a set of complex, often oblique rules within their respective contexts, and these circumstances have a major impact on the development of organisational narratives and practices of evaluation. One thing all actors included in this brief research had in common was searching for places and platforms that would enable more freedom from excessive coercion, places for experimentation, failure, and reflection. The next two parts of this research will look into more details about existing evaluation practices as well as new tendencies that organisations are developing in this regard.
New Metrics and Values Evaluation Website Prototype Proposal
A subgroup of six people was set up inside the Fair Governance Trajectory. They performed the research mentioned earlier, also connected with the work being developed by FARO, with the objective of proposing an interactive output to its current quest for a system that can be applied to evaluate projects and institutions in the socio-cultural sector, such as the ones participating in the RESHAPE context.
Culture has values that go far beyond numbers as it has the ability to transform societies, improve people’s lives and activate the global and local economy. In this sense, it is necessary to develop new metrics that allow us to evaluate all this and value the wealth of the intangible that will be the basis of the economy of the future.
FARO is a learning community that joins forces for a broader, more effective, and innovative action in the sociocultural field, formed by professionals of twelve organisationsBAC Biennale of Arts of the Body, Image and Movement, Madrid, Spain // Consortium of Museums Comunitat Valenciana, Valencia, Spain // Feboasoma, Buenos Aires, Argentina // Graner Artistic Residences Center, Barcelona, Spain // Invisible Pedagogies, Madrid, Spain // mARTadero project, Cochabamba, Bolivia // NAVE Artistic Residences Center, Santiago, Chile // LABEA – Art and Ecology Laboratory, Pamplona, Spain // Salmon Festival, Barcelona, Spain // Teatre L’artesá, El Prat de Llobregat, Barcelona, Spain // Teatro de la Abadia, Madrid, Spain // Uniflux, Sao Paulo, Brazil. from Spain, Brazil, Bolivia, and Chile who have been researching and discussing new values for the past two years. FARO’s challenge is to adapt their practices to an Ecosocial Transition scenario where linearity of people, material, resources, and time can be revised into the exponentiality of the nets, where a Culture of the tangible can become a Culture of the intangible, where Egocentrism gives way to Ecocentrism. They aim to create tools, systems, and methodologies using collaboration and new technologies for converging teams, talents, partners, resources, data, and time.
The Evaluation Subgroup started to study the theoretical base of FARO’s actions, the 4D Fluxonomy, which combines Futuring and New Economies, created by Lala DeheinzelinVideos by Lala Deheinzelin explaining fluxonomy here and here. (Brazil). A bridge was established through Eduardo Bonito (based in Spain) who participated in both projects. Fluxonomy works with four dimensions: cultural, environmental, social and financial, based on the four types of economy: creative, shared, collaborative and multi-value. The idea of the collaboration is to create a common metric system for measuring results highlighting the intangible wealth the projects and organisations of the culture sector have and create.
The collaboration enabled the Fair Governance Trajectory to do an exercise of a proposal for an evaluation website form based on 16 values/criteria, which have been constantly discussed within FARO over the past year and were put into an evaluation form website map with the support of the Evaluation Subgroup. The attached table of 16 questions for 4D-evaluation is the current version of a strong work of collective and progressive approach to this type of metrics and it reflects each word and each concept. The table has been translated into English by RESHAPE and reflects the current state of the research, with the understanding that these values are in constant evolution as they are tested with different projects, discussed and revised by the FARO members in a process which is expected to take many months of trial and error, tests and redefinitions before it could be released, as our Subgroup suggests, in a website format.
For us it has been a very important process to follow the development of the definitions of these values, discuss them internally using our own projects as reference and giving feedback to the FARO members for concept review. It is quite pertinent to point out that we have been looking at realities of smaller, peripheric as well as more established projects within the RESHAPE area of reach and exchanging our impressions with FARO members, contributing to the development of the values. This reflection process has been very enriching for us, and the prospect that it can inform a research that will generate tools for evaluation is already in itself quite satisfying for our Subgroup.
Fluxonomy values have been adapted to the socio-cultural sector, maintaining the theory’s fractal vision of reality based on its ‘zooming’ in on four dimensions: cultural, environmental, social, and financial, which are in turn divided into four, generating a chain of meaning that facilitates a more holistic approach to reality. For instance, the cultural dimension of a project that in turn includes a cultural dimension of the cultural, an environmental dimension of the cultural, a social dimension of the cultural and a credit dimension of the cultural.
By answering four questions about each dimension, we will be able to evaluate a project or organisation on four levels:
Cultural: The reason for being (transmission – relevance): How Convergent the organisation or project’s Idea is, how Revealing its Language is, how much Capacity to Affect its Interaction has, how much Reciprocal the Learning involved in it is.
Environmental: What structures it (transformation viability): How Transforming the organisation or project’s Knowledge is, how Sufficient its Infrastructure is, how Evolutionary its Regulatory Body is, how Interdependent its Multi-Capital financial resources are.
Social: The ability to do together (interdependness – scope): How activating the organisation or project’s Proposal is, how Conscientious and Translocal its Organisation is, how Co-Evolutionary its Governance is and How Influential its Credibility is.
Financial: What generates reproducibility (impact – exponentiality): How Revitalising the organisation’s or project’s Thoughts are, how Deconcentrating its Distribution is, how much Multiplier the Circulation it promotes is, and how much Regenerative Flow its Economy promotes.
As one can see in the table, each of these 16 aspects mentioned above are informed by five forces that support the questions formulated.
The Fair Governance quest was an exercise to propose a map of a website that may support the evaluation of any kind of organisation or project and prepare it to be implemented and tested. It will contain a fixed picture of the definitions as they were in September 2020, acknowledging that the definitions of values and metrics by FARO are still in constant redefinition and fine-tuning, thus suggesting a structure that can be easily updated at any point.
The website structure presented allows users to set up parameters for new projects, get multiple answers from the project’s users and receive automatic numeric evaluation analysis as well as a list of all the answers divided into each of the 16 values. A summary analysis is produced by the project’s evaluator with the information provided.
Each project’s or organisation’s member can register as a user and answer as many of the 16 questions as they can. The website form consists of one home page which directs to four dimension menus where users will encounter four values with four questions to be answered. There they can read and listen to definitions of each value, and give a numeric value of how advanced the project is on each aspect according to their perception. They also answer each question in writing.
Once users have completed their answers, their numeric perception levels will be automatically updated in the home page, visible to all users who have finished answering. These users will also be able to see all answers listed on each section. Once all users have finished, the evaluator(s) can read all the answers and produce final synthetic answers to illustrate the numeric average perception produced automatically.
After all the analyses are produced the evaluation can be available to be seen by all users, or by guests with a code, or by the general public, depending on settings previously defined by the evaluator.
The Evaluation Subgroup understands that the website tool can be very useful if put in place, but it also recognises that tests and reviews should be run before it is offered as a public evaluation tool to any project or organisation. As a result of this collaboration process, many of the Subgroup members were invited to continue to reflect together with FARO on these issues so they may act as invited task force consultants in future projects, for instance at FARO’s residence in November 2020 at the Cadiz Ibero-American Theatre Festival.
Our experience with researching on evaluation values and metrics has been a journey into a kaleidoscope of needs and views, giving us the certainty that the issue is very complex and diverse, and reflecting the immense possibilities of governance practices observed in the RESHAPE area of action and beyond. The process has enriched our perspectives and we hope that our reflections and suggestions described in this text, attached table, and website map may contribute to the development of practices more connected to the reality and needs of cultural projects being developed nowadays.
Developed in the framework of the RESHAPE trajectory Fair Governance Models including Helga Baert, Eduardo Bonito, Virdžinija Đeković Miketić, Fatin Farhat, Katarina Pavić, Ilija Pujić, Martin Schick, Sam Trotman and Claire Malika Zerhouni.
This text is licensed under the Creative Commons license Attribution-NonCommercial-ShareAlike 4.0 International.
Habib Enquist, J. and Möntmann N. (2018). Agencies of Art – A report on the situation of small and medium-sized art centers in Denmark, Norway and Sweden. OK BOOK, Oslo.
Zembylas, T. (2019). ‘Why Are Evaluations in the Field of Cultural Policy (Almost Always) Contested?’ In: Major Problems, Frictions, and Challenges in Arts and Cultural Management, Sense and Sensibilities in the State of the Field. Routledge, New York and Abingdon. Pp. 151–173.