I recently had to present about practice-based research to an audience of sociologists and social-scientists. Some of them liked it but some of them were keen to ask me about ‘rigour’ and how this method had ‘rigour’.
I found this annoying, partly because I was not entirely sure what they meant, and partly because I was not entirely sure that they knew what they meant. But also, of course, I felt defensive, because the ‘rigour’ question seems like sciencey talk designed to intimidate people doing thoughtful qualitative work.
I also thought, well practice-based research is as ‘rigorous’ as anything else, surely. It seems like a weird thing to go on about. I’m trying to achieve deep and rich understandings, in a careful way, applying a lot of thoughtful effort over a long period of time. That sounds like ‘rigour’. But I don’t know, and it doesn’t seem like a fun topic to get hung up on.
Nevertheless, it would be nice to have a solid answer – and one that doesn’t just sound grumpy and defensive.
So, good news, practice-based research people! I wrote a thing recently – it is, in fact, part of my new book Practice-Based Research and Research-Creation: An Introductory Guide – which sets out how practice-based research is absolutely a proper research method, falling squarely in the middle of the pantheon of available research methods, and demonstrating the qualities of originality, significance and rigour that everyone rightly expects to find in things that call themselves research.
Here it is:
A question worth answering
As relatively new and non-traditional ways of doing research within universities, practice-based research and research-creation may face the question ‘But is it really research?’. For those of us who have been involved in this field for some time, the question can be tiresome, and we don’t really want to entertain what can often be a sneering or boorish kind of challenge. Nevertheless, it is a question we may face, and also it is a question that we may ask ourselves. We also want to be able to provide a good answer to friends, or colleagues, or people who challenge us.
The good news is, there are really solid answers to this question. Practice-based research is research. Research-creation is research. Absolutely. I’m going to spend several pages setting out the good clear answer on this.
Practice-based research and research-creation are now so well established in many universities that you could become a student doing these kinds of things without really knowing what the real justifications are – which means that, instead, you may have a vague grateful feeling that the people in charge have let artists ‘get away with’ doing creative activities and calling them ‘research’. But that does a disservice to everybody, because practice-based research and research-creation really are proper research methods which actually fall slap-bang in the middle of very formal established ways of understanding what research is, and what researchers do. You’re not ‘getting away with’ anything! It’s a real thing.
So how is it research?
To feel comfortable with practice-based research and research-creation as research, we need to be able to explain – to ourselves, or to others – how this is ‘research’. A stereotype idea of research is that it involves someone going to a library, or on the internet, to find some established written-down facts about something. This is compounded by the way that people sometimes say they did their ‘research’ before they did a creative thing – so for instance, when someone explains that they read a lot of books about New York gangsters in the 1930s before they created a video game about New York gangsters in the 1930s. This separates out the research from the creativity, and reinforces the idea that research is done by reading some informative material, which is sitting out there in the world, waiting to be looked at.
However, that’s only one stereotype of research. There’s another popular image of the researcher, which is the making-and-doing activity of a scientist in their lab. It’s what every ‘scientist’ does in the movies – rushing frenetically around their laboratory filled with bubbling test tubes, whirring machines or fizzing rockets. Actual scientists often find this Hollywood version of their lives to be a bit cartoonish, but for our purposes it’s helpful. Are these kinds of scientists doing research? Definitely! We can see it with our eyes – actual research happening, as they pour more elements into their test tubes and whack the machines with a hammer. And if this is what we witness as ‘actual research happening’ – what is it that we see? We see that they are making and doing – putting things together – creating – to arrive at new understandings. And that’s exactly what practice-based research is.
You might say ‘Ah, but the scientist explores the properties of the physical universe in order to establish knowledge about this objective reality – that’s not what’s happening when someone makes a painting or a movie or a novel, which is just a reflection of subjective emotions, feelings and ideas’. This sounds like a valid point. But let’s continue breaking everything down. What does this mean, this less-real ‘subjective’ thing? The point being made is that unlike the existing scientifically-measurable world, the ‘subjective’ world of human experience is fuzzy and complicated, and differs between people. But does that mean it doesn’t exist, or cannot be explored? No. Your experiences may be complicated, multi-layered, and individual – but that doesn’t mean they don’t exist. You do have experiences and feelings. Practice-based research and research-creation isn’t always about one person’s experiences and feelings – but even if it was – that is a subject that can be explored, and understood and articulated, for the benefit of others, because understanding the nature of human experiences and feelings is a very fundamental and important thing to explore. It is at the root of several big disciplines, including philosophy, literature, art, psychology, and sociology.
So this seems to be a way that practice-based research can be like a thing that people already recognise as research. But if we are digging as far as we can go, we should see if practice-based research will fit under an actual, proper, established definition of research. Now, different definitions of a word like ‘research’ might appear in different places, with different contexts preferring to frame it in different ways. But I am going to use the definition of research which seems likely to be the most stress-tested universal definition of research that the world has ever known.
Defining research
So bear with me while we set out a definition of research, and clarify how it was arrived at. This book is called Practice-Based Research and Research-Creation, which means that the word ‘research’ appears, clunkily, twice in six words. This means it is really important to define ‘research’. People in particular fields can probably say what ‘research’ tends to look like in the mainstream middle of their field. But, to show comfortably how practice-based research and research-creation are forms of research, we want more of a recognized general standard. It would also be good to be clearer about where the edges are. What exactly ‘counts’ as research, in a university?
This was the question faced by the people running the Research Assessment Exercise (RAE) – a bureaucratic process established in the UK when the government in the 1980s felt that it should have some kind of measure of how much research was happening in the universities, and if it was any good. That’s the kind of seemingly straightforward question that a government minister might ask. But of course, you can’t begin to properly answer it, unless you have a system, a way of measuring and rating this ‘research’. And it’s definitely impossible to do that if you can’t even say what is ‘research’ and what is not.
But, academics may cry – and they surely did – there are so many different fields! The things that a researcher in Afghan folklore does are quite different to the things that a researcher in quantum theories of physics does, and what that physics person does is a lot different to what a bridge engineering researcher does, even though both of those things are basically physics. These researchers engage in different modes of exploration; they present their findings in different forms; and each field has different expectations about what is important and how you go about it. If you focus on these details, it doesn’t sound like something that could be defined at all. But the RAE people managed something brilliant. For UK academics, to say that the founders of the Research Assessment Exercise did something good is absolutely wild, because everybody knows the research assessment process as a much-hated millstone around their necks, not least of all because the bureaucracy has ballooned over three or four decades to be terrifying and all-consuming. People hated the RAE so much that the government had to rename it in 2014 – which wasn’t exactly the solution its detractors had in mind – so now the research assessment exercise is called the Research Excellence Framework (REF), which suggests that these people had lost the ability to use words in a meaningful manner. But in their original definition of ‘research’, at least, they did a great job.
They stated that research is:
a process of investigation, leading to new insights, that are effectively shared.
Amazing! It’s simple, it’s memorable, and it sets out three important elements. It doesn’t matter if you are doing history, mathematics, computer science, zoology or ceramics, you can look at the ‘research’ stuff in your field and see if it involved a process of investigation; whether it led to new insights; and if it was effectively shared.
The incredible clarity, simplicity and precision of this cannot be overstated. You will notice that although it is clear, it doesn’t prescribe anything. You do need to have engaged in ‘a process of investigation’ – but they’re not going to dictate what that would look like. They want you to develop some ‘new insights’, appropriately enough, but it’s up to people in that area to decide if this has happened. And, perhaps most crucially for us, these insights should be ‘effectively shared’, which is as firm and memorable a requirement as you could have, but they don’t say anything about how or where this should happen.
This universal definition of research can be used, word for word, as a very good definition of practice-based research and research-creation. What is practice-based research? What is research-creation? Well: it’s ‘a process of investigation, leading to new insights, that are effectively shared’.
One of the several reasons why this is so delightful is that, while practice-based researchers tend to assume that they are doing quite a weird thing, far from the mainstream of research, in fact their research method falls bang in the middle of the most officious definition of research, and a definition that is very widely accepted, not only in the UK but internationally as a kind of gold standard.
Research quality
We have used the definition of research from the UK’s RAE and REF – I’m just going to call it ‘REF’ here now – to consider whether practice-based research or research-creation would even count as research, and the answer is a definite yes. That is great news in itself – very reassuring for creative researchers anxious that their work is ‘not really research’ and would be dismissed out of hand. But that’s just the bar for entry into the contest. While we’re here, we should also look at how the REF determines if the research is any good. (This is not because we necessarily assume that the REF is going to offer an exemplary way of measuring the quality of research, by the way. I’m instinctively very wary of all such forms of quantitative assessment, and I would never myself use it as ‘measurement’, as if we are calculating an objective reality, like measuring a wardrobe in IKEA. Coming to this cold, I would also assume that whatever the REF assessment people have cooked up, it’s likely to be produced by committees dominated by scientists and engineers, and be useless for the arts. But let’s put these biases aside and see how they did).
Their statement about how the REF will ‘assess the quality of submitted research outputs’ again appears remarkably simple. They say that all research, regardless of what discipline it is in, or what form it takes, should be evaluated in terms of three things: ‘originality’, ‘significance’ and ‘rigour’. That sounds straightforward. Of course, if you think about it for more than one second, there are huge problems about who would assess these things, and the ways in which academics police their disciplines and suppress outsiders and oppose innovation (which are discussed in more detail below when we look at ‘peer review’). But as things to aim for – as you understand them, and as you might imagine your colleagues would understand them – they are not bad.
Originality
Here are the REF guidelines on ‘originality’:
Originality will be understood as the extent to which the output makes an important and innovative contribution to understanding and knowledge in the field. Research outputs that demonstrate originality may do one or more of the following: produce and interpret new empirical findings or new material; engage with new and/or complex problems; develop innovative research methods, methodologies and analytical techniques; show imaginative and creative scope; provide new arguments and/or new forms of expression, formal innovations, interpretations and/or insights; collect and engage with novel types of data; and/or advance theory or the analysis of doctrine, policy or practice, and new forms of expression [quoted from here].
I’ve quoted this in full because it’s deeply reassuring. If you’re not from the UK it’s probably hard to understand the weight of this – and my apparent obsession with one country’s bureaucratic assessment procedures may seem bizarre and irrelevant. But I know that everybody doing a PhD, or other research, worries to some extent about the ‘originality’ thing. Even when we are confident that we are doing fresh new work that nobody has done before, we might wonder if this is a ‘proper’ success in terms of ‘originality’, or maybe we are just fudging it – who knows?! Because who knows how words like ‘originality’ would be defined in a really proper way that would apply across all the disciplines and be accepted across multiple kinds of universities? Except, here that is exactly what they have done. And I know that the REF involved – and still involves – absolutely masses of effort and argument to arrive at definitions which carry an absolutely existential importance for every UK university. These are not just definitions like someone might write in a textbook. Variations in these definitions could lead to institutions receiving millions of pounds more or less in funding, depending on what they allow and what they value. Whole fields, and whole universities, could rise or fall, as a direct consequence of the words and phrases used in these definitions.
So this statement of what ‘originality’ means, and how we might recognize it when we see it, is highly consequential. Look at it again. It doesn’t merely say ‘something that has not been done before’, probably because it’s quite easy to do something rather random and bizarre which has not occurred before. This definition sets the bar much higher, seeking ‘an important and innovative contribution to understanding and knowledge in the field’. Whether we would count something as ‘important’ or ‘innovative’ is, of course, highly subjective – there’s no getting away from human judgement. But, to be fair, you can test it, by seeing how you would complete the sentences ‘This work is important because …’ and ‘This work is innovative because …’. There’s no entirely ‘objective’ way of deciding if your answers are or are not sufficient. But if you feel you can complete those sentences in a way that seems solid and satisfactory – that means a lot. The definition goes to quite some lengths to help you out, indicating assorted dimensions in which this important and innovative contribution might be happening: you might have new findings, or methods, or perspectives, or angles; and it even says ‘new forms of expression’ twice, surprisingly, because you could either be creating ‘new forms of expression’ or developing theory or analysis about ‘new forms of expression’. In short, there’s lots of ways you can be original.
Significance
To help us work out what this all adds up to – whether, in short, your work is a waste of time – the RAE/REF offers this on ‘significance’:
Significance will be understood as the extent to which the work has influenced, or has the capacity to influence, knowledge and scholarly thought, or the development and understanding of policy and/or practice.
So ‘significance’ is about the impact that your work may have on other people. (The definition doesn’t say the impact has to be on ‘other people’ as such, going for the even more diffuse ‘thought’, ‘policy’ and ‘practice’. But fundamentally, because it would be other people who have the thoughts, policies and practices, that’s what it means. And obviously it doesn’t mean everybody – just somebody). And the phrase ‘has influenced, or has the capacity to influence’ means that the impact doesn’t need to have happened yet. But we are looking to see if your work has some potential to be helpful or useful or instructive for others. That’s a reasonable thing to ask, and helpful for us to think about.
It’s also an area you can do something about. There’s a somewhat detached view of research impact – held in particular by people who don’t like to think about research impact – which is that it will either happen or it won’t, it’s to do with luck and serendipity, and there’s not much you can do about it. This is passive, which is fine, but it’s also false, which is unhelpful when this view is communicated to others. Researchers can definitely do things that push their work in the direction of becoming significant to others, and although this does not, of course, guarantee that their work will become impactful, it is more-or-less a necessary step to avoid being ignored and neglected. The things you can do include: (1) putting stuff out in the world – that’s the first step, but everybody [who is ever going to count] does this step, and it is not enough; (2) doing a whole additional layer of activity to draw people’s attention to the stuff – to show them that it exists and that it may have some interest to them – via social media, newsletters or email lists, and speaking at conferences or events, if you are able; (3) more targeted things such as sending emails to particular individuals who may be interested or who it might be good to connect with. The main point is that after publishing or sharing a thing, you need to put a similar amount of effort into pointing to the thing and alerting people to its delightful existence.
Rigour
Finally, along with ‘originality’ and ‘significance’, there is ‘rigour’:
Rigour will be understood as the extent to which the work demonstrates intellectual coherence and integrity, and adopts robust and appropriate concepts, analyses, sources, theories and/or methodologies.
Again, this is a reasonable thing to ask for, and like the outlines of ‘originality’ and ‘significance’, it actually helps practice-based researchers see how they can present their work as research. Practice-based researchers have typically spent absolutely loads of time immersed in the worlds of their work – the materials and techniques of making, as well as other works in this mode or genre, and the theory and analysis surrounding this kind of thing. So ‘rigour’ is very achievable. Sometimes creative practitioners talk cheerfully but self-consciously about concepts like ‘rigour’ as though this kind of concentrated methodical approach is just for the scientists, and nothing to do with us. But that is not the case. We should not sell ourselves short. Sometimes artists are impulsive and discover things through intuition and luck. But sometimes scientists are impulsive and discover things through intuition and luck. More often, both artists and scientists are advancing carefully and thoughtfully, building on their research and experience, and previous practices and discoveries. Every successful creative work you can think of is the product of the learning, extensive thought and applied imagination that we call rigour.
By this point, I’ve made ‘originality’ seem easier than you might think, and talked about how ‘significance’ is attainable. It’s worth noting that ‘rigour’ is where you can do especially well or especially badly. It’s a matter of putting in the time, and making sure that you have looked at what you are doing from different angles – including the angles that you would not ‘naturally’ adopt. In the academic context, it is especially refreshing and exciting when a practitioner has connected their work with unexpected ideas and principles from unusual or adjacent fields. We also want to see that they have reflected on their privileges and assumptions. When a privileged person seems to think that they are acting with swift ‘intuition’ and is not especially interested in situating their work in a critical and theoretical context, the rigour is lacking when we assess it as research – but also, this lack of rigour will probably mean it is less rewarding as a creative work. So bringing in the rigour is important, and takes time, and should not be neglected.
Of course, this is the same for everybody. The need for ‘rigour’ is no easier or harder for creative practice-based researchers – and no more or less time-consuming – than it is for historians or botanists or political economists. Practice-based researchers often feel a niggling anxiety, even if we believe we shouldn’t, about the status of our work and whether it ‘really is’ research, or whether the formal serious authorities of university research can be expected to actually respect us. And that’s why I feel so touched that the UK REF definitions of ‘research’, and its component parts of ‘originality’, ‘significance’ and ‘rigour’ – which I take to represent a tough international standard of what research means across all disciplines – are all very fair. And they don’t tell you what to do. They just set some standards about the level of intensity and thoughtfulness that you should bring to your work; and they emphasize the need to communicate the work and connect it with others. I’ve met practice-based researchers who seem to think that these kinds of expectations are easy for science people and hard on creative people. And I wouldn’t like any ‘standards’ that preferred particular practices, ideologies, or modes of presentation – but these outlines are completely non-prescriptive, except in the sense that they want you to push your work in terms of being new, critical, interesting and making a difference. Which is all we want, as artists, anyway. The expectations set by these definitions are just as challenging or difficult for scientists and engineers as they are for artists and poets. Which is fine – everyone is going to have to work hard, to do our best. What I find rather lovely is that we are all being held to the same standard.
Leave a Reply