How to evaluate qualitative research

People on my qualitative methods training courses often ask me how they can demonstrate the value of qualitative research when it appears to be so subjective or anecdotal, small scale or not generalisable. Well I have many and varied answers to this, but in this blog post I will focus on just a few. I have a feeling I will return to this at a later date. 

I want to start by saying it makes much more sense to evaluate qualitative research based on what it set out to achieve rather than criteria imported from another approach. So, if you are pondering this yourself too, then I would go back to the beginning and ask yourself ‘why did I choose qualitative methods? What was I hoping to achieve?’ Or ‘why did my client ask us to do qualitative research? What did they hope for as an outcome?’ When working with external clients it is always a good idea to start by clarifying their goals, so that if they ask if your results are statistically representative or objective you can remind them of your goals at the outset, and the true value of qualitative research. 

I also want to emphasis, right here, right now, that qualitative research is useful, practical, and able to inform actions.But how did I get to that point? 

In qualitative research surprise and relevance go hand in hand

Qualitative research has the potential to reveal deep insights that come to researchers as a bit of a surprise, or a bit left field. Often, these insights show that the research designers were asking the wrong questions in the first place, or had based their initial questions on some misunderstandings. If we really listen to people, and hear them, allowing our questions and preconceptions to be challenged, we not only understand them better but work together towards better, more relevant, solutions.

Here is one example. Anand Menon and Matthew Bevington in collaboration with the Joseph Rowntree Foundation conducted a few workshops (we might call them focus groups) with low income voters in order to tap into what Brexit means to people three years after the referendum. They unsurprisingly found a frustration with politics and politicians, feelings of being let down, of anger and betrayal. But as they continued to talk to people, they found that what unites them, what they really want to talk about passionately, is their local areas, local needs, and even some fairly straight-forward local solutions. Menon and Bevington argue “So while Westminster contorts over extra runways at Heathrow and high-speed train lines to London, it risks missing the bigger picture: many, perhaps most, poorer areas lack even the simple provisions that would allow them to succeed….Brexit has confirmed low-income voters’ worst fears about politicians. They see a litany of issues that need addressing in their local areas, and yet they sense there are no resources or political will to do anything about them.” 

These are findings that are exciting, interesting, meaningful, rich, and contextual, and what is more, they are findings that, if there is a will, we could actually do something with! This is because the researchers listened, heard, and therefore achieved relevant understandings of complex situations in people’s daily lives. 

Qualitative research is rich and contextual,enabling understandings to be acted on in a way that would be useful. The point of qualitative research is not merely to identify relationships between actions, behaviours or perceptions, as in a survey, but to look more deeply into how things relate in what ways over time and under what conditions. this leads to the use of words such as nuance, richness, depth, detail.

Qualitative research is sensitive

It is often argued that qualitative research is useful for sensitive topics,or issues that are difficult to talk about. Why is this of value? 

Let’s look at another example. Kate Reed, Elspeth Whitby and Julie Ellis used ethnographic methods to explore practices around infant post-mortems. They followed hospital staff as they went about their work, and interviewed staff and parents, and uncovered all sorts of caring acts such as bathing, dressing and talking to babies that meant the whole process could be far more meaningful and gentle than might be imagined. Their research led to changes in professional training and information sheets for parents that led quickly to increased take-up of minimally-invasive post-mortems in Sheffield, where the research took place. 

The research is meaningful, insightful, and revealing of the things people sometimes do without even realising they do it. It touches on issues of love and care, and is sensitive to emotions. But these alone are not of value. The added value comes in at the point of practical and useful applications of these insights; applications (or implementations) that have a chance of working because, first, the qualitative researchers took their time to listen, to hear, and to observe

Qualitative research is responsive

To take the ideas above just a little further, qualitative researchers learn as we go and we develop our ideas as people talk to us, as we observe practices, and as come to understand. This means we are better equipped to question our own biases. Indeed, I would argue that qualitative research can be less subjective than research that determines the definition of the problem, the questions to be asked and the order in which to ask them before starting to even engage with participants. Arguably qualitative research is objective in its desire to understand the human world in all its diversity and without prejudice. This does involve working with participants to achieve a shared understanding of a situation. But is trying to remain detached, refusing to engage or respond, trying to extract difficult and sensitive responses without any reciprocation actually any more objective? Where does that leave our definition of ‘objectivity’? I think I am going to have to come back to this one….

To conclude for now: we often talk about qualitative research as meaningful, focused on understandings, interested in emotions and experiences. We call it exploratory, giving insights into lived experience, or examining the standpoint of the research participant. But I would argue these amount to its values. 

Qualitative research is OF VALUE because, when it does all these things well, the insights produced (or the results, findings, or conclusions) are far more likely to lead to actions, interventions, recommendations, that are likely to work in practice. And this is the case, even where the only intervention we can come up with is to understand people better rather than condemn, criticise or wonder at them. 

Qualitative research: producing understandings and insights rather than ‘findings’

I’ve been feeling a bit frustrated lately because of the number of times people have asked us what are the findings of our Brexit Brits abroad project.
We have produced podcasts, government reports, policy reports, academic papers, accessible reports, infographics, and even an animation. We comment on Twitter, Facebook, and communicate regularly and personally with our participants. We’ve presented at academic seminars, at policy roundtables, and we have a very dynamic website.
Those asking this question usually know this, or can find it out by looking at the website.
Similarly, when I teach qualitative research, I am often asked the best way to produce valid findings. I am starting to think that people have different understandings of this word findings.
So, let’s clarify.
The goal of qualitative research is to understand people’s actions and behaviours in the context of the ways in which they perceive and experience the world. We often begin by examining their desires, meanings and motivations. For some , the goal is to go further and to critically analyse these understandings with reference to wider structures of power and control.
The outcome of qualitative research would thus be better understood in terms of insights, understandings, and critical analyses. This is the shape our findings take.
I think the word findings is better suited to research that is more problem focused, research that is looking for a simply-understood and linear explanation of cause, effect, and solutions.
This is not to suggest that qualitative research cannot provide understandings of causes, effects and solutions. Instead I argue that the relationship between factors and outcomes in social life is complex and ultimately unpredictable. Nevertheless, the closer we get to understanding humans and their actions, in the context of the choices that are available to them, the closer we can get to suggesting interventions that might actually work in practice.

What is the value of an anecdote?

Last week, at the end of a long day training in the art of qualitative interviewing, one of the course participants asked me: ‘but how can we show it is not just all anecdotal?’. I felt frustrated, to say the least, but I must have failed somewhere in my explanations of the value of a qualitative interview. 

pinterest-441-anecdoteAntidoteThere is such a lot of angst about how to assess the value of qualitative research that still, it seems, we haven’t fully addressed this. In my experience, the problem often seems to come from the fact that people doing qualitative research feel they need to explain why they don’t do certain things – like testing for reliability, representativeness, and validity. 

How do we know quantitative work is good quality? How many people have actually asked to see all the questionnaires, and checked them all? How often is the basic operationalising of a concept or the choice of potential responses queried? How do we know the design of questions is not anecdotal? 

It seems to me that qualitative researchers need to 1) stop being defensive and 2) recognise what they aim to achieve, before 3) thinking about how to show they’ve achieved it. 

Let’s start with the notion of an anecdote. 

It is anecdotal that every week my neighbour puts her bin out near my car and it is not always easily visible. So, every week, I check before getting in the car to ensure I don’t drive over it (again). My experience has affected my behaviour, and it works in practice! 

This is the same logic we use for qualitative research. We are trying to understand how people’s experiences and understandings affect (or might affect) their behaviour. I can’t design a survey to test for this because I won’t know what questions to ask, or responses to suggest. It is irrelevant whether the experience or understanding is an anecdote or not. It is how the person’s life is shaped by experience and how they communicate that, that matters. 

Furthermore, if by anecdote we mean something said quickly and flippantly without depth, then one thing we should be ensuring as qualitative researchers is that we explore in depth what people tell us, by listening and hearing, and working to achieve complexity, nuance, and richness (and perhaps doubt and ambivalence).

Disembodied quotes and writing qualitatively

A plea to include the people, stories, and lives in qualitative research writing, and to avoid Disembodied Quotes

When I am delivering Research Methods Training, I often ask participants to please try to avoid using disembodied quotes when writing up, or presenting. I thought it was time I clarified what I mean by this, as it is not written anywhere (to my knowledge). By disembodied quotes I mean this sort of thing: 


One of the themes that emerged in the study was that research participants felt the GPs were using terminology they couldn’t understand, and they often felt both embarrassed to ask for clarification, and pressed for time. 

“I always feel the doctor is in a rush, so when she told me I’ve got xxx, I didn’t want to ask what she meant. I know the answer would take a long time and I still not might understand” (woman, 40s).


This is OK. It is clear what is going on here, but it lacks richness, context, and personality. We learn very little about this woman who is speaking, and she doesn’t even have a name. Compare it with this. 


Research participants were reluctant to ask questions when they felt their lack of understanding was vast, or when they believed the GP was under a lot of pressure with regards to time. Sophie, a mother of two in her 40s, had visited her GP for her own health (as opposed to taking her children) for the first time in ten years . She was very anxious about her symptoms, and had tried to look them up on the internet but had felt even more confused. Her appointment was the last one in the morning, the surgery was empty apart from herself and one other person, and the GP, Sophie told me, looked frazzled and tired. Under such conditions, she didn’t feel she could ask the questions she needed to. Sophie told me,

“I always feel the doctor is in a rush, so when she told me I’ve got xxx, I didn’t want to ask what she meant. I know the answer would take a long time and I still not might understand” (Sophie, 40s, mother of two children).


This uses many more words but if the point is to help GPs understand what meanings and understandings patients bring to the interaction, then it has done a much better job than the shorter introduction to the quote, above.

Also, when people use the disembodied quotes approach they usually follow an explanation with a series of quotes, as if they are trying to show how important this insight is quantitatively. So, the word count may well end up the same, but without the richness and context. 

This desire to demonstrate the quantity of a response illustrates a basic misunderstanding of the value of qualitative research, which is to reveal meaningful insights that can be acted on to improve society. 

 

Of theories and theoretical frameworks: hold your substantive theories lightly in iterative-inductive research

Note: the take-home message is at the end!!! 

Some of the participants on recent courses I have taught (for the SRA and for the Essex Summer School) have got me thinking about what I mean by the term iterative-inductive:

“This is a practice of doing research, informed by a sophisticated inductivism, in which data collection, analysis, and writing are not discrete phases but inextricably linked”.  (O’Reilly 2012, p.5).

As we know

“A deductive approach to research is one where a hypothesis is derived from existing theory and the empirical world is explored, and data collected, in order to test the truth or falsity of the hypothesis. A simplistically inductive approach to research is one where the researcher begins with as open a mind and as few preconceptions as possible, allowing theory to emerge from the data. A more sophisticated inductivism views theory as precursor, medium and outcome of ethnographic study and writing”. (ibid. p.29)

And we know that ethnographic research tends to be more inductive than deductive, to work hard to make ideas earn their way, to want to hear and sense the field, to learn from research participants, and to achieve a way of understanding that is meaningful.

Further, the notion of abduction (which may be a better way to understand iterative-inductive) recognises an on-going interaction between the researcher, the research field, and externally sourced and internally emergent (or created) ideas over a period of time and as part of the practice of research.

However, none of the approaches above adequately deals with the role of the literature review or the role of a theoretical framework in ethnographic research that wishes to be iterative inductive. As social science develops, transfers concepts to new settings, builds on prior understandings in given fields, it becomes more and more appropriate to consider extant (or external?) or pre-existing theories and concepts. Think of concepts such as institutional racism, stigma, theories of learning, cognitive dissonance theory, etc. Some of these have become almost common-sense terms in everyday parlance. No researcher can ignore what they already know. Furthermore PhD supervisors and colleagues often ask an ethnographer: ‘what is your theoretical framework?’ implying that they will have a toolkit of theories and concepts to pull out of a bag and use in the field.

As time has gone by, my own research has taken shape, developed, and built on previous ethnographic understandings to become something of a body of research on British abroad. I am increasingly using a practice theory (or structuration) theoretical framework, which shapes what I do and how I make sense of the worlds I inhabit. This theoretical framework is somewhat independent of the research, the field, and our iterative-inductive understandings. It is, for me, a starting point, but strangely it has come to me over time as an end-point, a reflection and reconsideration of earlier work. I use it to construct Practice Stories

“Practice stories pay attention to people’s feelings and emotions, their experiences and their free choices, but also to the wider constraints and opportunities within which they act. More than that, practice stories take account of how these different features of social life interact, and thereby how structures (like social classes, for example) get produced or reproduced. Theoretically, practice stories can draw from a wide range of social theory that comes under the description of structuration theory or practice theory”.

However, a practice theory framework does not exclude the use, application, or amendment of more substantive series that are created as part of the research or that are imported from previous studies. As I have argued in International Migration and Social Theory.

man in button up shirt

Photo by Khaled Reese on Pexels.com

So, perhaps a theoretical framework is actually more of a broad-brush all-encompassing approach to understanding wider aspects of human behaviour, while it is the more substantive theory and concepts that we wish to hold lightly (to quote one of the recent summer school participants, a design ethnographer), or use as sensitising concepts (to quote Herbert Blumer) in iterative-inductive ethnographic research.

 

 

Essex Summer School inspires me again!

I’ve been teaching Ethnography at the Essex Summer School the past two weeks. It has been amazing. The participants have worked incredibly hard and have produced some wonderful research projects. I’m trying to think what it is about the summer school that is so warm and creative and makes the participants so enthusiastic. I think it might partly be that we are all there together for an intensive period of time. Also, everyone has chosen to come and already has a passion for the subject they have chosen. All I can say is that it is the best possible teaching experience. Imagine students that do all the reading that you ask them to do and more besides, students who work well beyond the classroom hours and have to be reminded to take a break, students who have a lot to contribute, who think, challenge, contradict, and bring their own experiences and insights so that as a teacher you are not only teaching but also learning. I have come home with a list of references to follow up, things I am excited to read, lots of ideas buzzing around my head, and an enthusiasm for my own work. Thank you to the especially enthusiastic and inspiring participants this year. You know who you are. And thank you to Mel and Jo and Dan, who support us all so well.

IMG_0184

Participants eagerly take part in an activity

Ethnographic Methods or Ethnography? The answer is in the eye of the beholder

9780415561815   Whenever I teach ethnographic methods someone asks me a question along the lines of: “I am not sure if I am doing a proper ethnography”. Or “I am not doing a full ethnography “.What they mean is that they are not going to spend a very long time doing participant observation, or they are not going to move in to an area and live with the people they want to learn about, or they don’t have time to spend much more than some portion of each day in the context, or perhaps the setting or group being studied is part-time itself.
This reveals, perhaps inadvertently, that people assume there is some kind of gold standard or accepted definition of what ethnography is or should be. But, I don’t know where this gold standard or accepted definition is held, shared, or defended.
Usually, it is in the eye of the beholder.
As I always tell students, there are no ethnography police or watchdogs, guarding standards. The people who will check whether your work qualifies as an ethnography will be journal editors, PhD examiners, publishers, and diverse readers of your work. Each of these will have different ideas of what counts as a full ethnography, and it will be them ( or some of them ) that you will need to satisfy. There is no single standard definition. My recommendation is that each of you working in this field decides for themselves what they consider being worthy of being called an ethnography, and be prepared to defend this in your writing. Explain to the reader what you did, when you spent time, who with, under what sorts of conditions, and – especially – why. What did participant observation add of value to your research? This is what is important, rather than deciding whether your work is an ethnography or not. It is not labels that matter, but practices.
If you want to use ethnographic methods but you are unsure as to whether you can call what you are doing ‘an ethnography’, then you could say you are using ethnographic methods or that you are informed by ethnographic methodology. This is why I have called my book Ethnographic Methods and not Ethnography.
There I argue thatEthnography should be informed by a theory of practice that: understands social life as the outcome of the interaction of structure and agency through the practice of everyday life; that examines social life as it unfolds, including looking at how people feel, in the context of their communities, and with some analysis of wider structures, over time; that also examines, reflexively, one’s own role in the construction of social life as ethnography unfolds; and that determines the methods to draw on and how to apply them as part of the ongoing, reflexive practice of ethnography”. 

The SRA in Edinburgh

IMG_4920

I spent a really lovely few days in Edinburgh recently providing training for The SRA. The courses were Designing Qualitative Research; Qualitative Interviewing; Conducting Focus Groups. All three courses were full up, with waiting lists, and the participants were fantastic – so much expertise and enthusiasm in one room! Some participants attended two or three days so we got to know each other well, and people made new friends. I tried using ‘play dough’ for the first time in the Focus Groups course. I think it went well – we had fun anyway. Participants came up with some interesting interpretations of un/healthy! We also learned that it might be better to get participants to shape their dough in groups rather than on their own, to avoid the pressure to perform, and the silence. It went very quiet while they sculpted seriously.

It is always lovely to have a few days in Edinburgh and meet up with old friends as well.

The British on the Costa del Sol Twenty Years on

I have published two new papers about British in Spain, and they are free to access! Indeed, readers may enjoy the entire special issue, all about British and Swedish living in Spain. Open Access

2017. O’Reilly, K. The British on the Costa Del Sol Twenty Years On: A story of liquids and sediments. Nordic Journal of Migration Research, 7(3), 139–147. Open access 

2017. Olsson, E., & O’Reilly, K. North-Europeans in Spain: Practices of Community in the Context of Migration, Mobility and Transnationalism. Nordic Journal of Migration Research, 7(3), 133–138. Open access

 

The Brexit Brits Abroad Project Web site is up and running

What does Brexit mean for the 1.2 million British citizens living in the EU27? 

BrExpats: freedom of movement, citizenship and Brexit in the Lives of Britons resident in the EU27  is an innovative sociological study  that questions what Brexit means for Britons resident in other European Union member states. Working closely with Britons living across the EU27, It keeps a finger on the pulse of how they experience Brexit and its impacts on their lives as it unfolds.

Have a look at the web site to read about our project, our participants, and our plans