Welcome to the third blog in our series on Demystifying Evaluation from Karl McGrath and Dearbhaile Slane.
Here we look at some of the data collection methods that we often use in CES evaluations, outlining when and why you might utilise certain methods and the benefits and limitations associated with each.
Data collection methods usually fall into two categories: quantitative research methods and qualitative research methods. When methods are combined, this creates a third category known as mixed methods research.
Quantitative research methods use numbers as data. They are often used to identify relationships between different aspects of the data (variables) that can be generalised to a wider population. In evaluations, quantitative research methods work well with questions like ‘does X lead to Y?’ or ‘how much impact did X have on Y?’. Compared to qualitative research methods, quantitative research methods are usually less time-consuming and resource-intensive and lend themselves to larger sample sizes.
Qualitative research methods use words and images as data. They usually seek to understand people’s experiences within their specific contexts. In evaluations, qualitative research methods work well with ‘how’ and ‘why’ type questions. Compared to quantitative research methods, qualitative research methods tend to be more time-consuming and resource-intensive and lend themselves to smaller sample sizes.
Mixed methods research uses either multiple quantitative research methods, multiple qualitative research methods,or some combination of both. Gathering and triangulating evidence from a mix of different sources can strengthen the robustness of your evaluation findings.
While lots of factors can influence the choice of data collection methods, your evaluation questions and aims should be your main guide.
Common data collection methods that we’ll explore in the rest of this blog are:
Our first three data collection methods -interviews, focus groups and questionnaires - all share a common feature: they ask people questions that the evaluator would like to know the answer to. However, they each come with different strengths and weaknesses for asking questions.
Interviews can be used to capture in-depth individual perspectives. Interviews tend to produce rich qualitative information and be more adaptable than questionnaires, providing opportunities to clarify or expand on people’s answers. However, interviews can be more time and resource intensive in terms of data collection and analysis.
Focus Groups also produce rich qualitative information. Focus groups can be used to capture a diverse range of perspectives and encourage interaction between participants, or when you want to confirm your analysis with a wide variety of ‘service user’ profiles. Focus groups can be less resource intensive than interviews in terms of data collection and capture a greater number of perspectives. People need to feel safe to share their views and as such, they are not always appropriate for discussing sensitive topics. Focus groups also require a skilled facilitator who can manage group dynamics and ensure that everyone has an equal opportunity to participate.
Questionnaires can be used to understand the general characteristics or opinions of a group of people. It is more common to see questionnaires collect quantitative data, but they can also be designed to collect qualitative data.
If you want to collect quantitative data, the questionnaire should ask closed questions with a pre-defined set of options that respondents have to choose from as their answer. You can then count the frequency of selection for the various answers. For example, a closed question with pre-defined response options might look like this:
When trying to design closed questions, using previously developed standardised measures are a good place to start. These are questionnaires or assessment tools that have already been developed to measure a particular set of behaviours and/or attitudes. CES hosts a database of standardised measures called the Child, Youth and Family (CYF) Interactive Database. This is a free resource which pools detailed information about standardised measurement tools that have been used in Irish research with children, young people and their families and allows you to search for specific tools by theme and age group.
If you want to collect qualitative data, the questionnaire should ask open questions with space for respondents to write their answer freely. For example, an open question with space for a response might look like this:
Compared to interviews and focus groups, questionnaires are less time and resource intensive. They can also reach a much larger number of people, which is useful if you want to generalise your results to a wider population.
Our next set of data collection methods -- observations, monitoring systems and document analysis --are (usually) less reliant on asking people questions and gather data in different ways.
Observations involve the researcher carefully ‘seeing’ things (e.g. behaviours, activities, processes, relationships, events, etc.) in specific settings and then recording what they have seen. The observations are usually guided by an observation schedule, which provides guidance on what and how to observe, as well as providing a space for recording notes. In evaluations, it is rare to see observations used as a method on their own. They are more often used as part of a mixture of methods, where the data is then compared and triangulated with that of other sources.
They are many different types of observations, such as structured, unstructured and semi-structured observations, and participant and non-participant observations.
Structured observations are carried out according to explicit rules or guidelines, and the observation schedule will be designed like a questionnaire with closed questions that the researcher completes. In unstructured observations, there is no specific guidance on what to observing or record. Instead, the observer will try to observe and record as much detail as possible about everything they’re observing. And then semi-structured observations fall somewhere in the middle by, for example, providing some guidance but also some flexibility on what to observe or record.
Participant observation is a qualitative method where the researcher actively ‘participates’ in the thing they are observing. They are both an ‘observer’ and a ‘participant’ with the goal of fully immersing themselves in the environment they are observing. Most participant observation is unstructured and is often unsuitable to evaluations because the evaluator will usually want to maintain their independence from the thing being evaluated. With Non-participant observation, on the other hand, the observer does not get involved in the thing they are observing. They simply observe as an outsider and record notes. Non-participant observations can be structured, semi-structured or unstructured, and are far more common in evaluations.
There are strengths and limitations too bservations. Perhaps the main strength is that it gives the researcher a first-hand view of the situation of interest, rather than relying on the views of others, as is the case with interviews, focus groups and questionnaires. However, some of the limitations include:
Monitoring Systems can be used to collect data about a service or intervention in a systematic and continuous way. The data is usually recorded by the service staff and management, rather than the researcher. The kind of data often collected is quantitative data about the amount of activities, processes, outputs and initial outcomes a service produces. Monitoring systems can also be designed to collect qualitative data that helps to understand the quantitative data in more detail, though this is less common. Either way, by collecting data continuously, monitoring systems provide a useful way to track progress over time.
When developing a monitoring system, some good practices include:
Document Analysis, like observations, is often used in combination with other data collection methods as a way to compare and triangulate findings. It involves systematically reviewing documents that are relevant to your evaluation questions. There are lots of different ways of ‘systematically’ analysing documents, but all document analyses should follow the same basic procedure of finding, selecting, analysing and synthesising the data contained in documents.
In theory, there is no limit to the types of documents that can be included in your document analysis. What matters is that they are relevant to, and help you to answer in some way, your evaluation questions. In evaluations that CES have conducted, document analysis is often used to collect and analyse in-depth qualitative data about, for example, the way an intervention works and the context it is taking place.
