An Evaluators Toolkit: Understanding Your Data and How to Analyze it.
Introduction
This integrative reflection paper is based on International Development 319: Quantitative Methods and Statistics for Evaluators. Each topic builds on the previous topic, making it easier to see and understand its importance and relevance. Introduction to statistics was preceded by how to examine data, followed by measures of central tendency (the mode, median, and mean) and measures of variability (the range, interquartile range, variance, and standard deviation). The understanding of normal distributions was the starting point of our journey to learning inferential statistics. Normal distributions was followed by understanding the concepts of probability and hypothesis testing. After that, introduction to evaluation, research, and surveys was followed by inferential statistics (t-tests, correlation, regression, ANOVA, chi-square). The course culminated in an evaluation proposal that made use of all learnings during the course. The text used for this course, “Fundamental statistics for the social and behavioral sciences,” by Tokunaga, H. was easy to read, had many examples and screenshots of analysis steps.
Quantitative Methods and Statistics for Evaluators
Introduction to statistics
Statistics are a part of our everyday life. For example, when searching for a new apartment, we consider the prices of the options available to us and choose which apartment to go for based on criteria that are important to us. Statistics is not only about data analysis, but it also includes understanding data collection and the interpretation and communication of results from our analysis using the scientific method. The scientific method “uses the objective and systematic collection and analysis of empirical data to test theories and hypotheses” and can be described as consisting of five main steps. These steps are as follows:
· Developing a research hypothesis to be tested
· Collecting data
· Analyzing the data
· Drawing a conclusion regarding the research hypothesis
· Communicating the findings of the study
Stage 1: Developing a research hypothesis to be tested
A prior understanding of the topic of research interest helps the researcher or evaluator identify questions that have not been answered or partially answered and the contribution of their research or evaluation. The hypothesis is the researcher’s expected or predicted answer to their question of interest. This hypothesis is then subject to testing. When stating a research hypothesis, the researcher uses terms such as “independent” (variables manipulated by the researcher) and “dependent” (variables measured by the researcher) variables. Researchers examine the effect of the independent variable on the dependent variable. For example, a researcher interested in the effect of a baby meal formulation on stunting in children under 5 years old, subject to ethical approval, might have groups of children on various meal formulations and measure the children’s heights periodically. The different meal formulations are the independent variable, while the children’s heights are the independent variable.
The research hypothesis could also be a directional or non-directional research hypothesis. In our baby meal formulation example above, a non-directional hypothesis is that “the baby meal formulation has no effect on the height of children.” A directional research hypothesis would be stated as “the baby meal formulation contributes to an increase/decrease in children’s height.” In addition, “the ability to form a directional research hypothesis is dependent on the state of the existing literature on the question of interest, if little or perhaps conflicting research has been conducted, researchers may not be able to form a directional hypothesis before they begin their research.”
Stage 2: Collecting the data
After a research hypothesis has been formulated, the next step is to collect data relevant to the hypothesis. Collecting data comprises of three main steps:
· Drawing a sample from the population
· Determining how the variables will be measured
· Selecting a method by which to collect the data
It is not possible to collect data from all members of a population because of time and cost constraints. Therefore, data is collected from a representative subset of the population. Still using our example above as a reference, our sample could be 200 mothers in our study location, 100 younger mothers aged 21- 30, and 100 older mothers aged 31–40. Variables are measured by assigning categories or numbers to objects or events using four distinct levels of measurement: assigning names to categories using nominal variables; assigning ranks or sizes relative to other values using ordinal variables; using intervals equally placed along a numeric continuum; when these intervals have a true zero point they are called ratio variables. In our baby meal formulation, our independent variable, the type of baby meal formulation, is a nominal variable. Our dependent variable, the height of the children, is a ratio variable. An example of an ordinal variable could be asking mothers to list the options given to their children and indicate their order of preference — first, second, third, and so on — depending on the number of options listed.
Finally, data is collected using an experimental or non-experimental research method. Experimental research methods are used to prove a causal relationship between the independent and the dependent variables. For example, a researcher might be interested in showing that the baby meal formulation mentioned earlier causes a significant increase in children’s height. To do this, the researcher would need to eliminate all other possible causes or explanations for changes in the dependent variable besides the independent variable, the baby meal formulation. If it can be claimed that a variable other than the independent variable created the observed changes in the dependent variable, the research results are said to be confounded. To address confounders (confounding variables), the researchers either exercise experimental control by making the research setting the same for all participants or creating a control group where participants are not exposed to the independent variable the research is designed to test. Researchers can also minimize the influence of confounding variables by assigning participants to each category of an independent variable so that each participant has an equal chance of being assigned to each category.
Non-experimental methods, also referred to as correlational research methods measure naturally occurring relationships between variables without inferring cause-effect relationships. Examples of non-experimental research designs include quasi-experiments, survey research, observational research, and archival research. Quasi-experimental research compares naturally formed or pre-existing groups rather than employing random assignment to conditions. Continuing with our baby meal formulation example, a quasi-experiment would identify persons who already use the baby meal formulation of interest and those who use another baby meal formulation. Those who use the baby meal formulation of interest are assigned to a group whole those who use another baby meal formulation are assigned to a second group. Survey research obtains information from people using a questionnaire. This information is used to describe a phenomenon, predict future behavior, or estimate the views of a larger population. Observational research is used to study phenomena that the researcher either cannot or should not deliberately manipulate. For example, a researcher studying the effects of opioids on teens’ academic achievement should not make a group of teens take opioids. By observing opioids use and academic behavior, the researcher can obtain data to understand the relationship between the two variables better. Archival research uses archives (records or documents of the activities of individuals, groups, or organizations) to examine research questions or hypotheses.
After data has been collected, the next step is analyzing the data, interpreting results, and communicating findings. Data is analyzed using either descriptive or inferential statistics. Descriptive statistics are used to “summarize and describe a set of data for a variable.” In contrast, inferential statistics are used to “test hypothesis and draw conclusions from data collected during research studies.” The results of the analysis are interpreted as they relate to the research hypothesis. The conclusions from inferential statistics can either support or not support the research hypothesis. Because researchers do not collect data from the entire population and phenomena studied by social researchers are complex, it is extremely difficult for one research study to prove a hypothesis or theory as entirely true or false.
Reflection — Introduction to Statistics
Any research or evaluation project’s planning phase is a crucial time because important decisions are made that influence other aspects of the project. As a professional in the health sector or monitoring and evaluation, choosing how to measure your variables impacts how research questions and hypotheses are stated and determines the statistical procedures used to analyze the data. For example, if in our study on baby meal formulations, if children’s height is measured as an ordinal variable or nominal variable with children grouped into categories based on their height, this data will not be analyzable using correlation (chi-square analysis). Chi-square analysis requires that at least one variable be an interval or ratio variable since the independent variable, our baby meal formulation type, is also a nominal variable.
Examining Data
Examining data involves using counts and frequencies to identify any trends, outliers, positive or negative skews in the data and identify numbers that provide summary data. These numbers are called point estimates. These point estimates have their advantages and disadvantages ad their relevance is often determined by the context.
Before analyzing data, researchers examine data to have an initial sense of the data, detect data coding or data entry errors, identify outliers, evaluate research methodology, and whether data meet statistical criteria and assumptions. A data entry error could be the height of a child measured as 360 inches, whereas the correct value was 36 inches. Examining the data would spot this outlier and data entry error. Also, in a situation where after examining the data, the researcher thinks the data on baby meal formulation types is inaccurate, the researchers could include in the methodology that the researchers take a picture of the baby meal type. An independent quality control staff can compare responses written down to ensure they align with the pictures taken.
Data is also examined using tables and figures. Tables show all the values of the variable, how many participants in the sample have each value of the variable (frequency) and the percentage of the sample that has each value of the variable. Figures used to examine data include bar charts, pie charts, histograms, and frequency polygons. Bar and pie charts are used for variables measured at the nominal or ordinal level of measurement. In contrast, histograms and frequency polygons are used for variables measured at the interval or ratio level of measurement.
From examining the data, the researcher can describe the distribution of values or scores for a variable. Three features of a distribution that are often described are its modality, symmetry, and variability. The modality of a distribution is the values of the variable that have the highest frequency or occur most often in a set of data. Unimodal distributions have one value with the greatest frequency, bimodal distributions have two values with the greatest frequency, and multimodal distributions have more than two variables with the greatest frequency. The symmetry of the distribution refers to how the frequency of values of a variable change in relation to the value with the greatest frequency. In symmetric distributions, the variables change in a similar manner on both sides of the value with the highest frequency. In asymmetric distributions, the variables change differently on both sides of the value with the highest frequency and can be positively or negatively skewed. The variability of the distribution refers to the amount of differences in a distribution of data. Peaked distributions have much of the data in a few values of a variable. Flat distributions have data evenly spread across the values of a variable. A normally distributed variable has a symmetric distribution that is neither peaked nor flat.
Reflection — Examining Data
Many health research and evaluation papers often begin their analysis by presenting data tables and stating frequencies as percentages. These tables give a good overview of the data. The modality, symmetry, and variability of the data can be gauged by examining the data in the tables or converting them to charts. Examining data is a very valuable first step before in-depth, or other inferential data analysis is conducted.
Measures of Central Tendency
A measure of central tendency is a single score that identifies the center of a distribution. These measures of central tendency are the mean, median, and mode. The mode, modal score, is the score or value of a variable that appears most frequently in a data set. The median, the value of a variable that splits the distribution of scores in half with the same number of scores above the median as below. The mean is the arithmetic average of a set of scores. The mean is determined by calculating the sum of a set of scores and dividing this sum by the total number of scores. The mean uses all of the scores in a distribution of data, rather than just the most frequently appearing scores or the one or two scores in the center of the distribution.
Reflection — Measures of Central Tendency
During this course, we identified research or evaluation papers that used measures of central tendency as a core element of their data analysis. In their study, Bloss et al. (2005) surveyed 175 children in three villages in western Kenya. They presented a table showing the means, standard deviations, and median z-scores for height and weight among boys and girls aged 5 and under in intervals of one year. They analyzed this data alongside other socio-demographic data and found out that “children in their second year of life were more likely to be underweight and stunted, children who were introduced to foods early had an increased risk of being underweight, up-to-date vaccinations were protective against stunting, living with non-biological parents significantly increased risk of stunting.” Lowder et al. (2016) made extensive use of the mean (average) in the study of the number, size, distribution of farms, smallholder farms, and family farms worldwide. The study shows that “average farm size decreased in most low- and lower-middle-income countries for which data are available from 1960 to 2000, whereas average farm sizes increased from 1960 to 2000 in some upper-middle-income countries and in nearly all high-income countries.”
These studies show how valuable measures of central tendency are despite their simplicity. Understanding the research question and objectives of the study will help a researcher determine how best to use measures of central tendency either as the primary data analysis approach or as an augment to inferential statistical methods.
Measures of Variability, Normal Distribution, Probability, Hypothesis Testing, Inferential Statistics
Variability is a third aspect of distributions and refers to the amount of spread or scatter of scores in a distribution. A measure of variability is a descriptive statistic of the amount of differences in a set of data for a variable. Measures of variability include the range, the interquartile range, variance, and the standard deviation. The range is the mathematical difference between the lowest and highest scores in a set of data. The interquartile range is the range of the middle 50% of the scores for a variable, calculated by removing the highest and lowest 25% of the distribution. The variance is the average squared deviation from the mean. The standard deviation is the square root of the variance, and it represents the average deviation of a score from the mean. Because the variance and the standard deviation (unlike the range and interquartile range) are based on all the scores in the distribution, they can be used in statistical analysis designed to test hypotheses about distributions. The standard deviation is also a key statistic in normalizing the distribution of a variable (conversion to z-scores), calculating the probability of data values, and hypothesis testing.
The image below from Aron, Coups, and Aron’s “Statistics for the Behavioral & Social Sciences: A Brief Course” (2011), is a decision tree to help researchers and evaluators decide on the inferential statistics method for their data analysis.
Source: Aron, A., Coups, E., & Aron, E. (2011). Statistics for the Behavioral & Social Sciences: A Brief Course. New York: Prentice Hall.
Reflection — Measures of Variability, Normal Distribution, Probability, Hypothesis Testing, Inferential Statistics
The researcher or evaluator’s understanding of what the research question is, why the data collection tools chosen are the best tools to answer the research question, and if the inferential data analysis methods chosen are the most appropriate for the hypothesis that is being tested lead to truly valuable research. A researcher’s clarity about the steps they have taken helps make the study replicable by other researchers and contributes to the validity of the research.
In conclusion
Quantitative methods and statistics is very important for health and monitoring and evaluation professionals. There is data to evaluate, inferences to be made, and the best decisions arrived at based on a thorough and in-depth understanding of the data. The tools and knowledge gained from this course prove very valuable in this regard.
References
Aron, A., Coups, E., & Aron, E. (2011). Statistics for the Behavioral & Social Sciences: A Brief Course. New York: Prentice Hall.
Bloss E., Wainaina F., Bailey R. (2005, October 5). Prevalence and Predictors of Underweight, Stunting, and Wasting among Children Aged 5 and Under in Western Kenya. Journal of Tropical Pediatrics. Volume 50, Issue 5 (pages 260–270).
Clark University (2021). Moodle page for International Development 319: Quantitative Methods and Statistics for Evaluators.
Lowder S., Skoet J., Raney T. (2016). The Number, Size, and Distribution of Farms, Smallholder Farms, and Family Farms Worldwide. World Development, Volume 87 (pages 16–29).
Tokunaga, H. (2016). Fundamental statistics for the social and behavioral sciences. Sage Publications.