For researchers and research-readers alike, the data analysis portion of a study is many things: complex, exciting, frustrating, intriguing, and sometimes even befuddling.
And, analytics are always on the move. With each new study, researchers are developing increasingly intricate and elegant ways to make meaning of their data. At the same time, powerful statistical software, like SPSS or Stata, is continuously expanding its capability to process such sophisticated research designs.
Certainly, many long hours go into choosing a study’s analytic approach. Researchers must develop and refine their hypotheses; organize their data in such a way that statistical software can read it; and choose a statistical method (i.e., a mathematical approach) to test their research questions.
That last part about choosing a statistical method is where things can get tricky. In general, different statistical methods are not simply multiple ways of doing the same thing. Whereas something like a division problem may use different techniques (e.g., long division, trial- and-error) to get the same result, different statistical methods can analyze the same data yet produce differing, and even contradictory, results.
Differences in design: A little goes a long way
Just as French philosopher Jean-Paul Sartre liked to say “we are our choices,” in many ways our research results are our choices, too.
A study conducted by Burchinal & Clarke-Stewart illustrates this well. [1] These authors noticed that two different research teams had analyzed the same longitudinal data set, yet found (and published) substantially different results.
These two research teams analyzed data from the National Institute of Child Health and Human Development (NICHD) Study of Early Child Care: a large, national study that followed the development of 1,364 children from six months of age. Both teams were also interested in the same question: what is the impact of mothers’ employment, and children’s subsequent nonmaternal child care, on children’s early cognitive growth?
The NICHD Early Child Care Researchers (Team 1) were first in line to test this question. After a series of analyses, this team concluded that the age at which children entered nonmaternal care, and the amount of time spent in such care, showed no relation to children’s cognitive performance up to age three. [2]
Next, Team 2 (Brooks-Gunn, Han, & Waldfogel, 2002) tested this same question. However, in contrast to the Team 1, they concluded that mothers’ full-time employment during children’s first nine months was indeed associated with impaired cognitive functioning when the children were three years of age. [3]
Speaking different analytic languages
The contradictory findings between these two research teams were not only curious, but also important to reconcile. After all, the difference between advising mothers of young children to work or not work is a big one. And, such a recommendation has implications for state and federal programs, such as Temporary Assistance for Needy Families, that assist young mothers in finding employment.
Burchinal & Clarke-Stewart therefore conducted a new, third study investigating how each team’s analytic design may have engendered the contradictory results.
Two approaches
First, Team 1 team used a conservative, top-down analytic approach. This approach:
- uses all available information, such as multiple outcome variables and data from all participants
- begins with a general test of significant relations between variables and works its way down to more specific comparisons
- helps researchers avoid exaggerating the significance of associations found when working with large data sets
Team 2, on the other hand, used an a priori comparison approach. This technique:
- examines hypotheses and variable relations chosen by researchers before (a priori) data exploration
- utilizes a small subset of participants and/or variables in order to conduct a small set of comparisons between explicitly chosen participants and/or variables
- is helpful when theory or previous research strongly implies a relation between specific variables or constructs
Thus, it seemed likely that investigating a smaller group of participants, and analyzing a smaller set of outcome data, contributed to Team 2’s finding of a relation between maternal employment and children’s cognitive growth. On the other hand, utilizing the full set of study participants, and analyzing all possible child outcome data, seemed to result in Team 1’s lack of such a finding.
To confirm this hypothesis, Burchinal & Clarke-Stewart analyzed the same participants and variables that Team 2 did; but, they used the top-down approach this time. The result of these new analyses? No significant findings.
The authors therefore reported Team 1’s findings—that is: it doesn’t hurt young children for their mothers to get a job—as being a more reliable take-away.
A cautionary tale
It is important to note that both the top-down approach and the a priori comparison approach are well-respected and well-established analytic techniques. And, as with all analytic techniques, each has strengths, weaknesses, and research questions for which its use is optimal.
But a study such as the one conducted by Burchinal & Clarke-Stewart provides an important cautionary tale. That is, when we, as consumers of research findings, draw conclusions from empirical work, it is important to remain attentive to the type of analyses that were used to engender such claims.
Of course, we probably won’t all end up being experts in all areas of analytic approach. But perhaps a good rule of thumb is that when we see a small amount of data being used to make big claims, it’s best to take a second look, get a second opinion, or see if the study has been replicated a second time.
References
[1] Burchinal, M.R. & Clarke-Stewart, K.A. (2007). Maternal employment and child cognitive outcomes: The importance of analytic approach. Developmental Psychology, 43, 1140-1155. [2] Brooks-Gunn, J., Han, W.J., & Waldfogel, J. (2002). Maternal employment and child cognitive outcomes in the first three years of life: The NICHD Study of Early Child Care. Child Development, 73, 1052–1072. [3] National Institute of Child Health and Human Development Early Child Care Research Network. (2000). The relation of child care to cognitive and language development. Child Development, 71, 960–980.
