Basic HTML Version
© 2014 Pacific Crest
327
O
ops
! A
voiding
C
ommon
E
rrors
●
Making assumptions about what the data represents
Example
: Assuming that a table of Olympic medals represents total medals won after the
Olympics are over.
Why?
Table of medals won are generated frequently while the Olympics are ongoing. It is
often the case that data is updated and reflects a point in time or a specific purpose.
You must verify what the data actually represents before drawing conclusions.
●
Not clarifying the purpose of the analysis or ignoring bias
Example
: A newspaper runs a story about gun control, much of which is based on an NRA data
analysis report concluding that gun-related homicides are down by 40%. This means
that there’s little need for gun control and that there’s less and less of a problem, right?
Why?
Before you use data analyses produced by others, you should know the objectives
and agenda of the group or person analyzing the data. Different groups have different
agendas: The interests of a citizen advocate for gun-control may differ greatly from
those of a politician trying to find out what is going on, a politician trying to be re-
elected, or a gun manufacturer. Your purpose in analyzing data tends to dictate your
process, questions, conclusions, and justification supporting your analysis, even when
you are unaware of it. Confirmation bias isn’t always intentional. For example, if a
researcher is paid by someone who has a vested interest in the results, the researcher
may subconsciously steer the findings in the direction of his employer.
●
Not presenting the results or conclusions with the appropriate tools
Example
: Using stacked bar graphs to show contribution to annual homicides by category
Why?
The form of the communication should clearly frame the results and conclusions
in a convincing way. While a stacked bar graph does represent the data, it does not
highlight the dramatic rise in a single category. Trend analysis is a major conclusion
from this analysis and a line graph is a significantly more powerful representation of
this conclusion.
●
Generalizing conclusions inappropriately
Example
: A focus group produced compelling evidence that few teenagers consider attending
state universities. The focus group spoke with 15 eleventh graders from two different
private high schools in New York City.
Why?
A researcher should not generalize results if the original data comes from a small
and not representative sample. In general, the larger the number of items in a random
sample, the more reasonable it is to extend those results to the larger group. The
limitations of this study include that the students were all enrolled at private schools
(which typically indicates more a secure family financial situation than for students
at public schools), they were all in New York City (an extremely metropolitan area
and vastly different from rural New York state, for example), etc. These kind of
limitations and constraints must be noted in the analysis and should send up a red flag
when it comes to potentially generalizing the findings.
7.3 Data Analysis