Definining the sample -- distortion by consent

"The New Know Nothings" -- screwing up survey research data
(Morton Hunt, 1999, Chapter 7, pp. 215 ff.)

Under the guise of concern for family privacy, participation in survey and other research involving children has apparently operated under different rules from one political administration to the next in Washington (DC).

(1) In some cases, a consent form sent home had to be returned checked "no", otherwise the child was to participate (unreturned forms meant "yes").

(2) In other cases, the consent form sent home had to be returned checked "yes" to participate (unreturned forms meant "no")

Either way the sample consists of students who have "given consent," but with consent defined quite differently (and the difference will seldom, if ever, be noted in a report's Method section).

Consider research from the schools for a 4 (or 8) year period under plan #1, then research switches to plan #2 for 4 (or 8) years, and so forth. Who would be surprised if there emerge conflicting data patterns in studies over the years? (Meta-analysis cannot fix this)

(3) Then there is plan #3, "the government way" -- no choice, no consent, just total compliance ("ve hav vays, you vil consent"). For example, it's state achievement test or provincial exam time, and ALL students participate -- barring an act of God (oops, Mother Nature), in which case there is a rain date.

Is it any wonder we get different answers to what appears to be the same question? Is it any wonder findings are so hard to replicate? No, if anything it's amazing we get any agreement at all.