Significance testing and the real tasks of social science
from Lars Syll
After having mastered all the technicalities of regression analysis and econometrics, students often feel as though they are masters of the universe. I usually cool them down with the required reading of Christopher Achen’s modern classic Interpreting and Using Regression. It usually gets them back on track again, and they understand that
no increase in methodological sophistication … alter the fundamental nature of the subject. It remains a wondrous mixture of rigorous theory, experienced judgment, and inspired guesswork. And that, finally, is its charm.
And in case they get too excited about having learned to master the intricacies of proper significance tests and p-values, I ask them to also ponder on Achen’s apt warning:
Significance testing as a search for specification errors substitutes calculations for substantive thinking. Worse, it channels energy toward the hopeless search for functionally correct specifications and diverts attention from the real tasks, which are to formulate a manageable description of the data and to exclude competing ones.