Robert sent me the link to a review of a book about decision-making that features the opportunity to take the self-assessment survey featured in the book - how do you rate on the 6 traits they identify as comprising personal decision-making style? There are 5 yes/no questions for each trait.
The test is seemingly pretty bad. I mean, what should we think of things things like:
* The question "Do you always lock your door no matter how long you'll be gone?" somehow measures my short-term vs long-term viewpoint rather than risk while the question "How many speeding tickets have you had in the last five years?" measures risk rather than time preference.
* One of the 5 questions about information use in decision-making is "Do you watch The Daily Show or The Colbert Report?"
* Another one for information is "Do you like to try different flavors of ice cream before choosing one?"
* Two of the questions about status-seeking relate to what is culturally rather than personally valued.
* "Enjoying foreign travel" makes me not a loyal person.
Yet we know that face validity is a very poor indicator of whether scales are measuring what people want them to measure. These could be really good questions (in that they are correlated with what we want them to correlate to, don't correlate with what we don't want them to, and predict the right kinds of things) even though they look crappy. Without any information on how these scales were developed and validated, and what the reliability of the scales are, it's hard to know for sure.
But they sure look stupid, don't they? Especially for a mass-market (or perhaps small-business-focused) book, you'd think they could have developed scales that had both decent psychometric qualities and didn't look so idiotic on their face. I mean, we are to believe that one of the best 5 questions for getting at the "using information to make decisions" construct is whether a person tries different flavors of ice cream before choosing one? How often do people even eat ice cream in a place where they can sample ice cream and yet they don't already have a favorite flavor? (And this question doesn't load on the "risk" factor?) It just seems implausible that these scales are doing a very good job of assessing people's preferences, and I think that face implausibility can only hurt their credibility unless it is explained really well in the book why their seemingly dumb questions can yield useful insights.
Subscribe to:
Post Comments (Atom)
6 comments:
Actually, that would really be a useful business book. "Stupid Question, Smart Answer - how seemingly irrelevant information can lead to shrewd decisions"
Or sub in "big profits" for "shrewd decisions".
You need more of a hook, like the "who moved my cheese" people, or the fox vs. hedgehog thing. Maybe something like "Think Like a Cucumber, Move Like A Fighter Jet."
Or, you know, make up your own moronic analogy.
I like it.
Tam, you have to pick a) or b). "I like it" is not an appropriate answer.
**
I like to try different flavors of ice cream even when I already think I know which flavor I want (though I am open to changing my mind based on what these flavors taste like). I'll even try flavors that I'm sure I won't like (such as Guinness flavor, which turned out to be not bad at all except for the beer aftertaste). This shows that I am a thrill-seeking risk taker. Or maybe I just like extra free ice cream. Actually, this one sounds like another test for loyalty.
I'm going to vote that these are crappy questions. A good question should be answerable. Unlike "Do you prefer to buy the most extensive or the minimal insurance?" I like high coverage amounts (= extensive) but also high deductibles (= minimal).
Post a Comment