Economists Reinvent Wheel, Only It's Not Very Round

Over at the Odd Numbers economics blog comes this post about measuring people's preferences using what the researchers call a "new survey format." The study, which is basically a proof of concept paper for their "new" methodology, can be read here by clicking document download in the upper left corner (at least, it can if one is both willing and able to put up with a little bit of economic jargon).

Here's how this "different lives survey" works:

"To test the feasibility of the different lives format, we conducted a study of 40 students in a university in London and 32 students in a university in Philadelphia....There were two levels of each of the four dimensions of well-being.

Life expectancy was 65 or 75; health was specified as being ‘able to move around freely’ or ‘hard to move around without assistance’; and happiness was expressed 95% or 80% of the time in a good mood. Income in the UK was £45,000 or £30,000, whilst in the US it was $300,000 or $100,000. The numbers were different because of different expected earnings of the students and the relativities were different to test whether the results were sensitive to this.

This generates 16 different lives. The study design asked individuals to rank two pages of eight scenarios. We excluded the logically best and worst scenarios so that two scenarios could be placed on both pages. This enabled us to infer a ranking for all possible lives.

From these data, we run a rank-ordered logistic regression model..."

There is a lot to be said about this study, and I hope to get around to talking about at least some of my many questions/criticisms, but the most immediate, stunning thing to me is this

What differentiates this "new" methodology from conjoint analysis, which has has been a mainstream marketing research technique for decades, other than the fact that it is more poorly thought out and executed?

Wikipedia summarizes: "The objective of conjoint analysis is to determine what combination of a limited number of attributes is most influential on respondent choice or decision making. A controlled set of potential products or services is shown to respondents and by analyzing how they make preferences between these products, the implicit valuation of the individual elements making up the product or service can be determined. These implicit valuations (utilities or part-worths) can be used to create market models that estimate market share, revenue and even profitability of new designs.

If these researchers are not actually living in an intellectual bubble, from which they have no knowledge of what is going on in their fellow social science fields since 1971, I would have expected them to explain how this model varies from the well-known and popular method of conjoint analysis (aka discrete choice modeling). It appears to me that they actually think they are introducing a brand new idea to the world.

Let's take a moment to think about this study. Just based on the following levels of the four attributes included in the survey, which variables do you expect people would be shown to value more highly? Note that not only are the researchers choosing the levels of the attributes, but they are also operationalizing potentially complex concepts - they are assigning a particular definition to general concepts like "health" and "happiness" that can have a lot of dimensions. Remember that the students are looking at various combinations of these attributes to determine which of the two possible "lives" they would rather have.

- Life expectancy: 65 or 75

- Health: able to move around freely or hard to move

- Happiness: 95 percent or 80 percent of the time in a good mood

- Income: $300,000 or $100,000

I look at this and I think:

OK. The difference between 65 and 75 is pretty big, but that's something that affects a person way down the line, so you would expect people to discount it such that it would have a smaller impact on their immediate decision. (This is standard economic discounting.)

The difference between being able to move around freely and finding it hard to move around (i.e. being crippled in some fashion) is very large and will impact a person's life in a big way. I expect people, especially young people who are likely to be completely mobile right now, will place a lot of importance on "health" defined in this manner because the idea of taking on that life will be scary and bad. (In general, research finds that able-bodied people assume that becoming disabled will diminish their quality of life more than it actually does.)

So happiness means being in a good mood? OK, I will go with that for now. The idea of being in a good mood 95% of the time sounds amazing and really desirable. 80% is also pretty great, but 95% is incredible (more on this later). People will want this.

$100,000 is a lot of money. $300,000 is even more money. Even if these students are in a profession in which $100,000 is a reasonable expected income (law?), the diminishing marginal utility of money means that $300,000 won't seem that much better than $100,000 a year.

My prediction is that they are all important, but as defined by the researchers:

Life expectancy: low importance

Health: high importance

Happiness: moderate to high importance

Income: low importance

The results? (The higher the number, the more important the attribute was.)

"...all the coefficients are significant at the 1% level, which means that the differences in the levels of these dimensions were all seen as being important to respondents."

Income 0.777
Health 3.109
Happiness 1.672
Life Exp 0.684

The researchers acknowledge that "the size of the coefficients on the four dimensions is sensitive to the high-low endpoints for health, happiness, income, and life expectancy used to generate the 16 different lives."

Some issues I have with this study (which, they admit is "exploratory" in nature, but that doesn't give a researcher license to not do a literature review, ignore theory, or be generally sloppy):

1. The researchers conclude: "Our preliminary results suggest that happiness is not the sole determinant of utility." The results do nothing of the sort. The results suggest that good mood is not the sole determinant of utility.

This should not be surprising to anyone who is a human being or has ever come into contact with a human being.

It seems clear to me that people do not want only happiness (narrowly defined) out of life. Thought experiment: Given the choice to go into a cage and have your brain wired up to a machine that can deliver doses of pleasure directly to the pleasure center of your brain with a mere push of a button, would you do it? Do you know anyone who you think would?

2. The inclusion of "good mood" in this model is kind of strange because to the degree mood is variable, it itself could be a function of health and income. (I'm not saying that it is, only that it could be.) Although regression is good at handling correlated variables, it still strikes me as odd that the researchers do not discuss whether they are viewing happiness as endogenous or exogenous.

3. As I understand it, other happiness research (and anecdotal evidence) suggests that people have what I will call a happiness "set point" from which they do not wildly vary. This isn't to say that individual people's moods are stable over time, but that some tend to be happier than others overall, and this happiness is not predictable based on the objective qualities of their lives. Some people who are poor, unhealthy, with little social support, etc., feel happier than other people who are wealthy, healthy, and have wonderful and supportive families.

So as a subject in this sort of experiment, how do you make sense of the fact that your level of "good mood" differs from one scenario to another? A related issue is how realistic do subjects find the various scenarios.

It is much easier to see how events in the world could cause my income to increase or decrease, my life expectancy to take differing values, or me to become disabled, with all other factors of my life being the same. (e.g. inherit money or get a plum job; get cancer or another terminal illness; get in an accident that puts me in a wheelchair.) But what would it mean for me to be in a good mood 80% of the time versus 95% of the time that does not itself rely on other things in my life changing? To make sense of the scenarios, do I have to smuggle in other variables they aren't talking about?

For example, what does a person think when presented with a scenario in which they have lower life expectancy, lower income, and are disabled, but are happy 95% of the time? (Especially if their current level of happiness is well below that.) Do they assume "something about me changes so that I am just in a better mood" or "I inherited money so that I am living on a lower but still high income without having to have a job" or "I married the girl/boy of my dreams" or what?

4. If I am in a "good mood" 80% or 95% of the time, what is my mood the rest of the time? Am I in a good mood 80% and okay mood 20%, or good mood 95% and deep black depression 5% of the time? Robert asks, "Is any of the good mood euphoria?"

5. I am really curious where they came up with their levels of the attributes. They explained that they chose "realistic" values for the income levels, so presumably they did reality-check those figures, but wouldn't it have been better to have a lower-than-expected income as one of the options? The options for income versus options for "health" were particularly unmatched in my opinion. Income was "Good" versus "Great" while health was "Good" versus "Terrible."

Apparently it's possible in the experimenters' universe for a young person to suddenly become paralyzed but not to flunk out of law school (or flunk the bar exam) and be forced into a job paying $40,000. And seriously, isn't the actual likelihood of a law student not ultimately becoming a lawyer much higher than becoming disabled at a young age? (Yes, this is an empirical question.)

And I wonder where the 80% and 95% good mood numbers came from. Is this based on other happiness research showing that these are typical numbers? It seems remiss that the paper didn't say anything about this. Anyway, do people even have a good sense of what percentage of the time they are in a good mood now, to provide some basis of comparison? How would somebody's overall estimate of their good mood percentage compare to a percentage derived using the DRM (Day Reconstruction Method) or some other approach.


To the degree that the numbers selected were arbitrary and/or unbalanced, the results are meaningless. I think I could come up with levels for these four attributes to generate any outcome I please.