Guesstimation

Alethiography has a post about estimating vs. guessing today and why it's better to perform some kind of structured estimation process, even if you're mostly making things up, than to simply take a wild guess.

I've often needed to come up with guesstimates for things like: how many man-hours will it take to enter and check data for 25 quarterly reports? Given the number of workers who will be at my disposal, when can I promise the client that the reports will be ready? One time I had my most average employee enter old data for one section of a spreadsheet and my second most average employee review his work so I had some clue as to the time involved. Once you have established some part of the process, extrapolation becomes your friend.

Last month, I was working on a survey for someone who will be having a crew of people interviewing users of three different lakes and she wanted to know how many surveys she needed to collect. That was easy to tell her without even looking up a formula - 400 completed surveys from the target population at each location, if she wants +/- 5% sampling error for each lake - and she said OK. But I decided I'd better sit down and do the math on this, taking into account the survey length, the number of interviewers available, the hours of the day they would be working, the proportion of lake users she could expect to be in her target age group, the response rate she could expect, and so forth, and it quickly became apparent to me that she didn't have the resources lined up to actually accomplish 400 surveys at each location in one day; she is looking at something closer to 200 per location. Now her dilemma is whether to try to find more interviewers, do surveys an extra day, or give up on wanting to look at each location separately.

Not everything is as neat as these examples, of course. One question that really stymied me this week: I got a call from a guy wanting to know much I thought it would cost him to get a web survey administered and data analyzed, given that each respondent (a bunch of fish experts that he said he already had contact info on, eliminating the need to estimate sample costs) would answer a series of questions about 0 - 187 different fish species, depending on how many species they claimed familiarity with. (And no, he didn't have a clue the average number of fish an expert will be familiar with; that was part of the reason he needs to do the survey. Pity.) After a bit of discussion, it came down to the fact that it would be enough for him to know whether $10,000 in grant funding would cover the costs. To that, I felt confident in saying yes. Every way I thought of estimating this, I came up with something under that amount. (If it comes to it, I can take an extended vacation, purchase the SurveySelect software, and do it myself for $10,000. Well, I guess I technically can't do that and stay within the law.)

One odd experience is when you come up with a really close estimate of something through a process that is way off in almost every particular but in which the errors of judgment mostly cancel themselves out. The hell is when you have way overestimated the amount of time you will need early on in a project and let that make you think you are sitting so pretty you can start making extravagant promises about delivering early, sending people home to save costs, and other such folly. It's like a rule of the universe that the gains you make early in the process will be undermined by some stupid thing at the end that you never saw coming... like getting 40,000 paper surveys that have all come from the printer folded with the address on the inside of the survey rather than on the outside for a self-mailer... or getting 25,000 survey forms for a follow-up mailing in which you need your workers to pick through the stacks to manually remove the ones with IDs that have already come back from your previous mailing, but the printer sent them to you in zip code rather than ID order. Oh yes, the last-minute disasters that I have experienced. I don't even want to think about what is going to go wrong at the first Houston fishing event next weekend. (It is an open question in my mind whether the bilingual interviewers from the survey firm I've hired will actually outnumber the families who show up for the event. I guess if they get bored, they can interview each other. "Oh please, can I be the illegal immigrant with 6 kids who hates the government this time?" "Only if I can be the very confused person who is here on the wrong weekend for a family carne asada.")