Getting Better and Worse at the Same Time

Friday was the day that, supplied with contact names from yesterday's meeting, I started calling people working in the programs to find out what the hell was going on.  I started calling them up around 8:15, and it's really convenient how leaving a message that starts "[Your boss's boss] asked that I call you about..." gets a prompt response.

Unfortunately, that response was generally: "What?  You say that is happening?  Let me look this up in [the front end of our database].  Huh.  I see what you mean.  But you can't find these records?  That's weird."  So people were eager to help but unable to supply much information.  But I did get a sort of consensus view that I should talk to M, a person in our IT department who is responsible for the database stuff.

So I know from previous encounters that when I talk to M, he will want me to supply the identifying number of a person in the database where we're seeing the problem.  I found a problematic person (no X data) for Program Y, and I decided that before talking to him, I would also find a non-problematic person (have X data) for the same program.  But when I looked up the "non-problematic" person, I found out that even though I had some X data on this person, the data were incomplete!  The front end of the database reported 4 units for this person, but only two of them were in the data download.  This meant that I had to go back through and spot check individual people's records in the database for all the programs to ensure that this wasn't happening for other programs too.  And I found another program with the same situation -- some but not all the data were there.  (For all I know, the problem could exist in other programs but I just didn't happen to find a record with the issue.  If I could identify the problem records without looking up individual people....well, I wouldn't have a problem then.)

That was one way I discovered that I had more missing data than I knew.

The second way was when I did some basic crosstabs on the X data I had matched up to the records supplied to us, I found oddities like, This person has data in period Z for a program that did not occur in period Z, WTF?  Oh.  Damn.  When the regulatory people had given us their records, they had recoded a lot of specific program codes under one code.  So when I aggregated X data across those codes, it was summing all X data across all programs under the umbrella code.  If a person participated in two different programs, the data were screwed up.

I explained the issue to my officemate and she said, You mentioned a couple days ago you almost wished you were starting over knowing what you know now, and it looks like you're getting the chance.

It's true -- I did say that because I had done a bazillion different "get some data, clean it up, match records to this primary datafile, see what happened, rinse, repeat" cycles and everything was confusing.

So I did start over, and it was a much more clean, satisfying procedure.  And it fixed all those weird mis-matched records so everything looked kosher on that front.  But now I had 2700 records without data instead of 2300!

At the end of the day, I told my officemate that I needed to stop working on this and go home before we lost all our data altogether.

Summary:  The data are getting worse because we have ever more missing records but the data are getting better because they are more accurate.  But damn.

Pattern Challenge Day 22: Polka Dots -- Friday, 1/30/15

This outfit resulted from a last minute substitution.  I had planned to wear my argyle cardigan with it, but when I put that on, it was short and small enough that it looked very weird with the tunic.  The tunic itself was a substitution for a short-sleeved knit top that I had planned with the outfit when I was brainstorming ideas during the fall.  But luckily I have a polka dot blazer that could easily be substituted, fulfilling a different pattern challenge.

I normally would not have paired this necklace with this blazer (it worked better with the cardigan, I think), but I didn't feel like trying to make yet another substitution so I just went with it.

*Plum tunic (Kohls)
Cream polka dot blazer (thrifted, Target)--I love this classic school 
Black ankle pants (thrifted, Eileen Fisher)
Tall black boots by Fitzwell
Purple/gold necklace (Macy's)

This photo is somewhat better because it was a sunny afternoon and with my new awesomely short commute, I got home from work in time to take another photo before losing the daylight.