Trust the Math, not the ForceApril 11th, 2010 by Eric Kesselman in General Guidance, Theoretical
So Chris Liss (coupled with an annoying refusal to give me Alberto Callaspo) has been saying a number of things I want to respond to. The debate begins in my last post 'Intuition v Quants' and continues in the comments area, and then on to Chris' post on Rotosynthesis.
Chris has a number of objections, but the one I'd first like to tackle is his complaint that baseball is not a math game, and that since a baseball season is "organic", or non-repeatable (2010 only happens once) all of our projections will be 'wrong'.
I think its hard to argue that baseball is not a math game. We count statistics for all sorts of outcomes, and can say some pretty meaningful things about them. For example, I can tell you that if an average batter is facing a 1-2 count instead of a 2-1 count he is about 15% more likely to get a hit on the next pitch. I can tell you if a runner is on 2nd base with no outs, this is worth on average a little over 1 run. These are mathematical statements with quite a lot of probative value. In the sense that we are counting things, and can make probabilistic predictions about the future with a fair amount of accuracy its hard to argue that baseball is not a math game. This is true even though no math, or even much thinking necessarily is involved in the actions themselves. Math is a way of describing and predicting the game, its inputs, and its outcomes. Baseball (and I would argue pretty much all of life) is a math game in this sense.
I don't think Chris can argue that there are all sorts of situations in baseball (and fantasy) where I can say mathematically meaningful things like this. To whatever extent I can, these statements have real value.
What I think is really bothering Chris is the lack of complete information. He craves certainty and exactness. We will never know the real 2009 Derek Jeter, or his real baseline performance. Did he overperform expectations in 2009? Or did we underestimate his baseline performance? We can never know for sure. Worse, as Chris notes, baselines are often changing.
Chris seems to take from all this that because we can never know for certain, all projections are "wrong." This isn't true, and isn't really the point. The question is how useful they can be. Sure, I won't be able to tell you exactly what Derek Jerek will do in 2010, any more than I can tell you precisely whether the batter with the 2-1 or 1-2 count will get a hit on the next pitch. However, that doesn't mean I can't say some meaningful mathematical things about his likely outcomes. That is all a projection is. To the extent that I can tell you with a high probability what a player will do, that knowledge has value. For Chris though the lack of complete accuracy seems to mean the complete lack of value. Not true. To the extent that I have better projections than the field, I have an advantage on the field. I have more accurate information. I don't need perfect information to gain a big edge. Accurate projections aren't completely 'right', they're just a bit more useful than some others. Chris seems to suggest that since we can't know anything for sure, everyone is equally wrong except for those few people who miraculously happen to get it exactly right. Not true.
Chris also tends to note that these seasons are not repeatable and therefore that probabilistic forecasting isn't appropriate. It's really just another way of saying 'we can never know for sure if we were right.' Clearly probabilities of any given thing don't change depending on whether or not you can or do repeat them exactly in the future. Chris is just hung up on the fact that we can never point to something in particular and say 'this had exactly a 53% chance of happening." It's too inexact, too filled with variance, and uncertainty for him. In actuality, the better way to approach the issue is just to ask whether or not you can say something useful about the predicted outcome. I suspect Chris has a hard time saying that you can't. If you want to quibble over how useful or accurate a model or a projection is, fine. But clearly suggesting that all forecasts are "wrong" is not the way to get anywhere insightful.
All Bill has been claiming is: If we do agree on reasonable projections (and on most players we more or less generally do), but other owners can't price their expectations accurately- that costs them. Bill also likes to say 'If you guys aren't pricing Paul Konerko right by now, something is seriously wrong with you." Its hard to argue with this. If we more or less agree on who the player is, but cant agree if he's a 10$, 13$, 17$ or 19$ player someone is just flat out wrong. Being wrong costs you edge.
If our projections are meaningful (they are, subject of another article) then the model's move from projections to accurate values is also valuable. To the extent we agree on who a player is, the people with good models will price him better.