Responding to Liss, Part 2
April 11th, 2010 by Bill Phipps in TheoreticalA little while back, I made some remarks to the effect that:
1) Universally, the game of FBB is not played very well
2) A quantitative approach, coupled a healthy dose of game theory, could dramatically elevate the level of play
3) Anyone who is not approaching the game from a quantitative perspective is placing themselves at a competitive disadvantage
This is hardly earthshaking stuff, and I never suspected that I was saying anything controversial. But not so fast. Here comes Chris Liss’ take on the subject:
“But more importantly, I wonder whether quants can ever make serious inroads in a complex game like baseball. Backgammon, chess, poker sure – those are math games. But baseball is like the weather or the stock market – organic, unpredictable. Didn't the quants nearly take down the entire financial system a couple years ago because the housing market dropped more than they conceived it could? Is this kind of analysis really optimal when it comes to things like baseball players, stocks and weather patterns? Or is it much more useful with games where there are less unknowns like poker or chess?”
WOW. With one paragraph, Liss manages to be dismissive of the entire sabermetric community, three of my favorite games, global warming research, and the smartest guys on Wall Street. Impressive. Where to start…..
Baseball not a math game? Tom Tango, Fangraphs, Baseball Prospectus and Bill James might as well all close up shop. Sorry guys, you’re out of a job.
Baseball not a math game ?? Somebody should tell Billy Beane to forget Moneyball and rush right out for a copy of Liss’ new book “Geniusball- How to Use Your Gut in an Unpredictable World”.
Baseball not a math game ??? We all know this to be absurd. Nowhere in American life, not even in our media obsession with the political poll, are statistics more on display than in the game of baseball. It is a perfect pairing. If you google the word baseball, there are 119,000,000 hits. Pair it with statistics in your search and you get 39,000,000 choices. Math is an indelible part of baseball. With its one on one confrontations and bookkeeping friendly pace, the game is a treasure trove of statistics. It is part of what sets the sport apart, and accounts for much of our love for the game.
Liss is right when he suggests that baseball is not a game of complete information, like chess. It IS like the weather in that we cannot precisely predict if it will rain tomorrow. But that is the very nature of statistics and probabilistic thinking. No matter how much data I gather about the weather tomorrow, I will always be missing something no matter what.. Therefore, the best I can do is to give a statistical projection for the weather tomorrow. This is why weather reports are given to us as percentage chances of rain.
I would ask Liss this, if math is not the optimal approach to studying baseball, what approach would he rather use? Math and the scientific method have given us the ability to decode the human genome, understand the evolution of life on the planet, harness the power of the atom and peer 10 billion years into the past with the Hubble Telescope. Of course it is up to the task of analyzing baseball.
I work a Wall Street job, trading options for a major firm. The use of statistical pricing models is an integral part of what I do. They are a vital tool, without which I could not compete. However, equally important to my job is knowing the limits of these models. Success in my field involves a pairing of human intelligence with a concrete mathematical foundation. I am not suggesting that there is no room in FBB for human insight. Far from it. What I am saying is that the game currently lacks the type of strong mathematical framework that allows us to focus on the areas where human insight excels.
Imagine if we created a league today, using the statistics from 2009. We would be turning FBB into a game of complete information. How do you suppose the auction would go? Let’s say that Elvis Andrus was the first player brought in. We know that he will have 6 hr, score 72 runs, hit 40 rbi, steal 33 and bat .267 in 480 at bats. What is the correct bid for him? If I asked 10 different experts, how many unique answers would I get? Several, I suspect, and this is an example of why I say that the game is not well understood. I don’t think the fantasy baseball community is able to answer this question. Certainly in poker, if the game was played with the cards face up, many professionals would know how to play near perfectly.
Now, imagine we are back in 2010 and the statistics are not known. I currently own Elvis Andrus. What do I do if Eric calls and offers me Jose Lopez straight up for him. Let’s say that I project both players to have seasons very similar to their 2009. Forget for now where my projection comes from, be it CHONE, Chris Liss, or some bodily cavity. Suffice it to say that projection represents my very best guess for their 2010. Should I do the trade? Wouldn’t the decision be much easier if I knew what their 2009 seasons were worth?
This is where things get more interesting. Because, the complicated truth is, Andrus and Lopez value in 2009 varies from league to league. In some leagues, Andrus was worth more. In others, Lopez. What is important is, how did the team statistics distribute in the league? Lopez gets his intrinsic value from his power. Andrus, from steals and scoring runs. If in 2009, the power categories in your league were tightly clumped, owning Lopez was worth more. His power helped move you up further in the standings. If, on the other hand, steals were a tightly contested category, Andrus and his 33 swipes might well have been worth more.
On the surface, it might seem that we are back where we started. I can imagine Chris Liss triumphantly saying “I told you so. We can’t say for certain which player is more valuable! “ Fortunately, this is where we can turn to statistics to help us answer the question. We can measure how all of the categories in our league are currently distributing. We can explore how they are expected to disperse if nobody changes their roster. We can also do a sampling of how leagues historically distribute. If we do this, we are likely to find that the HR category is the historically tighter of the two. We can see a range of expected values for Andrus and a range for Lopez. When we put all of these results together, they help to paint a picture of what we should do.
Maybe there will be other factors to consider. There almost always are. Andrus had a cortisone shot in his hand, Lopez is the more likely of the two to get traded, Lopez is about to get third base eligibility, Chad has a burning love for Andrus and I may be able to get more by trading with him etc etc etc. In varying degrees, quantitative analysis can help us weigh each of these factors. And, to the extent that we may not be able to quantify a factor, this is where human intuition gets its chance to shine.
What the statistical modeling has done is provide us a framework. Knowing how to solve the game with complete information gives us some firm earth from which to make decisions in the face of the complexities of the unknown.
It was never my intention to suggest that all that was needed for FBB success was a mathematical model and a set of projections. The game is much too involved for just that. However, I do think that a strong mathematical model is a powerful tool that would greatly enhance even an expert's game. Brian Hastings is a very gifted poker player. His talent alone is enough to guarantee that he will be a winning player. But there is no doubt that the use of hand histories, Poker tracker and starting hand calculators make him all the tougher.
Tags: brian hastings, Chris Liss, genius, intuition, quantitative analysis, Theoretical
Loading ...
Baseball not a math game ?? Somebody should tell Billy Beane to forget Moneyball and rush right out for a copy of Liss’ new book “Geniusball- How to Use Your Gut in an Unpredictable World”.
Bill, you're quoting something I didn't actually say: "Baseball is not a math game." I contrasted it with pure math games like chess or backgammon – whether all the information is known. So you're arguing against a straw man – someone who denies any role for math in baseball or science in predicting weather patterns.
Obviously, that's just ridiculous. And I shouldn't even have to waste time refuting that. Bill James, Ron Shandler – those guys are great. Have done wonders for advancing our understanding of the game. But ask Bill James or Ron Shandler about the accuracy of their projections sometime – read what they themselves have written about them, and you'll see they don't think they're even close to getting the numbers right. It's just not possible with any kind of precision whatsoever. If it were, then maybe your model would confer some kind of serious advantage. Read my post on RotoSynthesis – I conceded that it does under that condition. (Same with your 2009 stats example).
But unlike poker or backgammon – games where there are no unknowns, that condition is very unlikely to obtain. Use Shandler, Bill James, CHONE, an average between all three – I don't really care. They'll be wrong, and there's no way to properly test them for reasons I've exhaustively explained. Even if you measure league-wide projections over time, and get an accuracy ranking, it still won't help you assemble your individual team. BP, for example, undercounts everyone's stats for the realistic possibility of injury. Since every player has a 10-15 percent chance of injury, it deducts that amount of stats across the board. As a result, its totals will grade better for total homers hit in the league. But we know that injuries won't be distributed that way – some players will miss the whole year, others just a couple games. So the numbers are too low for almost everyone and too high for those few who miss a lot of time. But we all know anyone can get hit by a bus. What we want are the numbers for a player if he's mostly healthy. So again – even if the numbers are right in the aggregate, they can be wrong for purposes of assembling a team.
I feel like I've made a fair representation of your arguments and done my best to refute them, but that you've distorted mine. I'm not offended by this, but I do think it makes for a duller debate – turning your opponent's views into an absurd caricature isn't going to shed clarity or light.
Finally, the smartest guys on Wall Street really did nearly destroy the system, the sabermetric community was really good at isolating what's skill and what's luck, (which thanks to them anyone serious about this game now knows), but is not good at predicting skill growth or regression – the key to winning in fantasy baseball – and the weather forecasts are so often inaccurate, I wonder whether some tribesman or farmer who really needs to know whether it will rain wouldn't outperform the weatherman easily if they were to have a contest.
I do not wonder whether some savant could be a world class poker player without knowing whether AA is a better starting hand that J-10s.
I agree we want a full, and accurate debate but I also took your quote above as essentially saying baseball is not a math game. I think its a bit unfair to lay the blame here on Bill. Read it again, you sure seem to be contrasting baseball with 'math games'. Anyway, now that we're all on the same page that baseball is a math game and meaningful things can be said about it I think the debate next has to move to how useful the projections are? Do you agree?
I totally agree there's a lot of bad, and 'safe' projections out there. The question really is whether we can have some meaningful idea of a player's outcome next season and price it accordingly.
I'll add one more thing – and it's the core of why your method will never make you an expert fantasy player (though perhaps you have other skills that might bridge the gap – such as better instincts or knowledge than you give yourself credit for) – quite simply – you overvalue the known. You "best guess" projections are important to you because it's all you have to go on. So you build a whole edifice based on that, and even though you know your foundation is shaky, you cling to it because it's the best you believe anyone can do.
But when you realize the vast unknown that exists in a complex enterprise like baseball or the stock market, then you realize how shaky these models really are. And the less one is inclined to cling to this small measure of knowledge and the more open one is to various possibilities – both likely and unlikely. In that way, one is able to move quickly and adjust to the unforeseen and be there as things are changing unpredictable. Or at least be there first.
You can choose to believe me, or not – but let's play out this league for a few years and see what happens over time. I think I know where you're coming from, but you admittedly do not see my point of view. I hope to bring you around eventually.
Right, Eric – but clearly that quote is totally out of context and distorts my obvious point that it's not a *pure* math game like ones with a deck of cards. I said that certain games were math games, but baseball was organic – meaning there was an unpredictable element to it. As I said, I'm not offended, but I don't think equating what I wrote to dismissing the work of Billy Beane or Bill James was remotely a fair reading.
I have no problem clarifying your position, but everyone I know who has read what you wrote took it to mean the way both Bill and I interpreted it. I think you've made a number of mathematically dubious claims (and certainly implications), and I think starting here was the reasonable choice. Anyway, I'm happy to move on and flesh out where we agree and where the actual points of dispute are.
I do agree you've generally done a fair job on characterizing our arguments, although I think you've built a bit of a straw man yourself by suggesting that all the quants can do is read off their spreadsheet. All that has been claimed is that these methods are valuable tools, which can be combined with whatever other methods you choose to improve your game.
I think the Lopez/Andrus example Bill gave is a good one where it can be very hard to make a good decision with no quantitative help. What are your thoughts there?
I'm not sure what's mathematically dubious – I don't think anyone's refuted my claim that it's impossible to know to what extent Derek Jeter's market beating 2009 was due to variance and what extent it was due to the market being wrong, for example. And that we'll never know because there will never be another 2009 Derek Jeter to run a million simulations with. Were there other ones?
But in any event, Bill's Andrus/Lopez example is one that any experienced player would know how to handle. If you looked at the categories in your league, you could see whether steals or power was more sensitive. You couldn't know how things would change going forward – (a trade in August is much easier to make than one in May, for example), and while home runs sometimes cluster together more in a given year, I've been in leagues where steals are very tight, too. I suppose you could study your league with it's distribution of teams and likelihood of trading and also study cluster tendencies for the different categories across leagues based on their uneven distribution among players – saves for example are interesting because only 30 players can really get them at any one time. All that is worthwhile, and I love reading those studies and integrating them into my model (the one in my brain). But I feel pretty comfortable either pulling the trigger or turning down a trade like that based on my overall assessment of the situation. If someone built an automated model that took into account enough of the relevant factors – historical data spanning enough leagues to be of use in terms of category clustering, I'd be curious to see what it said, but would still make my own decision independently.
And to the extent I've mischaracterized anyone's argument, I apologize. I don't mean to imply that Bill or any of you are necessarily going to use your projections robotically, or that you might not be very good at this. In fact, I think Bill probably underestimates his baseball knowledge skills, and that anyone who plays poker or trades options at the highest level can easily become a very good fantasy baseball player – perhaps even better than the experts. My point is that I find it implausible that it's going to be because of an advantage converting projections to dollar values.
It's funny, I read Chris's "baseball is not math" statement as an expression of available information, not a negation of all the math that describes baseball.
I think the key issue here is one of degree: Chris's foreign language translation metaphor in his rotosynthesis post seems most apt. Chris knows the numbers, the projections, the implications of price spreads and category excess and/or limitation, and so plays his auction against the room on the fly. It isn't that all that information isn't there, it's just that he doesn't see a virtue in pinning it down to one projection or one price. That doesn't mean there isn't information and assessment backing up the decisions.
Bill, on the other hand, thinks there is one price for Elvis Andrus as the first pick of a retrospective 2009 draft, because there is perfect information. This is the ideal situation for analysis, you would think, because we can all agree on the situation and the stats for the year. But there is most definitely not a single Andrus price, because a retrospective draft is rife with gamesmanship.
I know this because I ran a number of retrospective drafts in the late 90s. The original goal was to figure out whether the 65/35 hitting pitching ratio would hold if we knew how pitchers performed, but what I found out was much more important. BTW, I asked Shandler to participate at the time and he didn't see the point, saying that everyone would end up in a tie.
But what really happened, the first time (when a handful us auctioned and used lists for the others), was that it became clear that there was no systematic price list you could use to price a team's auction, because small variations in the pricing of saves (to take the most obvious example) would lead to a team buying all the closers or none of them. Clearly, once you bought a $20 closer, you didn't want another one and so ignored them, even if you had higher prices for them than the other teams.
Recognizing this limitation to lists, we did some more retros using all human entrants, who, recognizing that the spread in value would be very small (Ron's problem), adapted all sorts of dumping strategies in order to make the results less symmetrical. And, as we could have predicted if we'd thought about it better, dumping two categories (thereby altering all the values) was a winning strategy, as it always is in balanced leagues.
It would take many more iterations to find the balance point in this exercise, but there seemed to be little point. The message is that pricing throughout the auction is dynamic, based on what your team has and what other teams have. And the better information everyone has, the more the edge goes to the person who adapts best on the fly. (If two teams are dumping two cats, which one recognizes this earliest, and which one escapes it most effectively, for example).
For the record, I make projections and take it very seriously (I scored very well in two of the three trials Tom Tango ran in last year's forecasters challenge), but since I'm quoted all through Shandler's essay about the myth of forecasting accuracy, I'm very aware of the limitations. I also go into auctions with a price list, not because I don't know the pool or need translations, but they help me see whether the league is running ahead or behind of budgets. I don't buy every player who comes in under and I sometimes go over for a player, depending on all the usual factors.
I would ask Liss this, if math is not the optimal approach to studying baseball, what approach would he rather use? Math and the scientific method have given us the ability to decode the human genome, understand the evolution of life on the planet, harness the power of the atom and peer 10 billion years into the past with the Hubble Telescope. Of course it is up to the task of analyzing baseball.
I should leave this alone, but I can't resist. Math and science are good and useful, point taken. But these are all examples of backward looking analysis – using them to interpret the past. Give me an example of where math and science have helped us predict the future. Even with global warming research, they keep revising their projections up unfortunately – "the polar ice caps are melting faster than projected". They're looking at what happened over the last half century and guessing going forward. They don't know with much precision where it's going – only that if it continues at this rate, then we're all fucked. It seems like most of the predictive capacity of these models is conditional – if interest rates do x, then I'd expect y to happen. But has anyone build a model that can tell me which stocks to buy? Is there a big blue that can beat Buffett?
As for Peter's point, I think it's a good one and that I overlooked in making my case – the value of player's changes as the auction goes, even with known values. (Though the RW draft software does have a model that takes in-auction inflation into account). But even if you factor that in, it's still nearly impossible to optimize because timing is everything. Let's say the first two catchers are bargains, and you grab them, but other ones are even bigger bargains, and you're full. Or let's say everyone decides to punt saves, so you have a team full of closers. Moreover, one can always trade, but there's always a vig to be paid if you need the trade more than the other team. That transaction cost can easily outweigh the supposed benefit for having more perfect dollar values.
One of the most interesting dynamics of our Cardrunners' league auction were the prices for catchers, which were on another scale than for all other players. This has to be some sort of position scarcity adjustment that is amplified because of the 10 team (versus the usual 12 team league, which has minimal catcher inflation) format of the league, though this effect was far greater than I anticipated. What was interesting was that in general the so-called experts bought the cheap catchers, the poker players bought the pricy ones.
Does this mean that the so-called experts got it wrong? Or did the action guys overextend due to a radically different analysis, or maybe from the habits of mixed league play, or something else? I was fortunate, because of the money that was left when Wieters came out, to end up with a foot in both camps, but I'm sure that how this breaks, one way or the other, is going to play a big role in the final standings.