Intuition vs Quants
April 1st, 2010 by Eric Kesselman in Theoretical"How terrible is wisdom if it brings no profit to the wise?"
- The Devil, Angel Heart
So I've been having an interesting debate with Chris Liss of Rotowire.
Chris does a good job of framing the debate in his recent post here. It starts in the second section.
The debate is largely centered on the relative value of intuition vs quantitative analysis. Chris claims to largely be an intuitive player, having had such extensive experience (most of it extremely successful) that he relies on his instincts to quickly and accurately lead him through his decision processes. I am somewhat sympathetic to Chris' position, being largely an intuitive guy myself. My first instinct will very often be the right one. However, its important to check your intuition frequently, otherwise you'll never know if you start drifting and become very vulnerable to some group think problems. You also might become overly reliant on strategies that work against a particular group of opponents.
I've largely been arguing Bill's side. If you aren't quantitative you must be exploitable. While your intuition might be very good, it will occasionally be wrong. If you aren't doing the math, every time you are wrong costs you something.
Now the intuitive guy can make up edge in all sorts of places elsewhere. He can be better at player projections. He can be better at trading. He can be better at scouting and grabbing timely guys out of the claim pool. But in the auction itself, to the extent that there IS a right answer to what a guy is worth- the intuitive player is going to be hard put to ever beat a good quant with an accurate model. To the extent that being a 'genius' just means you have better player projections than the world, that's great and a huge source of edge. To the extent that it means you aren't being quantitative, that's going to cost you.
To frame the problem more precisely I gave Chris a question Bill likes to embarrass me with. Imagine you knew that a player, let's call him Skip Kazmir, was going to hit exactly 20 HRs, 100 RBIs, 90 runs, seven steals and a .313 batting average in 554 at bats next season. What would you value Skip at in auction dollars in a 5×5 mixed 12 team league?1 The point of the question is this- if with perfect knowledge of Skip's performance there is a wide disagreement over his value, how can we pretend to expertly price Skip in an environment where instead of perfect information I only have a probability distribution for his likely performance?
Certainly the experts are better at projecting players, but if they don't precisely know the value of the end results of their projections what good is the more accurate information? Poker analogy- Imagine being so good you can precisely know whether your opponent has AQ, AK, or 88 but you don't know how those hands match up against your own pocket jacks.
Also, what happens to the intuitive competitor who doesn't have Chris' genius type differences with the consensus projections?
I'm a huge believer in the power of intuition. It's pretty amazing how accurately and how quickly we can make extremely complicated decisions. However, in the context of fantasy analysis I don't see how even a great intuition can keep up with a great model.
1 This question is a bit like saying 'imagine you were dealt pocket jacks next hand, how would you value them?' You clearly want to know a whole host of other information like stack sizes, position, action in front of you, etc etc. It's really not the details of the question thats important though, so let's just assume any other details you might want to know are also specified.
Tags: Bill Phipps, Chris Liss, debate, Eric Kesselman, quantitative analysis, rotowire
Loading ...
I think you and Bill are assuming everyone MUST come up with a set of projections for the player pool and translate it into dollars. It seems to be your argument is: let's say Chris has great projections, but loses 3 percent accuracy in translating them into dollars, while Bill has slightly worse projections but perfectly translates them into dollars. Under that framework, all Bill has to do is boost his projection accuracy within 3 percent of mine to beat Chris.
But what if I don't even bother to come up with projections, period? After all a projection – a fixed number that represents the median expectation of a player's infinitely many possible seasons – is merely a guess. It's a guess both in terms of what that 50th percentile mark is and whether 50th percentile is even the way to value a particular player at a particular stage of the auction. For example, late in a 12-team mixed league where the replacement level is high, the 50th percentile numbers for a player are worthless. It's only his 85th-ish percentile numbers that matter.
Moreover, I don't even bother to do projections in preparation for my auctions. I merely research all the facts about every relevant player (past performance, playing time, team context, age, health, historical context, etc.) and put him in a rough order on a list. That way I know everything I need to about him and have him sorted by position and in the neighborhood of other players who I think have similar value. Then I watch how the auction goes (and also draw on past experience of auctions) to see where I find value. I don't have a fixed amount I'll spend per player necessarily, though often I'll have a loose idea. If I'm targeting a specific player, I'll sometimes go higher than what my loose idea of his max was, but I can also be pushed off if I feel it's exorbitant.
I don't think anyone's built a model for this that's better than the database in my brain for all of these factors. I do think if we all started from the same set of projections, converted them rigidly into dollar values and drafted with strict adherence to those converted numbers, Bill might have an edge – assuming that my possible "errors" in conversion didn't actually steer me through pure luck into correct decisions because our shared projections were off.
A model that simply converts numbers into dollars accurately doesn't do nearly enough in my opinion. Because in my experience that's not really where the rubber meets the road when it comes to winning fantasy baseball leagues.
We are evaluating players who have value to us because they provide us with HRs, RBis, SBs, etc etc. They have no other value to us. So i'm not sure what you mean when you say you don't have to have a projection for these numbers. It seems (intuitively or not) you must have an opinion of what these guys are going to do before you try to price that production. Whether you want to use a 50th percentile projection, or an 85th percentile projection (or a mixed approach based on player and how deep in the draft) you must have some idea before you move into the pricing. Otherwise how can you come up with a price?
It seems like the only other rational system where you could price guys while having no idea what they're going to do is to say something like 'Derek Jeter has always gone for 27$. I'll bid $27.' This system has some obvious problems.
I envision the approach to pricing fantasy guys as being a two step process. Phase 1 is evaluate the players and decide what the appropriate distribution for his performance looks like. Phase 2 is how to price that distribution.
I agree with your opening paragraph's characterization. I suspect Bill's edge comes from Phase 2 (moving from projections to prices) while your edge comes from Phase 1 (having more accurate projections). To the extent I could create a perfect fantasy owner, I suspect I'd want to take your ability to evaluate players and combine then will Bill's ability to price them.
It's true I have a vague projection, but who cares if I have Jeter for 22 steals or 25? I know it's just a guess anyway. I put him at 22-ish steals if I bother to think about it which I don't when I'm drafting. The more important thing in many ways is my overall impression of Jeter compared to the shortstop pool, not just in numbers, but in reliability, durability, etc. I'll target him as a nice piece of my overall puzzle. I'll determine how important he is based on league depth, quality of shortstops, etc. I'll do that with a few players, and then I'll take what falls based on budget and circumstances. I know the player pool well and I know the possibilities for my team at each position as the auction goes. I get a sense of which ones to go to the mat for and which ones to let go. Once in a while I make a mistake and regret a bid, but then I work around that – almost no auction is perfect. But the bottom line is I do well in these contests against what's allegedly strong competition. And I think the process of converting specific projected stats (which in my opinion are not only useless but counterproductive because they vastly oversimplify a player's value) into dollar values with extra precision is a small part of the equation.
As for Jeter always goes for $27, there's an element of that, too – but modified by the facts that have taken place since the last auction. I think the key to this game is knowing the facts and being able to interpret them. Bill might think he can just use RW's or BP's starting projections, but they're terrible. And those are two of the best! The projections are just a guess, a rough estimate based on the facts at the time they were made and updated by someone else who might or might not have looked at the evidence afresh. Even though my method seems easygoing and kind of zen, I think my prep is probably the most tedious and time-consuming of anyone's – I literally go through all the teams RotoWire Depth Chart by RotoWire Depth chart, clicking through every single player from 5th and 6th starters to middle relievers, to prospects, read their major and minor league stats, their player outlook and their latest news, look at what they did at what age, under what circumstances – even for players everyone knows. I'll look up Tim Lincecum again as if I'd never heard of him. Check his ratios and rates, etc.
I make a massive list by position as I go. I then order that list roughly in terms of value, make it fit into one page on a spread sheet and print it out. That's the only resource I take into the auction (that and my laptop to see if there's any news that broke, or to remind myself of why I have a certain prospect so high on my list).
Then I just try and pay strong attention for the five hours of the auction and trust that I'll make the right calls.
I don't think about numbers to dollars to numbers much at all. That's all going on (if at all) below the surface. I'm targeting players on my list, thinking about how to fill the empty roster slots with the best combination of upside and reliability with the money I have left.
I think we'd all agree that the kind of analysis (again intuitive or not) required has to be more precise than whether someone like Jeter is 'a nice piece of the overall puzzle." We are comparing all sorts of different assets hrs vs sbs vs strikeouts vs saves vs wins etc etc, and we need to be able to really compare them. Otherwise, it often gets very hard to intuitively answer (even in context specific situations) whether I'd rather have Mauer, Overbay, and $3 left to spend or A-Rod.
More so, it's very difficult to price extreme levels of uncertainty intuitively. The human brain is REALLY bad at recognizing the difference between a 10-1 chance and an 8-1 chance. As you noted in your rotosynthesis column you can often have players with similar price tags who are vastly different. Maybe Jack Wilson and Desmond Jennings both go for $2, but they have vastly different performance probability distributions. My guess is when dealing with thin possibilities even an intuition as honed as yours is going to have trouble. Is your intuition really going to be adequately price the chance Jennings comes up in May, vs June, vs Sept and the consequent value of his performance? Suddenly that very comfortable 22-25 sbs for Jeter kind of thinking is going to be very challenged, and since we're at the low dollar value of the spectrum even a mild mistake may result in a significant misvaluation.
I think the real bottom line here is that both Bill's model and your intuition can answer the same questions. However, those answers should be testable. If you say you'd rather have Arod to Jeter Overbay and $3, give me your projection distributions for each of these things and lets see what you wind up with. Again my suspicion is that even using your own numbers, you will SOMETIMES come up with demonstrably wrong answers.
Let me also ask you two questions.
1) How do you approach trades? Is it the same kind of thing, or do you have more quantifiable prices here?
2) What do you think of the intuitive player who is also an agnostic? Will you concede that an intuitive, agnostic owner who approaches the projections in a manner similar to Bill is at a clear auction disadvantage? Because it seems to me without having some strong and accurate genius opinions, such a player is in trouble.
This is more of a question than an argument. How do you think your model works in keeper leagues, or with hard-to-project players? For instance, how can you properly price Aroldis Chapman for this year, let alone in a league where you can freeze him at that price in 2011 and 2012? He's almost certainly an "all-feel" player.
Definitely a question for Bill, not me. I'll make sure he sees it. I'm pretty sure he will take issue with the idea that an 'all-feel' player exists. Can you define that term a bit better? It seems you are suggesting that a player has a performance probability distribution so wide, with a standard deviation so large that numbers are meaningless? I agree a player's postive value may be in his top 15% of projections, but why does that have to mean you can't quantify it?
Whats the real alternative here, we should just intuitively pick a value and those who wind up being right had 'good feel' and those who missed didn't?
You're right, I phrased that poorly.
My point is that we have so little data on a guy like Chapman, that we need to rely upon scouting and intuition more to determine our projection and valuation for him. Factors that mitigate against a successful projection for him include his age and the lack of an established conversion scale of stats accumulated in Cuban ball.
Of course, we still have to try to project what we think he's going to do, and quantify what that projection means. But the error rate on a player like him is going to be higher, no?
My point isn't to dissuade you from using the model. But I think if there's an area where you're less certain to have an advantage, it's here.
Actually, I think this is the point I've been trying to make above. I see fantasy auction as basically a two stage process. Phase 1- Come up with player projections/evaluations. All your scouting and even intuition can go in here. Phase 2- Price that information. My argument is basically that I think Bill is going to beat you guys on phase 2, therefore if you guys AREN'T beating Bill on Phase 1 you're in trouble.
Now with a guy like Chapman, I agree there is a lot of uncertainty in phase 1. But I suspect that is equally hard for both you guys and Bill. Then when you go to price that information, I argue the intuitive guys are taking another impariment that the quants are not.
In other words, if you are getting outauctioned off projections (going from numbers to dollars as Chris calls it) you guys need to find edge elsewhere to beat the quants. This can be in having superior projections, claims, trades, whatever other facets of the game exist. I'm just trying to start with the premise that you guys are starting off at a disadvantage in this one aspect of the game. Chris does not seem to want to concede that point. If we do agree on that point, I think we can move on to discussing other sources of the non-quant expert's edge. Personally, I hope it exists because I also like to believe I am competetive and I am very much not a quant.
There is so much to say on this topic, I am not sure where to start. Since I am a bit short of time I am just going to quickly address Jeff's point. The other issues I will save for later.
The model I use is a tool. It is far from perfect and certainly only as good as the inputs i give it. It doesn't tell me how good Aroldis Chapman is going to be, I have to tell it. Fortunately for me, a lot of pretty savvy people have posted their opinions about Chapman online, it is just up to me to sift through it all and come up with an educated guess. What the model CAN do is give me a very good idea of what the player i expect Chapman to be is worth in various formats.
As I have said, this model is a tool. One of the useful things I can do with it is put an assortment of projections in for Chapman and get a good feel for what his range of value is.
Let me make one last quick point about scouting before I have to run. I am getting my forecasts for players stats from some very competent sources. In order for you guys to beat me by projecting more accurate seasons, you will have to out think the likes of CHONE, ZIPS, Bill James, Ron Shandler and an assortment of other high level brain power. If you can really do that, you have earned the Genius label.
There is definitely more to come. But for now, I am off to get a haircut.
I do concede the point if we assume we're all doing the same thing at Phase 2. If I were forced to draft off a set of projections, sure. But Erickson does do all the projections for RW, and even he doesn't draft off them strictly. We just don't really care that much about projections – they're just an exercise, a guess, something we do so that our data is usable for our users. But they're conclusions – opinions, really – not facts. Bill can take a set of opinions and translate them into dollar values – he can say, "If you really think x, then you should play y." But we're aware of the flaws of x, and even the flaws in the conversion process. Essentially, when you start with a false premise, is sounder reasoning more likely to lead you to a truer conclusion? His dollar values are only true to the extent the projections are true. But to the extent the projections are false, being slightly off on the dollar value conversions might even be of benefit. But more importantly, I wonder whether quants can ever make serious inroads in a complex game like baseball. Backgammon, chess, poker sure – those are math games. But baseball is like the weather or the stock market – organic, unpredictable. Didn't the quants nearly take down the entire financial system a couple years ago because the housing market dropped more than they conceived it could? Is this kind of analysis really optimal when it comes to things like baseball players, stocks and weather patterns? Or is it much more useful with games where there are less unknowns like poker or chess? And finally, if you look at what the goal of this is – picking off mistakes in valuation relative to the "market's" projections, isn't this a pretty unambitious system, more designed to finish in the top half of the league each year rather than win it? Won't one of the six "genius" drafters almost always cash in bigger than the guy making a small but riskless profit from the vig? It seems like much of Wall Street, Vegas, etc. is simply making money off of suckers rather than having an ability to get ahead of the market and make the genius calls. Or the "geniuses" look smart for a year or two and then crash down to earth.
I agree Bill could build a model that crushes a room of timid agnostic experts, but I don't think picking off a buck or two here and there will beat a group of aggressive, "genius" drafters looking for and occasionally finding a $20 score or two.
I do leave room for the possibility that I'm wrong, however, and Bill's edge is far bigger than I'm acknowledging. But my bet is that Eric is the real shark. The question is whether he's read enough RotoWire notes to be on an even playing field.
One other quick point – now that I've seen Bill's post – those projections systems are all terrible. And they're some of the best. I would imagine it's much harder to beat Shandler the person than it is to beat his projections system.
Hmmm. I like your Kesselshark conclusion. Maybe you've convinced me!
Actually you're saying a lot of things I'd like to respond to, hopefully tomorrow…
What someone ought to do is come up with wagering odds on the 10 teams so people can have action on the league. I'm too biased to do it, so my odds would be worthless, but maybe someone not in the league… Also, here's a question: If the squares stopped betting NFL football – and I'll define squares as all the tourists and weekend gamblers who don't rely on it for income – what kind of vig would the house require to take bets from the rest of the bettors?
Well, I'm not involved in fantasy baseball whatsoever. However, I'm a friend of Eric & Bill, and I happen to also do some sports betting. So I think I can answer your question about NFL.
If all action were from sharp bettors, there is no amount of vig which would be sufficient to make providing sports lines profitable. The point here is that by definition, professional bettors are supposed to only place bets with edge, and are never required to place a bet. It's a classic zero ante game, in poker terms.
Let me revise the hypothetical then – let's say there were no squares, and a billionaire was backing the sharps in a contest to see who could lose the least against the book. (Billionaire pays all losses, plus a big bonus for whoever does best). In other words, these guys would be angling for an edge relative to each other even if the size of the vig gave the book an edge against them. How big would the vig have to be for the book to let them play? For the book, whose losses would not be reimbursed to go along with this contest, what would it have to be – every game vs. the spread -120? -125? My point is that setting a line which is necessarily imperfect because no perfect line exists because the exact distribution of outcomes is impossible to know, is dangerous when smart people are free to pick the side. When biased people are picking the side, the vig could be nearly zero, and the book would probably still come out ahead in the long haul by setting "trap" lines. And -110 gives them a comfortable profit, But against only sharps, the vig would have to be pretty high for the book to play along.
saeed میگه:salamheyataee ke raaesaye khob daran mtoanin bedan.vali bazi heyata ke raaesaeee mese aghaye hajari dar shiraz daran na ona az in kara nemikonan
Another Day…
[...]we like to honor other web sites around the web, even when they aren’t connected to us, by linking to them. Below are some web-sites really worth checking out[...]…
That’s really tinkhing of the highest order