The Problem with Projections
April 15th, 2010 by Chris Liss in General Guidance, TheoreticalI posted this as a comment in the last thread, but figured it merited its own because it's a new topic.
I think the reason it's hard to make accurate projections even setting aside injuries and variance, i.e., luck, is that healthy players' skills don't remain the same. And anticipating skill growth and regression, apart from luck, is the key to the game. One can plot general growth curves for players (hitters peak and 27 and slowly decline), pitching, I think, at 31, but the general curve is just an average and does not apply to individual players whose growth and regression can be gradual or abrupt, or proceed in fits and starts. So whatever projections you use will be wrong unless they can anticipate growth or regression optimally for each player – apart from the variance that accompanies their growing or regressing skill set.
The problem with projections whether you use a range or a specific number is that by their nature they assume an average amount of growth or regression given the player's age or experience level. If I think Justin Upton will hit 45 homers this year (that that's really his new baseline given his park, his rapidly approaching peak, his growing experience, etc.), I cannot possibly put that number into the model because it's too far away from the normal growth curve of a player his age and with his history. But outlier careers exist! Not just as a matter of variance, but outlier baselines exist. If you were doing career projections you could not have given Pujols or ARod their actual baselines. It was too unlikely. But some player is going to be Pujols or ARod. So your model either has no outliers or random outliers. That's the trouble with projections – they're impossible to do right not only because of variance, but because they're either too timid, or too speculative.
So I've found it's better to just practice identifying the breakout players, and go the extra buck for them at auction (within reason). Just get really good at identifying them by being honest and rigorous at the end of the year about which ones you got right, wrong and why. Condition one's brain to synthesize the data well enough to get a sense of what collection of variables gives the best chance to find the breakout guy. If you're good at that, then you'll win your leagues more often than you should. If you could build a model that synthesized those variables better than my brain, I'd be concerned. I don't know if that's possible, but if there were a projection-creating model that was more than as Peter says, weighted averages, that would be something.
As I've said many times before, I'm skeptical that a pricing model (as opposed to a breakout-finder model) would confer a significant advantage. And I've probably argued it more vigorously than I've needed to. The bottom line – in my experience playing against really strong players some of whom use a model of that kind and some who don't, it's not a difference maker. It's possible in the 30 or so high-level experts leagues I've played in, and the 50 more regular leagues, I've missed something, and there's more to it.
But apart from my experience, that most valuation models depend on projections, and projections, it seems to me, are necessarily timid or random makes it hard for me to believe any system that merely seeks to translate projections accurately is going to be a winner – unless the other players aren't any good at finding the outliers, i.e., are as likely to leave the breakout players behind as they are to choose them. Then the vig might be enough to make the difference.
Tags: Chris Liss, fantasy baseball, projections
Loading ...
If you are good at identifying potential breakout players, then you should be able to incorporate that into your projections. If someone is as good as you at identifying these players, but then also has a robust pricing model, they are going to have an advantage over you. Every time you choose to attack the "pricing model" method, you attribute to it some other weakness which makes it not work. You says A"unless the other players aren't any good at finding the outliers". Well why can't the guy who is making projections and using a model be good at finding the outliers? You continually act as if the guy is using the model is bad at every other aspect of fantasy baseball. Well, what happens if he is good at all the stuff you are good, but he also uses a good model! Is that just not a realistic possibility to you?
Well, I'm only assuming that because the whole debate started with Bill declaring that he's a complete agnostic as to which players would and would not break out. That was the whole point. That his stats to dollars model was good but that he had no particular talent or preference as to which players he got, and that he'd just use whatever projection system CHONE, Shandler, whatever as his inputs.
My point is those inputs will almost necessarily be timid because they purport to be scientific, and they'll have a hard time justifying an aggressive new baseline for players. My example of career projections is that they would never give Pujols or ARod those numbers – look at BP's career projections – the most HR they ever give Justin Upton is 25 in 2014-2017! Obviously those are ultra conservative, but when you factor in the possibility of injury or whatever, they think that's his 50th percentile number, the 3.5 on the six-sided die for him.
Could you put in absurd projections for your pet players – 45 HRs for Upton? Yes, you could. But you'd have a hard time justifying that as a rigorous model. So I've found – okay, I like Upton, but I can't in good conscience give him 45 as his median HR number in my model even if I think that whatever the market number will be is going to be way too low. So I just vow to go the extra buck on him at the auction – within reason. If he's going to cost ARod money, I'll probably let him go.
It's not that I don't think models are legitimate – it's that to the extent they're legitimate, they make you an agnostic. And I've found through experience that one must make a call – one must pick the players that one wants aggressively to win in this game, provided one has an aptitude for that sort of thing (how you tell whether you're actually good at this or merely have deluded yourself into thinking you're good at it is another topic).
I think that's because once you get out of the first four rounds of your draft (in football it's once you're out of the first round), then upside and volatility mean more than median projection. You can toss the 3.5 out the window and just ask what's the top 25 percent result of the die, not the average. So you need to project players (except the most expensive ones) based on upside, not median. The deeper you go in the draft/auction, the more extreme this is. So at the very end, your $1 players should have a 10 percent chance of being $20-$25 players, even if they don't have a job now. Because you can always mix and match if they don't pan out.
I suppose you could put this into your pricing model, too. But then you're not using projections – or at least any kind of projections you'd release to the public as your expected returns for players. You'd be using predictions and again, your model would be very subjective, depending on the player. Because even though I think generally, it's a matter of bumping the more volatile cheap players' numbers and bumping the more safe expensive ones, it varies from player to player. Even two players with the same upside are not the same bet to grow their skills to the same extent. So in the end there's a subjective element that makes your inputs unscientific, and so I've found it's easier and better just to express these unscientific hunches/reads on the game when the player's name comes up than have them blow up my model so that I've got all kinds of crazy outlier projections.
A lot of the stuff you say here makes some sense, except I just don't get why you don't think all these concepts can't simply be represented through the numbers in your projections and priciing
I also feel you tend to conflate your arguments a bit when you say stuff like 'Could you put in absurd projections for your pet players – 45 HRs for Upton? Yes, you could. But you'd have a hard time justifying that as a rigorous model.'
ALL we are talking about at this point (we've had to back track several times in search of a consensus) is the value of an accurate pricing model. There is no reason you can't have a very rigorous PRICING model and insert any kind of projections you like.
All we're basically asking you to acknowledge at this point is that Chris Liss with a hyper accurate pricing model for his projections (thoughts?) has an advantage over Chris Liss without such a model, and that such advantage MAY be significant. Is such an advantage dwarfed by other factors? Maybe! But you haven't really let us get there because you keep fighting what I feel are really fairly uncontroversial and general points.
Eric, I don't know. I'm resisting this because I think the person who uses a model of the kind you're describing is not the person who's willing to go to the mat for Upton. It's someone like Bill who has no preference. Once you get into the realm of hunches and your own personal scouting, I'm not sure you should be putting fixed numbers on that stuff for there even to be a translation model. And even if I did try to quantify my hunches in specific numbers, and used Bill's model to get values, I don't think I would abide by it during the auction. I might consult it, I might find it interesting or curious, I might even learn from it, but I don't think I would adhere to it.
I'm just not convinced that there's an advantage to picking players by turning them into numbers and then dollars. So you're asking me: if you were forced to turn your hunches into numbers, do you agree that it would be better to have those numbers convert more precisely into dollars or not? I'd say probably. More often than not.
But I don't buy into the first premise that I should be converting my hunches into hard numbers in the first place. Or even a range of numbers. At least not explicitly.
Back to the language analogy: you need to translate French into English before you can understand it. Your translation program is uncannily accurate. You can convert french into English without losing a single word. BUT I SPEAK FRENCH. So you're asking me – if instead of understanding French words directly, I were to translate them into English first the way a non-french speaker would, would I rather have the translation program be better or worse. Sure better. But if you're asking me if someone with a great translation program has an advantage in responding in English than someone who's bilingual and hears French, understands it and speaks English with no need for any kind of translator, I would say no.
I think that's where we're stuck. You just can't imagine that I understand what players are worth at auction just by knowing the pool and having experience. You think everyone must translate them into specific numbers first. Maybe I'm doing what you say unconsciously but I can assure you that at no point in my preparation or during the auction am I consulting specific projected stats. At no point do I even make projected stats. I just make a by-position list based on my research, and I don't even adhere to that! Sometimes I'll pay more for someone lower on it if I change my mind as I know the list isn't ordered that carefully. It's just for me to know what's available at each position as the auction goes on.
So I think it's a false question that you're asking me. Would someone who had my hunches do better if they had precise translations of whatever my hunches amount to in numbers – to dollars. I can't agree to that because I don't agree with the premise of the question.
So I'll agree to this. If we're all using the same projections and they have some modicum of accuracy – some historical basis – then yes, the one with the better translation model will have some kind of edge. If we're using equally good but different numbers, whatever that means – if such a thing is even possible – then probably there's an edge, but I'm not 100 percent sure. But if I'm not using numbers whatsoever, I don't see how I can answer that question.
I'm fluent in English, and I still use a spell checker constantly (and not just for typos). Sometimes I'm mildly wrong about how to spell a word. Every once in a while, I'm really wrong.
If I were in an essay writing contest, I would be really glad I had a spell checker even if I would have preferred a dictionary.
I wouldn't care for a spell checker personally. I know how to spell just about any word I would want to use. That's why I typically don't buy players I don't have a good read on – unless they fall to me really cheap. As for typos, I make them. And I make some mistakes in just about every auction. But it's not because I lack a pricing model. My worst mistakes are not being able to predict what other people will pay for players, so misjudging what I can get for what I have left, or bidding on a player that my "model" says is cheap but who I really don't want and getting stuck with him.
Well, at least you didn't assert that inaccurate spelling might somehow help you in the essay competition, like some kind of creativity bonus
It's always a good sign when your opponent is reduced to picking on some tangential, throwaway point you made. I'll take that as concession.