April 2, 2013
Coach Kevin Kelley of Pulaski Academy in Arkansas runs a very unusual football team. Kelley never punts. He also rarely even attempts to return punts by the opposing team, and nearly always attempts onside kicks.
His teams have had an exceptional record, going 114-22 from 2003 through 2012, making it to the state quarterfinals in 2005, the state semifinals in 2009,the state championship game in 2006 and 2010, winning state championships in 2003, 2008 and 2011.1
Kelley developed this unconventional strategy around 2005, influenced in part2 by Berkeley professor David Romer’s paper “Do Firms Maximize? Evidence from Professional Football”, a comprehensive analysis of whether NFL Football teams attempt to convert on fourth down (rather than punt or kick a field goal) as often as they should. The paper not only makes the case that fourth down conversions are dramatically under-utilized in the NFL, but it also lays out a framework for how to compare completely different types of plays, such as kicking or going for it, in a consistent way.
Nearly every profile of Kelley I could find reports him discussing the numbers behind his decisions. Kelley’s unorthodox strategies are actually the result of a solid process for analyzing decisions, backed up by statistics, research, and data.
The process starts with a consistent way to measure the value of of a football play, along with a strategy for thinking about the range of outcomes, and the probabilities and values of each outcome from a particular decision. Each potential outcome has an expected value derived from the probability of that outcome occurring and the expected number of points that outcome would yield. A team is more likely to score if they have the ball one yard from the endzone than if they have the ball 99 yards from the endzone, so the first game position will be more valuable than the second. Using data from actual games, it is possible to estimate how much more valuable.
This process is the foundation of Kelley’s unusual strategies.
Let’s examine a hypothetical fourth down and five conversion attempt from a team’s own five yard line to see how this works. Kelley estimates3 that when a team punts from near its own endzone, the other team will end up with the ball inside the 40 yard line. The odds of the other team scoring from that position are about 77%. The expected value of punting from our starting position, then, is -5.4 points. If you go for it, let’s say you have a 50% chance of converting and ending up with a first down around the 10 yard line, and a 50% chance of turning over the ball inside the 10 yard line, which gives the opponent a 92% chance of scoring. The expected value of going for it, then, is 50% of the value of a first down on the ten yard line (let’s call that -.5 points) plus 50% of the value of a turnover at the 5 yard line (-3.2 points), for a total of -3.7 points. In other words, if you have a 50% chance of converting this fourth down, there’s a 1.7 point advantage in going for it.
|Opponent first down @ our 40 yard line||100%||77% scoring chance = -5.4 points|
|Opponent first down @ our 5 yard line||50%||92% scoring chance = -6.4 points|
|First down @ our 10 yard line||50%||-1 point|
In my example, I have estimated most of those numbers, and although I’ve picked numbers that look reasonable given the values that Kelley has provided in the press and the data that Romer published, executing a strategy like this requires a constant supply of good data in order to estimate outcome probabilities and expected values. Kelley is constantly on the lookout for new information to refine his system.
Here’s Kelley in an email to the New York Times NFL Blog, less than a week after winning a State Championship in 2008:
I am constantly doing research on going for it on fourth down and was wondering if you could guide me toward some tools, books, or Web sites that could further help me in my research. I am a numbers guy and would like to get this down to a science as far as the percentages go.
Kevin Kelley’s program is not only a great example of how data and analytics can influence high level strategy and decision making in football, it is a model for how to build a successful analytics process in any domain. Here are a couple of key points that translate directly:
- Focus on decisions rather than measures. When you see or hear Kelley discussing numbers, it is always for the purposes of evaluating a coaching decision that he has to make. In sports, and in any domain, you can sink a lot of time into creating measures to compare and rank performance, but even the best measure is useless if it doesn’t help you make better decisions.
- Evaluate decisions based on value delivered. When Kelley talks about numbers, his core measure of value is usually points scored, not yards. Points scored is usually (though not always, in some end-game situations) correlated with winning, which is what teams really care about. When you choose measures to evaluate your decisions, make sure those measures stay correlated with success. There is a tendency for anything you measure to improve, but you want to make sure that any improvement in the measure is actually correlated with value.4
- Experiment. The process and tools of analytics are much better at giving you feedback on things that you try than they are at explaining what might happen for things that you haven’t tried.
- Keep collecting more data and refining the analysis. As we saw above, Kelley is always looking for data to improve his analysis, even when he’s at his most successful. Successful analytics operations are always refining and testing their conclusions. Often, when an analysis of data reveals a correlation, it is impossible to know whether that correlation reveals a useful effect. The correlation might reveal something that leads to more success, or it might reveal something that cannot be used to derive value, such as something that is caused by another factor. The only way to find out is to form a hypothesis, test it out, and see what happens.
If you look carefully at what Kelley says about his process, you can see that he is doing all of these things, and he has an impressive record of success to show for it.
Compiled from several sources, including MaxPreps.com, Colleges should use ‘Madden’ strategies by DJ Gallo for ESPN, August 14, 2012, and It’s time to punt that punter!, by Gregg Easterbrook for ESPN, August 28, 2012 ↩
Gregg Easterbrook for ESPN’s Tuesday Morning Quarterback, November 15, 2007:
Coach Kevin Kelley reports that he stopped punting in 2005 – after reading an academic study on the statistical consequences of going for the first down versus handing possession to the other team…
Jon Wertheim cited Kelley’s estimates of 77% and 92% in two articles for Sports Illustrated: Down 29-0 before touching the ball on September 15, 2011 and Just Go For It! on September 21, 2009. The other numbers here are my estimates for example purposes and are not derived from any real data. ↩
An interesting violation of this principle has happened with advertising on the web. At some point, people decided to measure (and charge for) the reach of advertising by the number of page views received, which was correlated with the value delivered by an ad campaign. Once that happened, people started to do things to maximize their page view count, even at the expense of their user experience and their core value to advertisers. So now you see many sites that split their articles into pieces and force you to click through all of them to read the entire article. This provides a user experience that is clearly worse than if they could just scroll through the entire article in one piece, it makes it much less likely that a user will make it through the entire article, and it devalues the page view measure, since the same amount of content presented to the same number of users (in roughly the same amount of screen space) is now counted as multiple page views where before it was counted as one. If a measure like page views starts to deliver less value to those who actually pay for it (in this case, the advertisers), then they will not be willing to continue to pay as much for it. What these sites have done is a classic case of improvement in a measured (and compensated) statistic without any real change in the underlying amount of value being delivered. ↩