Coach Kevin Kelley of Pulaski Academy in Arkansas runs a very unusual football team. Kelley never punts. He also rarely even attempts to return punts by the opposing team, and nearly always attempts onside kicks.

His teams have had an exceptional record, going 114-22 from 2003 through 2012, making it to the state quarterfinals in 2005, the state semifinals in 2009,the state championship game in 2006 and 2010, winning state championships in 2003, 2008 and 2011.¹

Kelley developed this unconventional strategy around 2005, influenced in part² by Berkeley professor David Romer’s paper “Do Firms Maximize? Evidence from Professional Football”, a comprehensive analysis of whether NFL Football teams attempt to convert on fourth down (rather than punt or kick a field goal) as often as they should. The paper not only makes the case that fourth down conversions are dramatically under-utilized in the NFL, but it also lays out a framework for how to compare completely different types of plays, such as kicking or going for it, in a consistent way.

Nearly every profile of Kelley I could find reports him discussing the numbers behind his decisions. Kelley’s unorthodox strategies are actually the result of a solid process for analyzing decisions, backed up by statistics, research, and data.

The process starts with a consistent way to measure the value of of a football play, along with a strategy for thinking about the range of outcomes, and the probabilities and values of each outcome from a particular decision. Each potential outcome has an expected value derived from the probability of that outcome occurring and the expected number of points that outcome would yield. A team is more likely to score if they have the ball one yard from the endzone than if they have the ball 99 yards from the endzone, so the first game position will be more valuable than the second. Using data from actual games, it is possible to estimate how much more valuable.

This process is the foundation of Kelley’s unusual strategies.

Let’s examine a hypothetical fourth down and five conversion attempt from a team’s own five yard line to see how this works. Kelley estimates³ that when a team punts from near its own endzone, the other team will end up with the ball inside the 40 yard line. The odds of the other team scoring from that position are about 77%. The expected value of punting from our starting position, then, is -5.4 points. If you go for it, let’s say you have a 50% chance of converting and ending up with a first down around the 10 yard line, and a 50% chance of turning over the ball inside the 10 yard line, which gives the opponent a 92% chance of scoring. The expected value of going for it, then, is 50% of the value of a first down on the ten yard line (let’s call that -.5 points) plus 50% of the value of a turnover at the 5 yard line (-3.2 points), for a total of -3.7 points. In other words, if you have a 50% chance of converting this fourth down, there’s a 1.7 point advantage in going for it.

In my example, I have estimated most of those numbers, and although I’ve picked numbers that look reasonable given the values that Kelley has provided in the press and the data that Romer published, executing a strategy like this requires a constant supply of good data in order to estimate outcome probabilities and expected values. Kelley is constantly on the lookout for new information to refine his system.

Here’s Kelley in an email to the New York Times NFL Blog, less than a week after winning a State Championship in 2008:

I am constantly doing research on going for it on fourth down and was wondering if you could guide me toward some tools, books, or Web sites that could further help me in my research. I am a numbers guy and would like to get this down to a science as far as the percentages go.

Kevin Kelley’s program is not only a great example of how data and analytics can influence high level strategy and decision making in football, it is a model for how to build a successful analytics process in any domain. Here are a couple of key points that translate directly:

• Focus on decisions rather than measures. When you see or hear Kelley discussing numbers, it is always for the purposes of evaluating a coaching decision that he has to make. In sports, and in any domain, you can sink a lot of time into creating measures to compare and rank performance, but even the best measure is useless if it doesn’t help you make better decisions.
• Evaluate decisions based on value delivered. When Kelley talks about numbers, his core measure of value is usually points scored, not yards. Points scored is usually (though not always, in some end-game situations) correlated with winning, which is what teams really care about. When you choose measures to evaluate your decisions, make sure those measures stay correlated with success. There is a tendency for anything you measure to improve, but you want to make sure that any improvement in the measure is actually correlated with value.⁴
• Experiment. The process and tools of analytics are much better at giving you feedback on things that you try than they are at explaining what might happen for things that you haven’t tried.
• Keep collecting more data and refining the analysis. As we saw above, Kelley is always looking for data to improve his analysis, even when he’s at his most successful. Successful analytics operations are always refining and testing their conclusions. Often, when an analysis of data reveals a correlation, it is impossible to know whether that correlation reveals a useful effect. The correlation might reveal something that leads to more success, or it might reveal something that cannot be used to derive value, such as something that is caused by another factor. The only way to find out is to form a hypothesis, test it out, and see what happens.

If you look carefully at what Kelley says about his process, you can see that he is doing all of these things, and he has an impressive record of success to show for it.