How often does the best team win the championship?

Imagine we have a four team single-elimination tournament. Team A is good enough that you'd expect them to win about 80% of all games, Team B ought to win about 60%, Team C should win 40%, and Team D ought to win about 20% of games (against random opponents). If we seeded a single elimination tournament with these teams, it could look something like this:

Given the information above, how likely is it that the best team in this tournament, Team A, ends up being the winner? In other words, how effective is this tournament structure and seeding system at determining the best team out of the four?

The first tool we'll need for this is the Log5 formula - given the true winning percentages of two teams, this formula tells you the odds of a given team winning. So for instance, Team A playing Team D is pretty lopsided, and Team A has a 94.18% chance of winning that game based on this formula. Similarly, Team B and Team C are much closer in relative skill, so Team B only has a 69.23% chance of winning that game.

Based on all of this, we can come up with the relative chances of any of the four teams winning the tournament overall:

So in our hypothetical situation, this single-elimination four-player seeded tournament with known team skillsets resulted in the best team, Team A, winning 72.26% of the time, and the worst team winning 1.06% of the time. Not shabby.

However, what if you didn't know the skill levels of the teams going into the tournament? How confident could you be that the eventual winner of the tournament was, in fact, the best team when they signed up? One way to determine this is by running a Monte Carlo simulation - let's sign up four teams of random skill values, run them through a tournament exactly the same way as we just did with our sample teams, and see who the winner is. Then let's do that 10,000 times, and see how often the best team wins.

The results are interesting: with randomly drawn skill values for all teams (mean= 0.5, stdev=0.13 [see below**]), we'd expect the winner of the tournament to be the strongest team only 44.3% of the time. About 10.5% of the time, the weakest team in the tournament would end up winning the whole thing!

So is a single elimination tournament a particularly good way of determining the best overall team from a pool of four teams? Probably not. What happens if we up the ante, and have a double elimination tournament? Double elimination tournaments are exactly what they sound like - any given team needs to lose twice before being eliminated. They tend to look something like this:

This sort of format ought to improve the chances of the best team winning, as a single unlucky (and unlikely) loss won't eliminate them too early. If we run the same sort of analysis, but with a double elimination tournament, we end up with the winner being the strongest team 51.1% of the time, and the winner being the weakest team only 7.7% of the time. A reasonable improvement all in all.

Unfortunately, this modest increase in chances of determining the truly best team is offset by the increased length and uncertainty in the tournament. A single elimination tournament needs three games total, whereas a double elimination tournament needs either six or seven games, depending on how the sixth one goes. This is also annoying to schedule and sell tickets for, as organizers have no way of knowing if the sixth game will be the exciting final or not.

Expanding a bit, we can do the same analysis with single, double, and triple (!) elimination format tournaments for tournament of as many teams as we want. Before we continue, though, I'll mention that for a 32 team single-elimination randomly-seeded tournament, the odds that the winner will have been the best team overall are 22.0%. Keep that in mind.

So why talk about these tiny little tournaments? It's to get you ready for the real deal: professional sports.

Professional sports leagues generally tend to fall into a regular season and playoffs, where the regular season is used to seed teams into some sort of order and filter out the best, who end up playing in a tournament bracket of one style or another. What happens if we do the same sort of analysis for each major (and some minor) sports league?

National Basketball Association

Overview: 30 teams play 82 games each in the regular season. Teams are sorted into conferences and divisions. The playoffs are a 16-team single elimination tournament bracket, where each round of the playoffs consists of a best-of-seven games series for elimination. Teams are seeded within conferences, and the bracket is fixed at the end of the regular season. This gives us:

Overall odds of the NBA Championship being won by the season's best team: 45.9%.

Pros: Best-of-seven series in playoffs reduces variability in results. Not seeding teams based on division standings, and only looking at conference standings, reduces the chances of weaker teams getting into the playoffs by virtue of leading weaker divisions.

Cons: Relatively long season for regular season, large range in potential length of playoffs.

National Hockey League

Overview: 30 teams play 82 games each in the regular season. Teams are sorted into conferences and divisions, and are more likely to play teams inside their divisions and conferences than outside of them. The playoffs are a 16-team single elimination tournament bracket, where each round of the playoffs consists of a best-of-seven games series for elimination. Teams are seeded within divisions such that the top three teams of each division are guaranteed a spot in the playoffs, and the highest-performing two teams of the conference that remain get in as wildcards. All of this results in the following distribution:

Overall odds the Stanley Cup winner was the season's best team: 45.4%.

Pros: Best-of-seven elimination in playoffs reduces variance and increases the chances of better teams triumphing.

Cons: Relatively long season for regular season, large range in potential length of playoffs.

Fun fact: Only real difference between NHL and NBA results is the seeding into the playoffs and how wildcards are handled, and that results in hardly any change at all.

Major League Baseball

Overview: 30 teams play 162 games each in the regular season. Teams are sorted into leagues and divisions, are are more likely to play teams inside their divisions and leagues than outside of them. The playoffs consist of the winners of each division and the two wildcard runners-up from the conference, and is a wonky sort of 10-team single elimination tournament where the first round are two single game wild-card playoffs, followed by four best-of-five division series, then two best-of-seven league winner series. Finally, the winners of each league play each other in a best-of-seven series to determine the winner. From this, we get:

Overall odds the World Series winner was the season's best team: 45.4%.

Pros: Teams are more likely to be correctly seeded heading into playoffs due to extensive regular season. Fewer teams in playoffs makes it less likely for weaker teams to get lucky.

Cons: Short playoff season with sudden-death games and best-of-five series increases variance.

Fun fact: If all rounds (including wildcard) in the MLB playoffs were best-of-seven series, the odds that the winner was actually the best team increase to 46.4%.

National Football League

Overview: 32 teams play 16 games each in the regular season. Teams are sorted into conferences and divisions, and are more likely to play teams inside their divisions and conferences than outside of them. The playoffs are a true 12-team single-elimination tournament, where each division leader and two runner-up wildcards are seeded from each conference. This results in:

Overall odds the Superbowl winner was the season's best team: 28.2%.

Pros: Short, fixed playoff schedule is predictable.

Cons: Single elimination sudden death games greatly increase the chances of weaker teams winning by chance and eliminating stronger teams. Relatively short season also doesn't guarantee accurate seeding of teams heading into playoffs.

Fun fact: Before, I mentioned that the chance of the winner of a randomly-seeded 32-team single-elimination tournament being the best team was 22.0%. That means the NFL season format isn't really all that much better than just having one six-week March Madness style showdown each season.

Some more minor tournaments that are still near and dear to my heart:

Canadian Football League

Overview: Nine teams play 18 games each in the regular season. Teams are sorted into two divisions. The playoffs are a single-elimination 5-team tournament, where the highest-ranked team in each division gets a bye to the division finals. Teams can cross-over into other divisions if the fourth-place team in one division has more points at the end of the regular season than the third-place team in the other division. This gives us:

Overall odds the Grey Cup will go to the season's best team: 38.0%.

Pros: Short, fixed playoff season is predictable. Still somehow have a longer regular season than the NFL. Higher ranked teams getting a bye to the division finals helps out the stronger teams.

Cons: Single elimination playoff format, which includes more than half the league, leads to a bit of a crapshoot. It's disappointing than the NHL with over four times as many teams is better suited to finding its best team each year.

Fun fact: Again, this isn't substantially better than just running a 9-team randomly-seeded single elimination tournament right at the beginning of the year. It's more fun, though.

Curling

Overview: Major curling tournaments involve 12 teams, who play each other once each in a round robin. The top four teams are seeded into a Page playoff system, where the top two teams are in quasi-double elimination system, and the remaining teams are in single elimination. This gives us:

Overall odds the winner was the tournament's best team: 37.3%.

Pros: Fixed playoff system is short and predictable. Page playoff system gives a bonus to the teams who perform best after a fair and balanced round-robin.

Cons: Single elimination format of playoffs increases variability.

Fun fact: Curling is fun and you should try it.

So there you go. Unsurprisingly, sports leagues with longer regular seasons and best-of-seven playoff series are better suited for determining the actual best teams each season, whereas leagues with shorter seasons and single elimination tournaments as less well-suited. Now you have numbers to show for it, at least!