Survivor Bias in Pokemon: Why Analysing Top Cut Teams is a Flawed Practice

Hello Hat Lovers!

Analysing top cut teams and drawing conclusions is a common practice in VGC. However, this is an imperfect process for many reasons, one of them being that it falls victim to survivor bias.

Survivorship bias or survival bias: the logical error of concentrating on the people or things that made it past some selection process and overlooking those that did not, typically because of their lack of visibility. This can lead to false conclusions in several different ways. It is a form of selection bias.”

The intent of this article is not to shed light that the definition of “good” in VGC is dynamic because of meta-game progressions, and it is not to contradict the specific, flawed conclusion that any successful team must be good because of its success. Rather, it is to bring attention to how survivor bias can mislead any conclusion people reach when analysing results.

Ignoring the Other Factors that Lead to Success

While teams often receive much of the attention, there are many (and arguably more important) factors at play that lead to the results you see at tournaments.

For one, individual tournaments are extremely volatile. Events typically only last eight rounds, and in such a small sample size, other uncontrollable factors can decide results. Whether it’s team match ups, bad luck, or lack of bad luck, people often read too much into specific placements. That’s not to say that every successful player only had success because of good luck and good match ups, but minute differences in placements are often decided by a bevy of other factors, in that a 4th place team can often be better than a second place team, or how a player who finished eighth can be superior to the winner of the tournament. This idea is not lost on anyone. Sometimes, however, people can overlook just how much these other components in Pokemon can influence results.

Furthermore, a defining aspect of success is the player itself. Beyond stating the obvious that mastering a certain set of skills are a driving component of success, similar teams will also play differently depending on the user. Different players will understand how the team functions better than others, as well as learning damage calculations and how to approach certain match ups. It is nearly impossible to quantify just how much of success is driven by teams or by player quality (player quality itself is nearly impossible to quantify), but the former receives the lion’s share of attention even though there’s little indication that it correlates more strongly.

Additionally, there is the fact that if a large number of players decide to use the same team, mathematically speaking, of course a greater quantity of them will make top cut. This is why Kangaskhan was on 6/8 top cut teams in Worlds 2015, even though Kangaskhan wasn’t actually the mega that had the greatest top cut percentage in that tournament. It is important to shed light to a common misconception that players have: usage does not equal quality! The chief reason that that tournament was so populated with Kangaskhan at the top might be that many players chose to use Kangaskhan in the first place, not necessarily because Kangaskhan was the best mega (note: I’m not arguing that it wasn’t, but it probably wasn’t as dominant as the usage suggested).

Another example of survivor bias is that, in 2017, a lot has been made of the fact that many teams in the format lack Ground resists. However, a grand majority of the teams that are observed are those in top cut of events, or those that are ranked highly on online ladders. However, given that those are teams that have seen success, could it not be argued that the lack of Ground resists may not really be that big of a problem? If it was, wouldn’t those teams fail to make top cut, or fail to rank highly on ladders?

So, why do teams receive such a large amount of attention when analysing results as opposed to the aforementioned factors? In my opinion, it is because teams are the most easily noticeable and replicable. You can’t plaster luck and match ups onto a fancy, easy-to-read info-graphic like you can with teams. You can’t replicate player-skill like you can with Showdown imports. There is undeniable value in learning what teams see success at tournaments, but by doing so, it is important to understand the limitations of this process, in that assessing teams prominently ignore a substantial number of other causes that drive results.

Ignoring Failed Teams

The reason failed teams get ignored is obvious: they don’t get enough exposure. Teams that are featured on streams are only those that have done well, as well as those that make it onto many usage stats. It’s hard to fault players for only analysing a select subset of teams in the meta game; however, in order to have a better idea of why certain teams succeed, it’s just as important to understand why other teams fail. Of course, it is nearly impossible to access the hundreds of teams that are played at tournaments. I am not suggesting that this should be done. The point is that failed teams are just as much a part of the data as successful teams are, even if they largely get ignored.

An example of failed teams being ignored is whenever casual players believe that there is no diversity in competitive Pokemon, because the teams that they are exposed to are often very similar. The fact is, though, is that there is diversity with Pokemon choices, but the problem is that unique teams often times don’t succeed, and as a result, are never shown to a mass audience.

My point is not that analysing top cut teams is a fruitless endeavor, or that teams have zero influence on success. Rather, despite the amount of attention they receive compared to other defining factors, teams are only a small fraction of the equation in results, and must be taken with a grain of salt. In general, survivor bias is a dangerous trap to fall into in VGC (even in other areas like evaluating player behaviour with last minute teams), but with analysing team results specifically, there are only so many conclusions that you can draw. There are many other important factors that are being ignored even when they shouldn’t be.

One comment

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s