Premier Challenges: a year in review

Hey there Hat Lovers,

Many players bemoan the low-attendance Premier Challenges (PCs), Premier Challenge distribution, and how easy Championship Points are to get in certain areas relative to others. I wanted to know just how common low-attendance events were, which areas had better access to PCs, and what the trends in attendance looked like as the season progressed.

Today’s article is going to go over how the Premier Challenge system has gone since its introduction in the pilot series back in April-June 2014. Since then, we’ve had three more Premier Challenge series, the Kickoff Series, the Alpha Series, and the Omega Series. The dataset I’ve compiled comes from organizer reports of attendance in each series, and is not a complete dataset but just a representative sample. That said, my numbers aren’t too shabby; I have tallied attendances from 306 PCs worldwide, with 200 being from Canada and the USA.

What I’ve done is compared the series to each other within Canada and the USA, and characterized what attendances looked like within each series. I will look at the distribution of states reporting events, and also the trends in attendance, how the Junior/Senior attendances stack up, and what attendances in other countries look like. So without further ado, let’s dive in!

What do these data look like?

I collected information on Premier Challenge attendances as was available up to June 4th 2015. How I’ve collected and organized these data is as follows:

  1. I have the country and when available, the state/province/city of each Premier Challenge, allowing me to look at country-level and state-by-state comparisons. The city-level distinctions don’t appear to be that useful outside highlighting the best attended event locations.
  2. I have Masters division attendance for every Premier Challenge in the dataset, but only recorded Junior and Senior attendance if the total number of Juniors and Seniors was six or greater
    • Junior/Senior attendance is stochastic, and anything less than having their own age division (6 minimum) I attribute to random chance more than a local community. This stance comes from being a TO for the last couple years, and ease of organizing the data.
  3. I have these data split up for each of the Premier Challenge series:
    • Pilot (Apr 2014-Jun 2014), 55 events, 50 in Canada and the USA
    • Kick-off (Just Sept 2014), 68 events, 44 in Canada and the USA
    • Alpha (Nov 2014-Jan 2015), 115 events, 63 in Canada and the USA
    • Omega (Mar 2015-May 2015), 68 events, 43 in Canada and the USA

One thing to consider is that I’ve analyzed these data relatively soon after the Omega series ended, so there will likely be reports trickling in after this post goes up. However, my dataset is not underrepresented for the Omega series, as it has similar numbers to the Kick-off series in total reports and distribution of events. I don’t expect the overall trend of Omega series attendance would look much different while taking extra reports into account, but perhaps some outlying high-attendance events would be included which would increase the overall attendances marginally. I don’t expect this has much impact on the general trends present in the data.

Canada and the USA, how has it gone?

canadaus

Don’t worry Mexico, I’ll get to you in a bit

Canada and the USA are my home region, and thus my primary interest. The motivation behind this whole data collection and analysis was to look at Canada & USA statistics. As such, I’ve focused my analyses here, and as two thirds (2/3) of my data come from Canada and the USA, it’s reasonable that it earns this focus.

 Overall attendance

First and foremost, I want to address the appropriateness of including the Pilot series in these analyses. The Pilot series contained Premier Challenge events that were using a different Championship Points distribution, and were used to qualify in a different competitive season. They were also the first PCs ever, and may suffer from poor advertisement, or may receive the benefits of “first-ever” hype.

What I can do to assess the appropriateness of including these events in my overall analyses is to look at if the Pilot series differed from the other series’ attendances in any way. To do this, I can use a statistical test called the Analysis of Variance (ANOVA) which lets me determine if any of the series are significantly different from each other, and gives some power to interpreting box and whisker plots of the attendance in each series. Here are my results:

Caption

Figure 1. Boxplots of each Premier Challenge series’ Masters division attendance within Canada and the USA. ANOVA did not find that any Premier Challenge series differed significantly in attendance from any other series (df = 3, p = 0.368). The following were the means of each Premier Challenge series: Pilot = 14.38 ± 8.36, Kick-off = 16.30 ± 11.18, Alpha = 15.35 ± 9.25, Omega = 12.95 ± 8.08. The medians (black bars) of each Premier Challenge series were: Pilot = 13, Kick-off = 12.5, Alpha = 14, Omega = 10.

What this says to me is that the four series did not differ much in attendance at all. That p-value of 0.368 is not even close to significance, so there isn’t much reason to think our data would look all that different if we repeated this ‘experiment’ again. Thus I will treat each Premier Challenge series as being pretty much the same in terms of attendance. That is also a bit surprising, as it means that overall attendances haven’t really changed much since the start of the program. If you really want to read into the data, the worst-attended series has been the Omega series, which means attendances have dropped in the last series despite organizers being more experienced and having the year to develop and advertise to their communities; which is kind of surprising honestly.

Since there wasn’t actually much difference between each series, an alternate way of visualizing these data is to use a histogram to show where most events’ attendances ended up. The following graph describes where most of the events fell in terms of Masters division attendance from April 2014-May 2015:

Slide2

Figure 2. Histogram of Masters division attendance over the first four Premier Challenge series in Canada and the USA. The bars are split up by groups of 8, so the first bar represents the number of PCs that had 0 to 8 Masters division players, the second bar represents the number of PCs with 9 to 16 Masters, etc… The height of each bar indicates how frequent that attendance level was. e.g. 56 PCs had Masters attendance between 0 and 8, while only one PC had attendance of 56 to 64. Overall the average Premier Challenge attendance was 14.80 ± 9.27 (95% C.I. = 5.53, 24.07). The lowest Masters division attendance at a sanctioned event was 4, and the highest was 64.

As you can see a vast majority of events (139/200, 69.5%) see at best a Top 4 cut and Masters division attendance between 4 and 16 players. Indeed, 56/200 (28%) PCs have only three rounds of swiss, and actually 35/200 (17.5%) had less than 8 players, meaning no Top Cut at all. The 95% confidence intervals fall at 5.5 and 24.07, which means that the bottom 2.5% of events have less than 5.5 Masters in attendance, while the top 2.5% of events have greater than 24 players in attendance. So for everyone in well-attended areas complaining about other areas’ easy CP… yeah that’s kinda justified.

Thus, if we use attendance as a proxy for difficulty, if your area has PCs that consistently get 25 or more Masters, you’re playing in one of the toughest areas in Canada and the USA. And indeed, if you have ever played in a Premier Challenge with only 4-5 Masters in attendance, you’ve had one of the “easiest” tournaments in Canada and the USA; though it was probably a gong show ranking-wise.

Series-by-series attendance

The next thing I was interested in was how the growth of Premier Challenge events had progressed throughout the year. The boxplots in Fig. 1 give a good indication of what attendances have looked like, but they treat high attendances as outliers (the circles above the boxes) because of the skew of the data, and that doesn’t exactly paint the whole picture for what attendances have looked like in each series. So here’s the series-by-series histograms of attendance:

Figure 3. Histograms showing series-by-series attendance for Masters division players in Canada and the USA. Again, the means of each Premier Challenge series were: Pilot = 14.38 ± 8.36, Kick-off = 16.30 ± 11.18, Alpha = 15.35 ± 9.25, Omega = 12.95 ± 8.08. Attendance sample sizes were: Pilot = 50, Kick-off = 44, Alpha = 63, Omega = 43.

Figure 3. Histograms showing series-by-series attendance for Masters division players in Canada and the USA. Again, the means of each Premier Challenge series were: Pilot = 14.38 ± 8.36, Kick-off = 16.30 ± 11.18, Alpha = 15.35 ± 9.25, Omega = 12.95 ± 8.08. Attendance sample sizes were: Pilot = 50, Kick-off = 44, Alpha = 63, Omega = 43. The lines are density plots overlaid on the histograms to indicate where the majority of events fell on a continuum rather than through bars. I was playing around in R and it seemed fun.

What we can see through these histograms and underlying statistics is that the Kick-off series (average attendance = 16.3 ± 11.18) was the best-attended series. But interestingly, the data are bimodal (two peaks) rather than normal-like (one peak). This means that PCs in the Kick-off series were either very well attended, or similar to average attendance from over the year without much in-between. Indeed, 10/44 Kick-off series PCs had over 25 players (above the overall 95% confidence interval), and 7/44 had over 32 players (6 rounds of swiss).

So early in the year, players in high-attendance areas were pretty eager to play in events, but this enthusiasm waned as the season went on; Alpha series and Omega series PCs returned to attendances more like the overall attendance.

Juniors and Seniors from Canada and the USA

Not to ignore Juniors and Seniors, as I think they’re both important members of the community and also important for bottom-up growth of the game at the local level, here’s a quick look at Juniors + Seniors attendance in Canada and the USA. I’m lumping them together because there was only one Premier Challenge this whole year that had enough of both Juniors and Seniors to split them into separate tournaments (Kick-off series: Vancouver, Canada). Here’s what Junior and Senior attendance looked like overall:

Slide5

Figure 4. Histogram of Junior and Senior attendance from Canada and the USA. A total of 67/200 PCs had at least 6 Juniors + Seniors. Thus 133 PCs had Juniors and Seniors play against Masters. Of the 67 with their own age division, only 1 had separate age divisions for both Juniors and Seniors. The highest Junior+Senior attendance was 25, and the lowest was 0.

While not surprising, Juniors and Seniors attendance was paltry compared to Masters attendance. As far as just Seniors go (the more populated age division) 21/200 PCs had greater than 8 Seniors in attendance, meaning 10.5% of PCs in Seniors have a Top 4 cut and four rounds of swiss in a Juniors and Seniors-only tournament. I could show the boxplot of what Junior+Senior attendance looks like, but it’s not much to look at and I haven’t collected the data in an appropriate way to represent it. You’ll have to trust me when I say that Junior + Senior attendance didn’t vary much over the year, though there was one tournament in each of the Kick-off, Alpha, and Omega series that had better attendance.

States and Provinces reporting Premier Challenge results

Before getting to the discussion, I’d like to highlight areas that currently have access to Premier Challenges.

Figure 5. A map of the United States of America, showing states that have reported at least one Premier Challenge (blue) over the four Premier Challenge series. This map was generated using www.amcharts.com's interactive maps tool.

Figure 5. A map of the United States of America, showing states that have reported at least one Premier Challenge (blue) over the four Premier Challenge series. This map was generated using http://www.amcharts.com’s interactive maps tool.

While the distribution looks great, this map doesn’t tell the whole story (see Fig. S2). Either way if you’re looking for Premier Challenges near you, and your state is highlighted, then check www.pokemon.com or labs.pokegym.net and find out who runs events and where! I’m also happy to answer any questions people put in the comments if I can.

Figure S2.

Figure S2. States that reported at least 5 PCs over the Alpha and Omega series.

The above map shows the patchy distribution of PCs over the Alpha and Omega series. If your area doesn’t have enough Premier Challenges, maybe think about applying to run them for yourself? If you start your application now, and host some tournaments over the summer, the chances of your area getting PCs by next year aren’t too shabby!

Figure 6. Canada

Figure 6. Canadian cities with Premier Challenges are starred. Only five Canadian cities reported Premier Challenges: Victoria (BC), Vancouver (BC), Edmonton (AB), Toronto (ON), and St. John’s (NL).

Canada is sparsely populated with Premier Challenges, but the dense areas are in Vancouver/Victoria, and Toronto. Toronto is also close enough to the USA to drive over to more events, and Americans could drive north to attend Toronto PCs. I know Calgary (AB) has a Pokemon TCG scene, so it’s a shame to see they haven’t been on top of Premier Challenges yet. If you live in Calgary, try asking the local league(s) if they would host Premier Challenges coming up! In BC, we take entry fee and use it to buy store credit as prizing. So if the resistance is from the store seeing no benefit, take that angle!

And to finish this section off, here are the state-by-state (and province-by-province) attendances from Canada and the USA:

Figure 9.

Figure 7. State-by-state attendances. The areas with multiple PCs and the highest average attendances are (in alphabetical order): British Columbia, California, Massachusetts, North Carolina, New York, Pennsylvania, Texas, and Virginia.

I haven’t been able to filter these data for PCs that were tacked onto States or Regionals events, so some of these might not be indicative of average attendances in that area (e.g. Washington). But the number of PCs help indicate what areas actually host highly-attended events, while others are the result of one-offs tacked onto larger TCG/VGC events.

Discussing these results from Canada and the USA

da13b-bills-maintenance-expedition-ex-137

So what does all this say about PCs in Canada and the USA? First, I think it’s surprising that there has been literally no growth in attendance as Premier Challenges have progressed. I think it’s likely that that’s a bias introduced by the timing of the lone 2013-2014 series. After all, we saw that the Kick-off series from the 2014-2015 had the highest attendance overall. If we had had a September series for the 2013-2014 season, perhaps it would look more like the Kick-off series attendance.

Explaining the Kick-off series’ success, I don’t think it’s that the organizers did anything different throughout the year to lose the attention of the player base. I think the best explanation is that those highly successful events were due to players being excited about the new tournament season, and the hype that these Premier Challenges got in terms of their importance for qualifying for Nationals stipends and Worlds this year. As the year progressed and players had a better idea about where they sat in terms of stipends/invites, they made less effort to travel to Premier Challenge events, and attendance dropped.

What this says about Premier Challenges for next year, provided the trend holds true, is that players looking for Championship Points should target the later PCs if they’re looking for easier competitions. But I also think that it’s a bit of warning sign regarding the state of PCs in Canada and the USA:

Currently, we are in the first year of this program. If we want to continue to receive support and continue to have these Premier Challenge events, we as the player base need to maintain our drive and continue supporting these tournaments year-round. To the organizers, this means you can’t get discouraged about your attendances and need to keep up that can-do attitude in hopes of generating lasting attendance.

Keep in mind that this is a pretty short time-series, and year-by-year growth isn’t best determined by growth over the season, but rather by comparing things year-by-year; It’s obvious, but it has to be said. So while we lost attendance over the course of the year, and that attendance was actually slightly lower than last year around the same time, nothing meaningful can yet be said about the state of Premier Challenge growth until next year concludes.

In the meantime it’s up to us players to continue supporting these tournaments, and show Play! Pokemon that it’s worth expanding the VGC circuit and continuing local-level support!

Worldwide results

To finish off the analyses available, here are the same graphs applied worldwide:

Slide2

Figure 8. Worldwide attendances organized by country. An ANOVA would be pretty meaningless for this data, as it’s almost certainly going to say “they’re different!” But you can see what countries have held PCs, and how large those PCs were. If a box plot only has a black line, that means there was only one event. Not shown is the 316-person Premier Challenge from Mexico, but it is included in determining Mexico’s attendance in this plot.

There were 23 countries that reported Premier Challenge results. One thing you can immediately take away from this plot is that PCs in Canada and the USA are some of the poorest attendances globally. That said, direct comparisons between countries isn’t entirely fair, considering some countries treat their few PCs as National-level events with impressive prizing.

As far as total Premier Challenges by country, the following is a list of countries with >2 PCs ranked by total PCs:

  1. USA = 164
  2. Canada = 41
  3. Italy = 26
  4. Colombia = 17
  5. Brazil = 10
  6. UK = 9
  7. Peru = 7
  8. France = 6
  9. Mexico = 6
  10. Germany = 5
  11. Ireland = 4
  12. Puerto Rico = 4
  13. Sweden = 3

Taking these considerations together, the USA certainly has many events that are reasonably attended, but also far more that are poorly attended. Canada is similar to the USA in that regard, but with nowhere near the number of PCs. Meanwhile, Italy, Colombia and the UK host many PCs worldwide, with quite high average attendances. Brazil also hosts a fair number of PCs, but doesn’t seem to have any that are well-attended. Mexico and Peru are noteworthy in having such high attendances while also holding more than a couple PCs over the year.

Figure 8.

Figure 9. Worldwide attendances follow the same left-skewed histogram, but better-attended overall. Average attendance worldwide (outside Canada and the USA), discounting the massive outlier that was the 316-person Premier Challenge in Mexico, was 27.70 ± 22.52  (95% C.I. = 5.18, 50.22).

You can see that the worldwide trends look similar to the trends from just Canada and the USA, but the proportional number of events between 17 and 32 Masters is actually a fair bit larger. Correspondingly, the number of events with 8 or fewer people is proportionally fewer than that of Canada and the USA. But event attendance outside Canada and the USA varies far more wildly, such that the lower bound of the 95% confidence interval is actually lower than that of Canada and the USA.

Overall though, with an average attendance almost double that of Canada and the USA, it seems like the rest of the world’s players deserve more PCs than they are currently getting. Given that PCs hardly cost the company anything prizing-wise, I’m tempted to attribute the lack of global PCs to either lack of reporting on the primarily english-speaking forums I went to, or a lack of international organizers. But I wouldn’t be surprised to find out there’s more going on behind the scenes of TPCi that prevent additional international support.

European event distribution

Figure S1.

Figure S1. A map showing premier events sanctioned throughout the EU, including Premier Challenges (red balloons), Regionals (yellow diamonds), and Nationals (blue stars). Much thanks to Ian Fotheringham and other EU organizers for the map! You can find the live map here.

Editing in here, but EU organizers got together to make a map of Premier Event locations. If it looks like you’re near one of these areas, be sure to check them out! And again, if there’s nowhere nearby, consider applying to be an organizer yourself!

Gripes about the current disparity in Championship Point access

It’s not fairrrrr

The importance of these events, as far as the community is concerned, was far greater than their attendances warranted. At the start of the year, Scott surmised that this was to promote the growth of video game tournament organizers as players petitioned local organizers to apply for Premier Challenge-hosting privileges. In other words, this year was a year for planting seeds. As the process to go from a non-organizer to a TO that can hold Premier Challenges is quite a lengthy one, we’ll only see if those seeds bloom next year. To give an idea of the process, to hold a Premier Challenge, players need to first apply to become a tournament organizer, and then hold multiple grassroots tournaments to show P!P that they are responsible and their area warrants support.

Ideally, this growth would see current areas increase in their attendance, and many new organizers expand into new areas.

However, as far as this year is concerned, when winning three Premier Challenges awards as much as a 1st place at Regionals, and 69% of Premier Challenges have at most four rounds of swiss, there’s a great disparity in the ease of getting points through these two event types. A great example of a player who did quite well at Regionals, but was unable to take advantage of Premier Challenge events is former US-National Champion Gavin Michaels (kingofmars) who is currently sitting at 252 CP with a Regionals BFL of 4, 32, 32 and a Premier Challenge BFL of 1, 2, 4, 12, N/A. This placed Gavin just outside the Top 64 in Canada and the USA for Nationals stipends, despite having a pretty solid record this season. Given a Regionals BFL of 4, 32, 32, you’d hope that would at least sit a player in the Top 64 for stipends. But such was not the case due to either access, or ability to attend events that were within travelling distance.

That said…

“There are no events nearby” – No, the problem is your perception of “nearby”

Whitney_Clefairy_Metronome

Follow Me to your nearest PC!

At Regionals, I’ve heard this comment many a time from friends and opponents. I live on Vancouver Island, and while I am a TO and host Premier Challenges there, I have to travel by bus, then boat, then bus to my nearest Premier Challenges in Vancouver BC. From my doorstep to the event, I end up travelling about 5 hours to and 5 hours from my nearest Premier Challenge event, and spend a minimum of ~$42 CAD on travel alone in doing so. Thankfully, Vancouver has a habit of holding their Premier Challenges as back-to-back events, so I can stay overnight at a friend’s place and make the trip more worthwhile. Keep in mind that Vancouver’s average PC attendance this year was 28.3 ± 11.1 Masters per tournament, so I’m heading out to these tournaments in one of the toughest fields in Canada and the USA, where (proportionally) making the Top 4 is like making Top 32 at a Regionals, i.e. a record of 6-2 at a Regionals.

Considering Regionals T32 awards 30 CP, and a Premier Challenge T4 awards 26 CP, I’d say that Vancouver is currently hitting the appropriate attendance for the Premier Challenge point payout; so by no means am I complaining about our PCs being too tough for the points. But I have no sympathy for anyone who complains about having to drive two hours to their nearest event, which statistically is likely to have an attendance of less than 17 players (where 3-1 in swiss earns you 26 CP).

Solutions to this disparity?

That said, as I just mentioned, our area holds back-to-back events which makes it far easier for me to justify travelling. While I definitely think players should be active in attending events nearby, I also think organizers should look to make it worth their time to travel. Money-wise, I couldn’t justify the ~$42 of boats and buses for only one tournament, especially trying to fill my BFL. But two tournaments is enough for me to commit.

So a message to organizers out there, back-to-back events have been a good way of promoting higher attendance in BC tournaments, and I’d recommend considering it if you aren’t doing so already.

And to the players:

In terms of their BFL, PCs are worth over half of what Regionals are worth. If you aren’t willing to travel about half the time you’d spend travelling to Regionals, I’d say you don’t have your priorities straight.

If you’re the kind of player that travels to two or three Regionals, you probably spend quite a bit of money and a lot of time travelling to Regionals events. If you attend three Regionals with an average travel time of 4 hours to the event (and thus, 4 hours back), then you spend 24 hours on travelling to Regionals in a year already. If you spent half that time attending Premier Challenges, you’d have little trouble getting 5+ events on your BFL.

Conclusions

Fwoo! Well that was fun. I even learned a bunch of things in R! I feel like I should say something at the end here that brings the whole article together, but honestly I’ve said everything I wanted to say throughout.

Canada and the USA have the best access to Premier Challenges, but don’t have great attendances overall. I was surprised to see so many states don’t actually have PCs, but considering the poor attendances in the states that already have PCs, I suppose it shouldn’t be too surprising. It was neat to visually present the best-attended states, so people can have that bit of pride about playing in one of the toughest areas in Canada and the USA. It was also neat to find out that Italy and Colombia’s PC scenes are bustling and well-attended. Considering their sizes relative to Canada and the USA, it hosts a pretty fair number of PCs and they’re all very well-attended; congratulations to the organizers responsible!

At the end of the day, increasing attendances in Canada and the USA is on both the organizers and the players. I hope I was also able to highlight the best-attended areas so players in those areas can bear their results with pride amidst the sea of poorly-attended PCs. I’ve also really wanted to write about the whole “I travel 5 hours to PCs, uphill both ways, with hail the size of cabbages pelting me!” since from my perspective a lot of players need a reality check regarding the level of these events. Their attendances may not match their importance currently, which may inspire P!P to reduce the Championship Point payout of PCs in the future; but for anyone who wasn’t attending events that were relatively nearby, the blame goes both ways. It’s up to both the organizers and the players to make a tournament happen, so be sure you’re part of the solution (and not the problem) and attend those events that are available to you!

Cheers,

Crawdaunt out

p.s. I have been updating this article as new information, if appropriate, is presented to me. New figures are added as supplementary figures given figure titles of “Figure S1, Figure S2, etc…” Much of the additional info does not change the trends in the article, and will not be added merely for the sake of completeness. However, I will be doing a brief follow-up article specifically focusing on the Alpha and Omega series in the USA.

Advertisements

5 comments

  1. Hello! i’m Rodrigo Rojo from Chile (and admin of the chilean vgc web http://www.evopokemon.com). The level of analysis in (for some people) a “trivial aspect” of the game blew my mind.

    this kind of analysis is certaintly a huge tool for the people of organized play and i beg you that you send this link to them (if they hasn’t read it already).

    About the Omega Series being the “worst attended”, i think that is because a group of players that i use to call “the droppers”: players that goes to a lot of tournaments but after seeing the results noted that their CPs are much lower than the people that is gaining their passes to world. So, they didn’t go to the last events because even if they win they can’t achive their season goals.

    But, of course, this is only my interpretation!

    Keep the good work and sorry for my bad english 😛

    Like

    1. Thanks! And actually, Chris Brown (head of VG organized play) tweeted this article, so it’s been seen for sure 😉 Yeah, that’s definitely one of the likely reasons for decreased Omega attendance.

      Cheers,
      Mark

      Like

  2. This is pretty amazing. Wow, I have like a million questions to ask you. The work you put in on these reports is like… a masters mini-research papers lol

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s