Having Your Voice Heard

[FireStorm4056]

One other quick follow-up: Again, I'm not saying data can't be extremely useful, especially for us players. Simply pointing out that, considering the RC is constrained in time and resources, I really don't think this is the train to board. It's far more work just to reach conclusions that aren't any more objectively valid than other approaches (and considering the whole point of data analysis is usually to arrive at objective conclusions, this seems like a major resource sinkhole).

[pokken]

So yes, the PE analysis "fits" the PE ban, but that procedure also ONLY fits the PE ban (and basically nothing else - a statistical error called "overfitting"). Practically every commander has cards with over a 50% inclusion rate, and yet very very few of them are considered broken, much less ban-worthy. If we were to take all the most played cards, assign them random names, then look at ONLY the random names, colors, and play rates, could we construct a good banlist? No. This tells us the analysis itself doesn't have any general predictive power. Rather, the analysis is artificially constructed to arrive at some answer we want it to "tell us."

EDIT: That's not to say PE shouldn't have been banned. There are good reasons to ban it... but none of them can be deduced from the EDHRec data. Consider the EDH Philosophy document's reasons for why cards could be banned:

- Cause severe resource imbalances
- Allow players to win out of nowhere
- Prevent players from contributing to the game in a meaningful way.
- Cause other players to feel they must play certain cards, even though they are also problematic.
- Are very difficult for other players to interact with, especially if doing so requires dedicated, narrow responses when deck-building.
- Interact poorly with the multiplayer nature of the format or the specific rules of Commander.
- Lead to repetitive game play.

None the data on EDHRec directly tells us about a single one of these bullet points, even though it's by far the best dataset we currently have.

So we worked on the PE ban for years when one of the biggest ban criteria was "problematic casual omnipresence" so our goal in analyzing the EDHRec data (LONG before it was banned, mind you) was to understand if it was indeed casually omnipresent.

The data on Birds of Paradise for example would show that it is casually omnipresent, guaranteed.

So the statistics are fairly useful on that front; the 'problematic' side was something that required a lot more analysis. One angle we approached it from was understanding how frequently Paradox Engine was a violator of the old criteria of "too much mana too quickly" -- TL;DR; it was about as bad as Metalworker in a typical deck.

If you want to use the EDHRec data to fit anything to the new ban criteria I think that would largely be impossible. But the old ban criteria it was pretty useful

But I do think that new data could surely shed some light on those various new criteria. "allows players to win out of nowhere" could be addressed with a large enough sample size of games ended with card X as the primary factor. Especially if you compared it to other known offenders.

Say you had a huge pool of games and you saw that Deadeye Navigator was the deciding card in a game 2% of the time. Then you compared that to say, Paradox Engine and Kiki Jiki, MirrorBreaker and you saw that Paradox Engine was a game ender 4% of the time, and Kiki 1%.

That'd be meaningful into even if the dataset wasn't perfect. And it could be gathered.

[FireStorm4056]

If you want to use the EDHRec data to fit anything to the new ban criteria I think that would largely be impossible. But the old ban criteria it was pretty useful

This is the major sticking point for me.

There are lots of ways we can use data to "confirm" things we already know, but if we already know them, is that really the best investment of time and resources? On the other hand, there's been no convincing argument to show that a big push for data collection/analysis would provide any NEW information that is unique, reliable, or actionable.

Also, the numbers you just gave sounded nice in theory. But what if they playrates were really: Paradox Engine (1%), Kiki-Jiki (4%), Deadeye (2%) and Craterhoof Behemoth (8%)? Would the banlist changes be any different? I don't think so, because the PE issues still persist and that's ultimately why it was banned. Put another way, although these numbers "sound" nice, they don't provide any unique insight and are simply a pretty citation that conveniently "confirms" a conclusion we've already arrived at by a different route.

[pokken]

If you want to use the EDHRec data to fit anything to the new ban criteria I think that would largely be impossible. But the old ban criteria it was pretty useful

This is the major sticking point for me.

There are lots of ways we can use data to "confirm" things we already know, but if we already know them, is that really the best investment of time and resources? On the other hand, there's been no convincing argument to show that a big push for data collection/analysis would provide any NEW information that is unique, reliable, or actionable.

Also, the numbers you just gave sounded nice in theory. But what if they playrates were really: Paradox Engine (1%), Kiki-Jiki (4%), Deadeye (2%) and Craterhoof Behemoth (8%)? Would the banlist changes be any different? I don't think so, because the PE issues still persist and that's ultimately why it was banned. Put another way, although these numbers "sound" nice, they don't provide any unique insight and are simply a pretty citation that conveniently "confirms" a conclusion we've already arrived at by a different route.

Well, I didn't go into it assuming that I would find casual omnipresence. I actually expected to find that PE was not played in casual decks very much. I was pretty surprised tbh.

With the new hypothetical, I was talking about winrate not playrate; but with those hypotheticals it does answer this question:

If you looked at the numbers and saw:
8% of games end with Craterhoof
1% of games end with Paradox engine
2% of games end with deadeye
4% of games end with Kiki Jiki

I think those numbers are very unlikely but if they were the case...

It would suggest that PE is not really winning games out of nowhere as much as people think.

Much like most nebulous data it doesn't really prove anything but it does give some useful points of comparison, and could surely make some people question their assumptions. I know I would question my assumptions if Deadeye was ending fewer games than Kiki Jiki.

[bobthefunny]

...but if those 1% of PE games feel 100x worse than the 4% of kiki jiki games...


At the end, this format is about communal enjoyment. How do you measure that?

[pokken]

...but if those 1% of PE games feel 100x worse than the 4% of kiki jiki games...


At the end, this format is about communal enjoyment. How do you measure that?

Same way people measure other subjective things like well-being. Rigorous questionnaires with a broad audience that you do your best to make representative.

The "salt score" poll that EDHrec did was a great starting point. More stuff like that.

I suspect that the Salt Score is probably more accurate in assessing the reality of how annoying cards are to the EDH population than almost any number of discussions.

[Sheldon]

I didn't know the salt score existed until very recently. Is it easy to find a comprehensive list of the saltiest cards?

[NotMyOwlbear]

https://edhrec.com/top/salt

Mass land destruction, stax, and extra turns all tend to be heavily represented.

The one I always found odd was Cyclonic Rift, but to me that is more in how it is used. I have always been a big defender of it, partially because of how I end up using it as part of the glue that gives some of my slower, clunkier decks a fighting chance. So much fair use potential.

[HoffOccultist]

[mention]Sheldon[/mention] :

Here's a link to the top scorers: https://edhrec.com/top/salt

Most of the top there are stax pieces and MLD, which are known quantities of annoyance, but also don't (as far as I can tell) seem to show up in lower power pods so much (and serve important purposes in heavily tuned pods).

Interestingly, a lot of the other high salt score cards are things that seem like the top of splashy high-mana spells that EDH is known for.

I don't know how long the scores were collected, and what places were polled (I remember seeing it on r/EDH on Reddit), but it still provides a benchmark--but should also be taken with a grain of salt.

[cryogen]

@Sheldon :

Here's a link to the top scorers: https://edhrec.com/top/salt

Most of the top there are stax pieces and MLD, which are known quantities of annoyance, but also don't (as far as I can tell) seem to show up in lower power pods so much (and serve important purposes in heavily tuned pods).

Interestingly, a lot of the other high salt score cards are things that seem like the top of splashy high-mana spells that EDH is known for.

I don't know how long the scores were collected, and what places were polled (I remember seeing it on r/EDH on Reddit), but it still provides a benchmark--but should also be taken with a grain of salt.

I've never heard of that either, so maybe word didn't spread quite as much. I'll add it to my survey next year.


Edit: I just looked at the list, and I'm not thrilled by it. As a scale of 1 to 4 there isn't much wiggle room, and it says it was done only a few months ago but cards that have been banned for years are ranked on it? Do people still run into Worldfire a lot? It also doesn't give me any real insightful data. Yep, people get salty with stax, no surprise there.

[pokken]

The point about it is if you get a half a million votes the odds of there being serious sampling error are pretty low. I doubt there are more than a couple million edh players in America.

It surely has its issues with rigor. But you get one or two good psychology researchers writing your questions and get it out to a this big an audience and you'll get very informative data.

One other quick follow-up: Again, I'm not saying data can't be extremely useful, especially for us players. Simply pointing out that, considering the RC is constrained in time and resources, I really don't think this is the train to board. It's far more work just to reach conclusions that aren't any more objectively valid than other approaches (and considering the whole point of data analysis is usually to arrive at objective conclusions, this seems like a major resource sinkhole).

This is something I seriously disagree with. Data analysis is probabilistic. You use the data to decide which option has a higher probability of being correct not which one is objectively correct. It's really hard to reach objective conclusions especially initially.

We use data analysis to give us new ideas as well. Rarely perfect. But you can learn new things and find new concepts.

[FireStorm4056]

The point about it is if you get a half a million votes the odds of there being serious sampling error are pretty low. I doubt there are more than a couple million edh players in America.

Obtaining half a million votes is not exactly easy, given /r/EDH only has 82,000 (even /r/magicTCG only has 330k). Also, you'd still not be getting a representative sample of the EDH population. No matter how many votes are cast, you are still drawing from a very distinct subset of EDH players; namely: players dedicated enough to MTG to find online resources/communities, AND actively participate in those communities; AND find the articles/discussion on the banlist; AND actually read the articles/discussion; AND go the distance to complete a survey. Sure, there will be plenty of people that do this, but they've inherently self-selected multiple times before they even open the survey to begin with, hence why sampling bias is a huge issue here.

As an analogy: If you poll 10,000 Hawaiians about the US economy, you're not going to get answers that are representative of all 50 states. If you then refocus your efforts and poll 100,000 Hawaiians, you're STILL not going to get answers that are representative of all 50 states. High raw response rate doesn't magically grant statistical validity when you still have systematic sampling error (and again, no way around that for a casual, but massively popular, format like this).

This is something I seriously disagree with. Data analysis is probabilistic. You use the data to decide which option has a higher probability of being correct not which one is objectively correct. It's really hard to reach objective conclusions especially initially.

Sure, but that data/analysis needs to be reliable in the first place; it needs to be a demonstrably accurate predictor in many different cases. Carefully cherry-picked data that only works for a single example (and nothing else) is, at best, a huge red herring.

[pokken]

Obtaining half a million votes is not exactly easy

No it's not but Edhrec did it.

As an analogy: If you poll 10,000 Hawaiians about the US economy, you're not going to get answers that are representative of all 50 states. If you then refocus your efforts and poll 100,000 Hawaiians, you're STILL not going to get answers that are representative of all 50 states. High raw response rate doesn't magically grant statistical validity when you still have systematic sampling error (and again, no way around that for a casual, but massively popular, format like this).

Self-selection of "people who get on the internet often enough to see a poll, or hear about it" is not anywhere near comparable to selecting a single state and trying to extrapolate to the whole country.

It would be more comparable to asking 10,000 hawaiians via the internet about economic conditions in Hawaii which will have some sampling error but nowhere on the same scale. Sure, the people who are on the internet might be inclined to answer slightly differently sometimes. But I doubt on the scale you're thinking.

Sure, but that data/analysis needs to be reliable in the first place; it needs to be a demonstrably accurate predictor in many different cases. Carefully cherry-picked data that only works for a single example (and nothing else) is, at best, a huge red herring.

By what metric would you say data is reliable or not reliable? And does data necessarily need to be predictive or could it just be suggestive of further areas of study and discussion?

This is not a situation where your data needs to prove anything.

For example, correlations don't prove anything but it doesn't mean we stop using them - they're useful even though they're imperfect an they don't prove what people often think they do.

[fusemaker4]

So maybe the other way to ask the question is "what data points that aren't win rate are important?"

Still have no idea how to collect this information but this can be an interesting thought exercise (idea discussed by the people at Commander Theory podcast):

On time and turns.
Time/minutes/turn played can be important to some people.
1) Which player had the longest average turns in the game? (Does the player with the most minutes played win the game?)
2) Average minutes per turn the winning player had compared to the other players?
3) What spells lead to more minutes per turn for the winning player?
(ie. does Paradox Engine, Prophet of Kruphix lead to one player playing more minutes per game compared to any other player? Or does stax really create longer turns for one player compared to the rest of the table?)
4) What spells lead to a significant imbalance in number of minutes a player played compared to the rest of the table?
5) What are the most commonly played spells during the longest turns in the game?
(This can help identify cards in pivotal moments in games)

On interaction (One can look at the data on the longest turn or the turn with the most action as this can be the pivotal turn of the game)
1) What was the turn with the most action?
2) What are the cards commonly played during these turns?
3) What spells/permanents cast must be answered/countered/removed by other players immediately? Which cards during these pivotal turns don't last the turn cycle?

On tutors:
1) Which spells are most commonly tutored?

[pokken]

My quick list of potential questions that I would care about

Game
1. How many players? (Int)
2. What were the archetypes and commanders? (list of card+archetype, archetype from a set)
3. Who won, and how? (player + how)
a. If a combo, what cards were in it (list of cards)
4. Who went first?
5. What was the last card played?
6. How much mana was the winner capable of making on the last turn? (approximate)
7. What card do you think influenced the outcome game the most? (card)
8. What card did you enjoy seeing played the most?
9. What card did you enjoy seeing played the least?
10. What card did you play that you liked the most?
11. What card did you play that you liked the least?
12. On a scale from X to Y, how much did you enjoy the game?
13. Who spent the most time as the active player?
14. How long was the game in minutes?
15. How many turns were there?

(Just spitballing ideas there)

[TempusFugit]

...but if those 1% of PE games feel 100x worse than the 4% of kiki jiki games...


At the end, this format is about communal enjoyment. How do you measure that?

The problem is that if casual/local groups dislike, if it is that much problematic for them - and lots of legit arguments as to why PE is not as problematic as RC makes it out to be can be made- PE they can very easily, based on rule 0, discussed and then baned in those metas/playgroups.

As I have argued elsewhere (in Post PE metagame in Competitive section of the forums), EDH or cEDH people are angry or dissatisfied not with the PE ban itself, but what appears to be inadequate reasoning behind it, and what seems to ne arbitrary decisions (at least to some people) of RC.

In its essence, the problem is that most "problems" that require a "ban" can be solved within individual groups based on local bans, but that competitive players, or at least that prefer optimized or %75 players, especially in GPs/magicfests and elsewhere, have to adhere to a non-competitive banlist in an otherwise mostly competitive/prize focused environments.

[pokken]

And people keep responding to that "rule 0 it" argument that the job of the banlist is to with the lightest touch possible make what the philosophy describes as "good gameplay" happen as much as possible.

In practice I believe the banlist succeeds admirably at making it very likely your deck is fine to play almost anywhere if you're sitting in the 50-75% area of deck construction.

It's a regular assumption that everyone is part of a playgroup as well. This is not the case. I couldn't talk to the 50 guys who play at my shop and I can't get 5 of them to agree on what to eat for dinner so I don't hold out a lot of hope for a "house ban" on a card that lots of people like but is toxic to the metagame.

There was a reddit poll recently that I really found interesting that showed that the common assumption that everyone is part of a private playgroup to be most likely wildly incorrect - most people seem to play at their shop with whoever happens to be there.

Rule 0 doesn't really work great at policing high powered cards that are broadly likely to see play in mid-powered decks. That was PE's problem. Rule 0 can't handle cards like;

• recurring nightmare
• prophet of kruphix
• primeval titan
• sylvan primordial
• griselbrand

Rule 0 can handle stuff like "do you want to play Cedh with me?" But you can't rule 0 a highly desirable card out of a metagame bigger than a handful of people.

[BaronCappuccino]

Rule 0 is a lot better for getting a card unbanned than it is for getting a card banned. It's a lot easier to manage your own expectations, bringing the banned card you'd like to include in your deck, as well as its replacement, than it is to expect others to bring replacements for whatever card won the last game. I knew a guy from when I had a shop that used to play a Xenagos deck fairly pimped out and wanting nothing - foiled, etc - and he'd always try to keep Prime Time in it. Nobody ever really cared.

[HoffOccultist]

Obtaining half a million votes is not exactly easy

No it's not but Edhrec did it.

So I think it's important here to point out a very pertinent limitation of this data--namely the way it was gathered. While it was undoubtedly quite the undertaking by EDHRec, it's also worth noting that those 500k votes are not from 500k individual voters, as the way the page worked was that it just showed you random cards and you voted on them. I got the same card multiple times in a single session, and got those same cards up again another time I visited the page (along with other cards mixed in, of course--it wasn't just the same set of cards each time). Because it wasn't limited to a single score per card per person, a particularly interested/vocal person or group could have skewed the data.

Which is all to say it's a very interesting set of information, but at the end of the day it has enough problems that I really doubt it's useful for anything at all.

[cryogen]

It's also worth noting that salty about a card doesn't necessarily mean you want it banned. Stasis came in first place in their salt list, yet only 10% of people who took my poll wanted to see it banned.

[pokken]

It's also worth noting that salty about a card doesn't necessarily mean you want it banned. Stasis came in first place in their salt list, yet only 10% of people who took my poll wanted to see it banned.

Yeah, I definitely appreciate the deficiencies in methodology and also get that we shouldn't interpret salt as banning.

But I really like surveys. it's crazy how much we're learning from your data set even. Now imagine it ran for a couple months and wizards put the link in insert cards prominently in the commander decks.

Wizards is making very successful decisions based on massive amounts of data. Commander can be data driven too. Though I think it proper to be more conservative than wizards because of how the flow of standard allows them to "fail fast" more than commander.

[Greedzito2]

Like many, I've created an account to tell my thoughts on this subject.

In my opinion, having my voice heard has nothing to do with the sour taste that recent events left in my mouth, the way I see it the problem are the incoherent sayings for both the RC and the CAG. Yesterday for example Shivam said in loadingreadyrun's show taptapconcede that he brought up Iona as a possible ban because he likes to play mono colored decks and eventually he gets iona'd. That goes against what was said regarding "we don't do bans based on what happens in our local groups" and I feel lied to.

Lastly, I can't accept the way you guys seem to hold my cards as "hostages" (Iona and coalition victory) I think it's the worse thing that's even been done to a ban list and if someone is capable of looking at cards such as Iona and not playing them because people wouldn't like the outcome they will be able to do so with or without it banned, and if they won't be able to do so they are just gonna substitute them by other miserable cards.

[RxPhantom]

Like many, I've created an account to tell my thoughts on this subject.

In my opinion, having my voice heard has nothing to do with the sour taste that recent events left in my mouth, the way I see it the problem are the incoherent sayings for both the RC and the CAG. Yesterday for example Shivam said in loadingreadyrun's show taptapconcede that he brought up Iona as a possible ban because he likes to play mono colored decks and eventually he gets iona'd. That goes against what was said regarding "we don't do bans based on what happens in our local groups" and I feel lied to.

Lastly, I can't accept the way you guys seem to hold my cards as "hostages" (Iona and coalition victory) I think it's the worse thing that's even been done to a ban list and if someone is capable of looking at cards such as Iona and not playing them because people wouldn't like the outcome they will be able to do so with or without it banned, and if they won't be able to do so they are just gonna substitute them by other miserable cards.

I'll be honest, I hate Iona. She should've been banned a long time ago. But, I hadn't seen it in a while and wasn't on my radar much anymore. My anecdotal evidence is meaningless, though. Ten years of legality is a long time to wait. If what you say about Shivam is true, then that's a bit disconcerting.

Coalition Victory, though, can stay banned. It just...sucks. It's such a lame way to win, and I imagine it would get old after the first few times.

[MRHblue]

In my opinion, having my voice heard has nothing to do with the sour taste that recent events left in my mouth, the way I see it the problem are the incoherent sayings for both the RC and the CAG. Yesterday for example Shivam said in loadingreadyrun's show taptapconcede that he brought up Iona as a possible ban because he likes to play mono colored decks and eventually he gets iona'd. That goes against what was said regarding "we don't do bans based on what happens in our local groups" and I feel lied to.

So thats the starting point of a discussion, not the end point. If lots of people on the RC and CAG have that same thing happen, then they have reports from a bunch of other meta's say 'Iona is a problem', what else do they do with that information?

And how do they cultivate a ban list without considering their own interactions with the cards? Shivam isn't saying 'I got locked out once, lets ban it'.

[cryogen]

For what it's worth, back in May there was an episode of the weekly wizards podcast (whatever its called) that had most of the CAG on it, and round table style they asked what they would ban. Every single one of them said Iona.