Automated Commander Power Level Evaluation

User avatar
pokken
Posts: 6629
Joined: 5 years ago
Answers: 2
Pronoun: he / him

Post by pokken » 4 years ago

Feyd_Ruin wrote:
4 years ago
I've created [PULLCODE] so you all can more easily link to specific lines, etc, as Rumpy requested.
It's a normall bbcode with line=LINE_NUMBER and the url inside.

Right now it only works with bitbucket since that's what you're using, but I can add github/etc later if need be.

Usage:

Code: Select all

[pullcode line=150]https://bitbucket.org/pokken/cmdrdeckrank/src/b95ccdb32e4f1ddf30cb36d36d8c2af6b12da90b/analyze_decks.py[/pullcode]
Result:
If you got any interest in writing a web form for us to collect synergies, lemme know ;) I'm planning to do it but I've got 7 years of web tech catching up to do. Absolutely understood if you're not just thought I'd throw it out there :)

User avatar
Feyd_Ruin
Elder Vampire
Posts: 5801
Joined: 5 years ago
Answers: 3
Pronoun: he / him
Contact:

Post by Feyd_Ruin » 4 years ago

pokken wrote:
4 years ago
If you got any interest in writing a web form for us to collect synergies, lemme know ;) I'm planning to do it but I've got 7 years of web tech catching up to do. Absolutely understood if you're not just thought I'd throw it out there :)
PMing you
To the beaten, the broken, or the damned; the lost, and the wayward: wherever I may be, you will have a home.

User avatar
pokken
Posts: 6629
Joined: 5 years ago
Answers: 2
Pronoun: he / him

Post by pokken » 4 years ago

Just minor updates today

* Added basic logging capability at a few key points
* Added support in card db for power,toughness, supertype and manacost (distinct from CMC)

User avatar
pokken
Posts: 6629
Joined: 5 years ago
Answers: 2
Pronoun: he / him

Post by pokken » 4 years ago

I added the ability to make html reports today (just for funs) and thought you guys might like a peek at what all the reports look like:

http://rpsands.com/cmdrdeckrank/reports

Taking Sunday to relax but I'll try to get to some of these outstanding action items next week.

User avatar
maeos
Posts: 64
Joined: 5 years ago
Pronoun: he / him

Post by maeos » 4 years ago

I'm sorry but I'm very technologically inept. How do I submit a deck and how do I add these .txt files to a post?

User avatar
pokken
Posts: 6629
Joined: 5 years ago
Answers: 2
Pronoun: he / him

Post by pokken » 4 years ago

maeos wrote:
4 years ago
I'm sorry but I'm very technologically inept. How do I submit a deck and how do I add these .txt files to a post?
Theres an attachments thing in the full editor but you can also just post the list in a deck tag with the commander at the top and I'll add it and link your report.

We're having some plumbing discussions about how to make it a thing that doesn't require so much manual shenanigans like a website or something but it's a long way off sadly.

User avatar
maeos
Posts: 64
Joined: 5 years ago
Pronoun: he / him

Post by maeos » 4 years ago

Hopefully i did it properly

edit: yay it worked

User avatar
pokken
Posts: 6629
Joined: 5 years ago
Answers: 2
Pronoun: he / him

Post by pokken » 4 years ago

maeos wrote:
4 years ago
Hopefully i did it properly

Sisay_Combo_Bullets.txt


edit: yay it worked
It's far from perfect but I think our script does a really good job of figuring out if a deck is medium power or higher. I would estiamte this deck as an 8 or 9 / 10. The main thing it's missing is that I haven't pointed captain sisay as a commander which is an oversight I'll rectify :)

I didn't have armageddon as a piece of interaction either, and I suspect I need to adjust some points. Also, the recruiter of the guard synergy hasn't been added,e tc.

All in all I think it's close. This is one of the top 10 decks I have rated so far, rating pretty close to Darrenhabib's korvold which I think is accurate most likely.

+-------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                            Deck Analysis - Total Decks: 116                                                           |
+--------------------------------------------------------------------+---------------------------------------------+---------+------+-------+-----------+
| Deck Name                                                          | Commanders                                  | Avg CMC |  Raw | Synrg | Adj Total |
+--------------------------------------------------------------------+---------------------------------------------+---------+------+-------+-----------+
| pokken_CEDH_Sushi_hulk                                             | TYMNA THE WEAVER,THRASIOS, TRITON HERO      |    1.74 | 1493 |   245 |   1977.89 |
| CEDH_consult-scepter-thrasios                                      | TYMNA THE WEAVER,THRASIOS, TRITON HERO      |    1.79 | 1619 |   166 |   1898.57 |
| pokken_CEDH_Opus_Thief                                             | TYMNA THE WEAVER,KRAUM, LUDEVIC'S OPUS      |    2.06 | 1557 |    85 |   1630.39 |
| pokken_Gitrog Lands                                                | THE GITROG MONSTER                          |    2.24 | 1158 |    41 |   1128.03 |
| MrMystery_GrusildaReanimator                                       | GRUSILDA, MONSTER MASHER                    |    2.69 |  978 |    15 |    796.77 |
| adnaus02                                                           | SIDISI, UNDEAD VIZIER                       |    1.17 |  748 |     0 |    780.58 |
| 4cEdric                                                            | TYMNA THE WEAVER,KYDELE, CHOSEN OF KRUPHIX  |    2.38 |  842 |     8 |    724.02 |
| maeos_CaptainSisayComboBullets                                     | CAPTAIN SISAY                               |    2.07 |  719 |    45 |    657.73 |
| DomSkies4                                                          | TYMNA THE WEAVER,BRUSE TARL, BOORISH HERDER |    2.63 |  836 |    16 |    639.24 |
| Darrenhabib_Korvold-Fae-Cursed-King-Bring-me-gifts-that-I-can-eat- | KORVOLD, FAE-CURSED KING                    |    2.82 |  844 |    39 |    637.45 |
+--------------------------------+
| maeos_CaptainSisayComboBullets |
+---------------------+----------+
|      Raw Score      |   719    |
|      Tutor Mult     |   1.2    |
|      Curve Mult     |   0.72   |
|    Synergy Score    |    45    |
|    Adjusted Score   |  657.73  |
+---------------------+----------+
+----------------------------+---------------+--------+
| Point Source               | Factor        | Points |
+----------------------------+---------------+--------+
| CAPTAIN SISAY              | Commander     |      0 |
| GAEA'S CRADLE              | CREATURE      |     37 |
| CROP ROTATION              | GAEA'S CRADLE |      8 |
| MANA CRYPT                 | Card          |    100 |
| SOL RING                   | Card          |     85 |
| MANA VAULT                 | Card          |     60 |
| MOX DIAMOND                | Card          |     40 |
| CHROME MOX                 | Card          |     35 |
| GAEA'S CRADLE              | Card          |     30 |
| MOX OPAL                   | Card          |     28 |
| CARPET OF FLOWERS          | Card          |     25 |
| SURVIVAL OF THE FITTEST    | Card          |     25 |
| LOTUS PETAL                | Card          |     20 |
| SYLVAN LIBRARY             | Card          |     20 |
| AVACYN'S PILGRIM           | Card          |     15 |
| BIRDS OF PARADISE          | Card          |     15 |
| WORLDLY TUTOR              | Card          |     15 |
| COMMAND TOWER              | Card          |     14 |
| ELVISH MYSTIC              | Card          |     10 |
| FYNDHORN ELVES             | Card          |     10 |
| CROP ROTATION              | Card          |     10 |
| ELADAMRI'S CALL            | Card          |     10 |
| FINALE OF DEVASTATION      | Card          |     10 |
| ELDRITCH EVOLUTION         | Card          |     10 |
| MISTY RAINFOREST           | Card          |     10 |
| TALISMAN OF UNITY          | Card          |      9 |
| ARID MESA                  | Card          |      8 |
| MARSH FLATS                | Card          |      8 |
| VERDANT CATACOMBS          | Card          |      8 |
| WINDSWEPT HEATH            | Card          |      8 |
| WOODED FOOTHILLS           | Card          |      8 |
| HUSHBRINGER                | Card          |      7 |
| GRAFDIGGER'S CAGE          | Card          |      5 |
| TRINISPHERE                | Card          |      5 |
| ETHERSWORN CANONIST        | Card          |      5 |
| GRAND ABOLISHER            | Card          |      5 |
| LINVALA, KEEPER OF SILENCE | Card          |      5 |
| NOXIOUS REVIVAL            | Card          |      5 |
| PATH TO EXILE              | Card          |      5 |
| SWORDS TO PLOWSHARES       | Card          |      5 |
| VEIL OF SUMMER             | Card          |      5 |
| GREEN SUN'S ZENITH         | Card          |      5 |
| EIDOLON OF RHETORIC        | Card          |      3 |
| SELVALA, EXPLORER RETURNED | Card          |      3 |
| FORCE OF VIGOR             | Card          |      3 |
| REGROWTH                   | Card          |      3 |
| SAVANNAH                   | Card          |      3 |
| BOUNTIFUL PROMENADE        | Card          |      1 |
+----------------------------+---------------+--------+

User avatar
maeos
Posts: 64
Joined: 5 years ago
Pronoun: he / him

Post by maeos » 4 years ago

@pokken im not sure if you guys added the vizier combo into the database as i was looking at the excel sheet and didn't see it.

User avatar
pokken
Posts: 6629
Joined: 5 years ago
Answers: 2
Pronoun: he / him

Post by pokken » 4 years ago

maeos wrote:
4 years ago
@pokken im not sure if you guys added the vizier combo into the database as i was looking at the excel sheet and didn't see it.
Nope but I will, good call.

User avatar
pokken
Posts: 6629
Joined: 5 years ago
Answers: 2
Pronoun: he / him

Post by pokken » 4 years ago

So in news no one is likely to be *that* interested in, Rumpy developed an algorithm and I implemented it (in a sweet object oriented way) that changes the processing time for 115 decks with 38,081 synergies from 800ms per deck processing time to 45ms. Very slight increase in pre-processing time of synergies (each synergy has to have the set of cards in it extracted which takes 0.5ms or so up front per synergy).

User avatar
Rumpy5897
Tuner of Jank
Posts: 1864
Joined: 5 years ago
Pronoun: he / him

Post by Rumpy5897 » 4 years ago

And yet this could still probably go quicker, as in local testing I had a similar algorithm do 5ms per deck :P

The reason for trying to maximise efficiency is pokken considering grinding up the entirety of edhrec with this, getting a score distribution. This could be useful for calibrating a "classic" 0-10 score, perhaps percentile based. Still, ripping through hundreds of thousands of decks is no joke, so everything needs to be efficient.
 
EDH Primers (click me!)
Deck is Kill Club
Show
Hide

User avatar
Rumpy5897
Tuner of Jank
Posts: 1864
Joined: 5 years ago
Pronoun: he / him

Post by Rumpy5897 » 4 years ago

Progress report on stuff!

Once again, hopefully for the last time this time, the dataset structure was redesigned. The query language remained the same, but now everything lives together - single cards, combos, various synergies pointed on a per-hit basis. After consulting with Feyd and pokken, the following column set was agreed on:
Synergy	Target	Points	Commander_Points	Tags
The commander points column only comes into effect if one of the cards is the deck's commander. It makes sense to have this split, Feather with Bandage is great with her at the helm but considerably less great if it's a synergy in the 99. And this way it just takes a single row in the dataset, plus it's good for boring reasons that will be explained later. Tags are manually curated flags for stuff, so Feather + Bandage would get tagged as draw.

At this point, given the hopeful finality of the dataset design, it'd be great to open the floor to various outside contributions. The other guys are working on some sort of putative ingester of synergies in a format friendlier than just keying in JSON, but I think the existing system is perfectly legible and can be typed by hand quite easily. That said, as its creator, I may be a biased :P
Comprehensive dataset addition instructions
Show
Hide
  • A dataset entry is made up of a query, and optionally a target. An example of a query without a target are giving points for a Mana Crypt, or various Kiki combos. An example of a query with a target are Monastery Mentor plus noncreatures, or letting draw and ramp tags amplify each other.
  • The system currently supports querying/targeting by: {"card", "tag", "type", "subtype", "color", "CMC", "text"}
  • If using cards as a query condition, all of the cards should be part of the query and none in the target. This makes things run faster, and if you're checking Kiki against whichever one of his friends then you'll still need to hit both of them.
  • The query/target are JSON formatted, without line breaks. The formatting is pretty intuitive, check out the current iteration of the TSV to get a feel for it:
  • Cards and tags are mutually exclusive with anything else - if you have a card query, that's that. If you check for tags, that's that. But the other stuff can be combined into complex queries, as once again evidenced by the immortal Sunforger example:
    {"card":["sunforger"]}	{"type":["instant"],"cmc":["<5"],"color":["r|w"]}	2	0	
  • If providing multiple criteria for the target, they will be evaluated in an AND fashion. So all the text, type etc. criteria will have to hold true. There's OR support via the | symbol, as shown above - instants that are white or red will clear the hurdle. There's also negation, achieved by starting a phrase to search with !, as shown here with Monastery Mentor:
    {"card":["monastery mentor"]}	{"type":["!creature","!land"]}	1	0	
Boring details about scoring synergies and fastness
Show
Hide
In a hypothetical world, the synergy dataset could grow to thousands of entries. It becomes critical to chew through that stuff quickly if an EDHREC-sized calibration is in the cards, and pretty much the only stuff that shows up in queries is cards or tags. As such, having the deck store its identified cards/tags as sets, having queries expose their desired cards/tags as sets too, and doing quick .issubset() operations is a good way to coax out good performance in discriminating which of the synergies apply to the deck. Once the query clears, the target is still evaluated using the old logic if needed, as there's complexity to it that needs to be handled. Still, that's a reasonable sacrifice to make - I've managed to get Feather with her sea of synergies to clear a fakely inflated 40k card synergy dataset in 10 ms, as the bulk of the needed speed-up comes from dismissing the initial criteria, not refining the subsequent hits.

The scoring got split up into two stages - previously, tags could look to single-card data to get there, but now need to wait for all non-tag synergies to complete. This makes sense, as mentioned Feather plus Bandage is draw. Plus the dataset also handles single card points at the same time now. So the queries with tags wait for the non-tag synergies to go through and apply tags, creating a data frame with one row per identified synergy and tags ripe for the picking, and then .issubset() themselves against that column. There's also the initial check against a master union of all the deck's identified synergy tags to similarly weed out any tag queries that won't resolve in the interest of run time.
 
EDH Primers (click me!)
Deck is Kill Club
Show
Hide

User avatar
pokken
Posts: 6629
Joined: 5 years ago
Answers: 2
Pronoun: he / him

Post by pokken » 4 years ago

Rumpy5897 wrote:
4 years ago
<cool stuff>
Thanks again for putting up with sharing a git repo with me =)

edit to say: Also, set operations are insanely fast and cool in python.

User avatar
TheAmericanSpirit
Supreme Dumb Guy
Posts: 2237
Joined: 5 years ago
Answers: 1
Pronoun: he / him
Location: IGMCULSL Papal Palace

Post by TheAmericanSpirit » 4 years ago

You folks amaze me. What a wonderful thing you're making! I can barely work bb code so I cannot offer any help of substance, but know that you all have impressed and astounded this American with your efforts. I hope that serves some moral support and I am rooting for all of you!
There's no biscuits and gravy in New Zealand.
(Except when DirkGently makes them!)

User avatar
pokken
Posts: 6629
Joined: 5 years ago
Answers: 2
Pronoun: he / him

Post by pokken » 4 years ago

Here's a sheets viewable version of the current TSV. If anyone wants to add to it please let me know and I'll get you an edit link or you can just work on your own version.


User avatar
pokken
Posts: 6629
Joined: 5 years ago
Answers: 2
Pronoun: he / him

Post by pokken » 4 years ago

I invite you all to give a try to my completely in progress 0.3alpha totally partially there and very ugly draft website where you can submit your deck -- note I do not save the decks at this time, just will give you your current ranking :)

http://cmdrdeckrank.rpsands.com/

Site may not be fully up for everyone since I had some dns updates propagading, and I've got tons of styling work to do on it. First major thing I want to do is make it a webservice as well so it can be interacted with programmatically and get a json response back.

TODO list
* store decklists (which means I need to add a name field, or randomly ID things
--need to add quite a bit more safety, as is it's pretty hard to blow it up but really don't want people filling up my storage with crap maliciously, so need to think bout that. Might be better just not to store.
* Fixing the stylings on the tables, ptable dumps them without style tags so gotta do some hacking on that.

note: Json format is done but I haven't figured out how to do the documentation stuff yet; it's a bear because I'm not using one of those sweet autodocumenting frameworks like swagger, but I'll figure something out. It's very easy

Summary of json
* must have accept:application/json and content-type:application:json
* decklist goes in the post body in the format attached
* url is the same: http://cmdrdeckrank.rpsands.com/

User avatar
pokken
Posts: 6629
Joined: 5 years ago
Answers: 2
Pronoun: he / him

Post by pokken » 4 years ago

Squared some bugs with the handling of utf-8 characters, though will still have to use the explicit spelling in Magic like LIM-DÛL'S VAULT for it to work -- that even was not working until today.

I've got some leads on making some friendlier matching for those characters that I will look at soon.

User avatar
Rorseph
Compleat Fool
Posts: 147
Joined: 5 years ago
Pronoun: they / them
Location: The Sixth Sphere of Phyrexia
Contact:

Post by Rorseph » 4 years ago

@pokken - Thanks for the web interface!

I've been doing a bit of monkeying around with assorted and I'm really digging how the curve multiplier works. As an example, I've got a version of your MW Landfall deck that I've been tinkering with on and off for a while that's a good 300 points weaker than yours in the raw score, but because my curve is "better" than yours, it's only 60 points behind with the converted score. I'm not sure it's the best way to account for good deck building practices, but it's definitely a good way.
"From void evolved Phyrexia. Great Yawgmoth, Father of Machines, saw its perfection. Thus The Grand Evolution began."
—Phyrexian Scriptures


Aurelia | Maelstrom Wanderer | Primer: Thassa | Uro | Primer: Volrath

User avatar
pokken
Posts: 6629
Joined: 5 years ago
Answers: 2
Pronoun: he / him

Post by pokken » 4 years ago

Rorseph wrote:
4 years ago
pokken - Thanks for the web interface!

I've been doing a bit of monkeying around with assorted and I'm really digging how the curve multiplier works. As an example, I've got a version of your MW Landfall deck that I've been tinkering with on and off for a while that's a good 300 points weaker than yours in the raw score, but because my curve is "better" than yours, it's only 60 points behind with the converted score. I'm not sure it's the best way to account for good deck building practices, but it's definitely a good way.
My intention with the curve multiplier is to come up with a modification for it to understand how many ramp cards you're playing and adjust the curve multiplier for that. It's just tough getting it right :)

I need to adjust for reanimation spells in it as well as lots of decks are boned because 4-5 reanimator targets destroy their CMC.

One thought I have is to basically apply a sliding scale based on count of 'cheat,' 'reanimation' and 'ramp' effects but getting the numbers right on that and making sure I don't miss anything is going to be very hard.

It's possible that just tightening the curve to something like 0.8 to 1.0 instead of 0.4 to 1.0 as it is now might work better.

User avatar
Rorseph
Compleat Fool
Posts: 147
Joined: 5 years ago
Pronoun: they / them
Location: The Sixth Sphere of Phyrexia
Contact:

Post by Rorseph » 4 years ago

pokken wrote:
4 years ago
My intention with the curve multiplier is to come up with a modification for it to understand how many ramp cards you're playing and adjust the curve multiplier for that. It's just tough getting it right :)
Good models that aren't over-fitted are a serious hassle for sure! Totally worth it, though.

I'm still too green in the applied stats area to have more than a gut feeling, but tightening the curve is probably a good place to start.
"From void evolved Phyrexia. Great Yawgmoth, Father of Machines, saw its perfection. Thus The Grand Evolution began."
—Phyrexian Scriptures


Aurelia | Maelstrom Wanderer | Primer: Thassa | Uro | Primer: Volrath

User avatar
pokken
Posts: 6629
Joined: 5 years ago
Answers: 2
Pronoun: he / him

Post by pokken » 4 years ago

Rorseph wrote:
4 years ago
pokken wrote:
4 years ago
My intention with the curve multiplier is to come up with a modification for it to understand how many ramp cards you're playing and adjust the curve multiplier for that. It's just tough getting it right :)
Good models that aren't over-fitted are a serious hassle for sure! Totally worth it, though.

I'm still too green in the applied stats area to have more than a gut feeling, but tightening the curve is probably a good place to start.
Indeed. I had like two stats class in undergrad and my math stops at calc 2 20 years ago so welcome all the ideas on this subject.

An easier thing to do might be to add a flat point total based on curve (e.g. 300 / sqrt(curve)) or something. Also considering using square and cube roots of the curve as a divisor to a 1/x calc,.

The reason I thought of it as a multiplier first is because Savannahlions.dec would probably score weirdly high if it was a flat number, but I'm really torn.

Happy to hear any brainstorms :)

User avatar
Rumpy5897
Tuner of Jank
Posts: 1864
Joined: 5 years ago
Pronoun: he / him

Post by Rumpy5897 » 4 years ago

I like the current shape of the CMC curve, could be messed with to get the bottoming out point right. I think that the tag system can come into play here - have ramp synergise with ramp, for a 1 point payoff. This way the more ramp you have, the higher you shoot up. And if you have a ton of ramp that feeds a crazy high curve, it kinda balances out.

The tag system is gonna be very powerful at conveying various gameplay things. Tag Elesh Norn as a "bomb", tag some reanimation spells as such, have reanimation synergise with bombs. Boom, gameplay information!
 
EDH Primers (click me!)
Deck is Kill Club
Show
Hide

User avatar
pokken
Posts: 6629
Joined: 5 years ago
Answers: 2
Pronoun: he / him

Post by pokken » 4 years ago

Rumpy5897 wrote:
4 years ago
I like the current shape of the CMC curve, could be messed with to get the bottoming out point right. I think that the tag system can come into play here - have ramp synergise with ramp, for a 1 point payoff. This way the more ramp you have, the higher you shoot up. And if you have a ton of ramp that feeds a crazy high curve, it kinda balances out.

The tag system is gonna be very powerful at conveying various gameplay things. Tag Elesh Norn as a "bomb", tag some reanimation spells as such, have reanimation synergise with bombs. Boom, gameplay information!
One of the issues with the ramp+ramp synergy as addressing this is that the penalty for curve is *so high* that it halves your bonus, and I don't think it's quite representative. But it's possible a ramp+ramp and ramp+<cmc>6> type synergy could do it?

It's probably correct to try to balance it with synergies. I think as is though that certain powerful strategies are probably a bit underpointed; my MW deck actually dominates my Ephara deck if they're both at the same table. It's the particular example that jumps out at me the most -- that deck does not really lose at mid powered casual tables because it's so oppressively inevitable.

It's also possible that some targeted bonuses for commanders that let you cheat on CMC could be helpful; things like MW + CMC 4-7, and Golos with CMC > 4, etc.

Fundamentally I think that the shape of the curve is fine but the magnitude is too high, and I need to poke at it to see how to fix that. Since it's under 1 I could possibly take the square root of it to de-amplify it a little-- a .6 curve multiplier becomes .77, and .4 becomes .63?

User avatar
pokken
Posts: 6629
Joined: 5 years ago
Answers: 2
Pronoun: he / him

Post by pokken » 4 years ago

Bugs squashed today
* lim-dul's vault problem *should* be okay, I think any combination of unicode or not should work and compare correctly. lim-dul's vault is officially both lim-dul's vault and LIM-DÛL'S VAULT.
---This is working both for card names and for synergies. If you make a synergy with LIM-DÛL'S VAULT in the card name it will work the same as if it were LIM-DUL'S VAULT.
---This SHOULD apply for all unicode character weirdness but let me know if you put something in that breaks it.

* Reporting fixed to report the full name of synergy whether it's commander or otherwise (previously commander synergies were obscuring the name, and synergies with multiple-types such as sunforger were not reporting the full criteria name)

Enhancements
* Added support for split card nomenclature in synergies, so boom // bust will now work in synergy definitions as will boom.

* Updated curve formula

Hopefully a little more user friendly. :) The latest report on our existing dataset with existing synergies is posted in the OP.

Post Reply Previous topicNext topic

Return to “Commander”