Howdy! arranged.team is a project from the friends behind replay hosting site highper.ch, iconic mountaineering strat highperching, and (back in the day) preeminent arranged team collective Clan TOWA. Our native gametypes are 3v3 and 4v4 AT, and as the war3 playerbase has dwindled it has become increasingly difficult to actually find games in these purest of gametypes.
In attempting to determine the hottest times for specific gametypes, it was clear that we'd have to gather information about all games. We only really care about 3s and 4s AT, but since we have all of this data, why not publish information of a broader scope? War3 has been on the decline for quite a while, but recently such events as significant patches by Blizzard, Grubby's return to the game, and dedicated tournament coverage appear to be imparting at least some new life to the game. This tool will, hopefully, help to quantify the renewed interest in the game.
Tons of thanks to iggyfisk, Mata, and pecanswa for helping build this. Additional thanks to the rest of the crew for playing war3 et al with me for 11 years: timg4strok, Blinn, Iceberg-Slim, dnkles, Thormjun, and Shamshel.
A:Here! GameIDs are sequential, so it's pretty easy to scrape them all.
Q:How does this all work?
A: Python 3, BeautifulSoup, SQLAlchemy/SQLite, d3.js, NVD3, and bash/cron. We first traverse each game's detail page in order (in wigscraper.py), grabbing the GameID, date/time, gametype, game length, and map and shoving it all into a database. Once we have enough games, we can generate useful collections of data to be interpreted and displayed (done in wigquery.py). After millions of games have been crunched to useful data in the form of CSVs, a static frontend displays the visualizations using d3.js, mostly via NVD3.
Q:How often is the data updated?
A: Daily. The previous day's games are fetched and new queries are run starting at 08:00 UTC. The previous day's data will be visible at around 09:00 UTC.
A: Logging each game requires an http request to the Ladders page, and with ~3.8 million total games played in 2016, that's a lot of requests! I'm trying to be considerate of Blizzard by not parallelizing the scraping process and blasting their servers with hundreds of requests per second, so it'll take some time - I'd estimate that 2016 data will be ready sometime in August. It's possible that eventually I'll have the entire ladders history (all the way back to June 2008), but that's about 150 million total games - we'll see. Currently an EC2 t2.small takes about 1 minute per million games to crunch down to CSVs. Some rearchitecture may be required...
Q:Can I play around with your data?
A: Sure! The daily CSV exports are here, and the SQLite database files are on Github (though they won't always be up-to-date).
Q:Does this represent RoC, or TFT, or both?
A: All data is TFT-only, with the exception of the "All Realms" time series which counts both RoC and TFT games. See the Statistics page for details on how games are classified as RoC or TFT.
Q:How are you accounting for lossbots/icon grinders?
A: By ignoring games shorter than 3 minutes. This is not foolproof, but this whole endeavor is pretty seat-of-the-pants.
Q:What about w3arena and netease?
A: I know - they're not included, this is battle.net data only. If you're aware of an API or datasource for the alternate ladders that could be parsed to [unique gameid, datetime, gametype, gamelength, map], let me know or send a PR and I'd be happy to add them!
Q:What is "Wig"?
A: "Warcraft" "III" "Games". e.g. "Do you want to wig?", "I'm wigging out", "It's prime time, there are wigs to be had"...
Q:Heatmaps are weird on gametypes with fewer than 10 total games played?