Hey, can I ask how you pull the data into your dashboard from ao3? I’m just another curious data nerd - thanks!
Sure! I use a webscraper (Beautiful Soup) to scrape all the fandom category pages (e.g. https://archiveofourown.org/media/TV%20Shows/fandoms) and basically use a bunch of string functions to parse through a giant piece of HTML to pull every Fandom Name and it's associated fic count.
(Note: some people have pointed out that occasionally the number on the Category page does not match the number at the top when you click through and I have no idea what causes this discrepancy. Take it up with AO3 lol. I just use whatever the Category page says.)
I have the scraper set to run once a week on Monday, so I just match up the new count with previously collected data and calculate fandom size change from there! I store the raw data in a bunch of nested dictionaries in a JSON locally, which then powers a Google Sheet where the calculations are run. The dashboard is then connected to the Google Sheet as a data source!
I pretty much taught myself Python just for this project initially (although now I use it at work too and am getting a pay raise for it - thanks ADHD!) so it's definitely not the most elegant solution, but it works!
The only part of the workflow that isn't fully automated is the weekly Tumblr post - I have a script that creates the post, but if I set it to run automatically, the images never come out right. So I have to remember to click the button once a week, which I'm not great at (thanks ADHD), so that's why they don't always come out on Mondays.
Hope that helps! LMK if you have any questions!