Visualizing AO3 data

Scraping AO3 to visualize data about fandom, my own writing, or anything I want to really. 

[visualizing-archive-data]

Want to visualize your own data or fandom data?

Send an ask! Currently, fandom analytics will only be accepted via ask.

Update 26/8:

If you’re interested in your own data, submit this (in development) Google form to pick and choose what you want to see.

What do you want to see?

Your works, bookmarks, text from a specific work, comments from a specific work? All possible! You’ll have to pick one and tell me though.

What about your data interests you?

Want to see the adjectives/adverbs that people use to describe their reactions to your work? Want to see what nouns you use most often in your writing? Or what tags you use the most often in a certain fandom? Which authors works you bookmark the most? Pretty much anything with either the information in an AO3 work blurb (title, author, characters, warnings, ratings, additional tags) or the raw text of your work or the comments left on it can be examined.

How do you want to see it?

Currently I’m only doing word clouds, but more functionality will probably come soon.

What’s your style?

Pick a font, background color, and color table for the word cloud! Follow the links for acceptable names (most font styles accepted).

How do you want to receive the images?

I’ll answer all asks privately, and direct messages in the same way. If I see something cool and SFW, I might ask if I can post it to this blog for publicity. Feel free to say no if you don’t want me to!

Do you need to give credit?

Nope! Data is free, but feel free to tag this blog!

Can you do this yourself?

Absolutely! Code is on GitHub, and it should be relatively easy to use. It’s constantly evolving though, so watch out for new commits.

Anything else?

Look, I have no way of knowing if the works or accounts you want to look at are yours or not. But don’t use this to stalk/harass/bully other people. If I find out that’s what’s going on, you’re going to be blocked.

Also, I will NEVER ask you for your AO3 password. I’ll only scrape data that is already public on the Archive.

Languages:

English/Deutsch/Español

[autocartograph]

Category Tags Over Time on AO3

@olderthannetfic comments/analysis on m/m across platforms got me thinking about how pairing prevalence might shift over time. And AO3′s handy-dandy Selective data dump for fan statisticians makes it easy-ish to compare (as least for that platform). ​

[image description: A 100% stack bar chart showing how each category tag contribute to the whole from 2008 through February 2021. The breakdown for 2020 is as follows: m/m 45%, f/m 22%, gen 16%, f/f 9%, while multi and other are too small to label, with each less than 5%. ] The chart shows the percentage each tag makes up for the total category tags used in a given year. That year is based on when the work was created. 2021 is only through the end of February. It should be noted that works can have zero category tags or all 6 or something in between. So it would not be accurate to say that “45% of all works on AO3 in 2020 were for m/m.” But you could say that “45% of all category tags for works created in 2020 on AO3 were for m/m.”  The ratio of m/m and f/m to the other tags looks roughly consistent since 2014 although it’s down this year from their highs. f/f, multi, and other have increased, mostly at gen’s expense. I’m going to ignore 2008 since there weren’t many works that year. Since then the high water mark for m/m was 2019. f/m was 2018. gen was 2009.  f/f, multi, and other are all 2021.

If you are interested in category tags on AO3, check out Older Than Netfic’s analysis and @toastystats’s femslash analysis which includes % of all works over time. Their numbers are slightly different than mine because of differing methodology - creation date vs updated date, percent of works vs percent of tags, etc.

[visualizing-archive-data]

Some pride for Pride! It’s been almost a year now since I started scraping my AO3 stats page, and here’s how the day-of-week trend stack up. I’ve broken the data into two groups: normal days where I didn’t post anything, and posting days (translucent bars, indicated by ‘new’ in legend).

[ID: Bar chart displaying average AO3 statistics (bookmarks, hits, kudos etc) by day-of-week. One could make an argument for periodicity with peaks on the weekend and mid-week, but it’s not hugely significant for this sample size. Feedback in the form of comments definitely peaks on Wednesday and Thursday, although for newly-posted/updated work, that trend is higher towards the later days of the week. The color scheme of the bars follows that of the rainbow Pride flag.]

[visualizing-archive-data]

100% prepared to bring my gay AF plots into a job interview next week

[autocartograph]

AO3 Freeform Tags Over Time

Late 2008 to early 2020

Biggest Movers (Fluff, Hurt/Comfort, Smut, Anal Sex)

Biggest Losers (Crossover, First Time, Drama, Episode Related):

Interactive chart can be found here. Check it out. It’s much nicer and well worth the hassle of leaving this place. Source is @ao3org′s Selective Data Dump for fan statisticians.  Freeform text racing bar chart inspired by Largest Fandoms on AO3 (youtube).

[kamorth]

I narrated this like a horse race announcer ANGST JUMPS INTO THE LEAD THAT’S GOING TO BE HARD TO CATCH, ROMANCE AND HUMOUR FIGHTING FOR SECOND BUT OH WAIT WHAT’S THIS? FLUFF OUT OF NOWHERE WOW WHAT A SPRINT

[visualizing-archive-data]

All my plots can look like Pride flags now and it makes me so happy

[ID: Random line and scatterplot using the Philadelphia inclusive pride flag colors as a color cycle. This is especially obvious in the legend on the right-hand side]

[fanhackers]

Call for participants in a research survey

Makenna Reaves, an udnergraduate at University of Washington is asking you (in case you are over the age of 18) to participate in a survey. This survey is being conducted for the purpose of gathering data about general opinions and attitudes toward fan creations. The survey will run until May 13, 11:45 PM, GMT-7. You can access the survey at this link. It consists of 15 multiple choice and 3 long answer questions, it takes between 10 and 45 minutes to complete. The participants will be asked questions about their age, gender identity, sexual identity, and ethnicity and their knowledge and general opinion of various fan creations and fan communities.

The survey has been checked and approved by the researcher’s Faculty’s Ethics Commission. This survey on reading engagement includes a consent form and information about participant privacy and data usage.

More information can be found by clicking on the above survey link. You can also reach out to Makenna Reaves at reaves [at] uw.edu.

[anneapocalypse]

If you don’t support AO3′s policies, then don’t post your fic there.

[samjohnssonvt]

Don’t read there, either.

[rainewynd]

..and don’t bitch about how you’re feeling excluded by the “cult of AO3” because hello, you’re against it, that’s what excluding yourself does….

[phoenix-ace]

None of this makes any sense…

[myminorityfeelsminor]

>_> its tone deaf you’re right. Criticism isn’t bad. 

[phoenix-ace]

I think that’s what it is. Folks cant be sensitive to criticism but then act like critics are “too sensitive”, the whole “you cannot utilize this site unless you 100 support it and are its primary demographic” doesnt work in any real life capacity and is extremely dismissive (and hypocritical), like we all know we didnt “exclude ourselves”…

[anneapocalypse]

I would like to clarify that this post was not made in response to people who would like to see AO3 add certain features, fix bugs, etc. You are right, criticism isn’t bad, but that’s not what I’m talking about here.

I’m responding to people who fundamentally disagree with the the philosophy at the core of the OTW and the Archive Of Our Own.

I’m responding to people who want AO3 to ban certain subject matter and certain ships. I’m responding to people who believe it is the job of the archive they utilize for free to curate content according to their tastes, preferences, and objections. I’m responding to people who are angry that AO3 does not act according to their desires.

And to those people, I say: you don’t have to use AO3, and if you feel strongly about it, perhaps you shouldn’t. 

Keep reading

[madgastronomer]

AO3′s software is completely open source. The code is here. You can set up your own site with all of its tools, but with your own rules. You can make it an entirely private site if you want. You don’t even have to be capable of continuing development on it, you can just keep updating with the things the OTW does. You just have to be capable of either maintaining it and editing the HTML, or paying someone to.

Nobody fucking does. Nobody wants to do the fucking work. People want AO3 volunteers to put in even more work than they already do to allow the objectors the power to control it. The objectors themselves don’t want to do the fucking work; they want other people to do the work while they make the rules. And that is some bull. shit.

[fremedon]

I was there, Gandalf, three thousand years ago, when the OTW was just getting off the ground. And everyone involved in starting the AO3 just assumed that fandom would take the code and build a bunch of single-fandom / character / pairing / trope archives with it. Because that’s what fandom was. That’s what it did with Astolat’s Automated Archive software, which ran the original Yuletide exchange and the big archives for Sentinel and Smallville and Due South and a couple dozen other fandoms at the time.

The AO3 was primarily meant to be a repository–one big backup for all of fandom. It was designed with the assumption that most of its content wouldn’t even be uploaded directly to AO3, but automatically cross-posted or imported from other, smaller archives. Or mailing lists. Or individual author websites. Or other parts of the fannish environment that just…don’t exist anymore.

When fandom olds say the AO3 changed fandom, we mean, it was fandom’s oxygenation crisis: such an immense, runaway success that it wiped out almost every vestige of the previous world. And enabled all sorts of new and more complex things to grow and evolve–but they’re growing in the archive’s world and breathing its air.

[white-throated-packrat]

Okay, I love the metaphor of AO3 as the Great Oxygen Catastrophe, because that’s hilarious and yet so apt. AO3 really did change the entire fandom ecosystem, and it’s hard to remember what we did in fandom before it.

Which is why Fanlore exists and you should read some entries when you have a moment.

[tigatog]

By The Numbers: The Lord of the Rings Films

[programmerhumour]

Easier than submitting a bug report

[sparrow-va]

[visualizing-archive-data]

In honor of the manga finishing, here’s something else that’s close to being finished! 

ID: Screenshot of a web app displaying the number of gendered pronouns per 1K words in the top 80 most popular fics in the Attack on Titan fandom on AO3, sorted by bookmark. The number ratio of male pronouns to female pronouns is overplotted on the right-hand axis using grey plus signs. There’s a sick new logo in the top-right corner, and noticeably way fewer female pronouns over all than male ones. Look at the scale on the right-hand axis and weep.

[visualizing-archive-data]

Levi “I used to do okay with the ladies” Ackerman apparently also does pretty well with the men.

Screenshot from a web app displaying my analysis of gendered pronoun uses in top AO3 fandworks of most popular Anime & Manga fandoms. This example is for all the verbs and direct objects associated with male or female pronouns in Shingeki no Kyojin/Attack on Titan’s most popular 80 fanfics, sorted by bookmark. Read more about how this plot was made here. (ignore the negative values on the x-axis, that was a cheap hack to make plots like these, I will fix it in the final version)

Oh, and I’ll end this with a personal observation: looking through this type of results from many top anime fandoms, one thing I consistently see is smile for women in the top 20 verbs. I don’t see this for men. Come on writers, women can do much more than smile and they do not have to smile all the time no matter what (American especially) society tells them. (rant over)

[d-naggeluide]

Fanfic Gender Studies - Behind the Scenes

A small example of how things work! Say I have this text:

ttext=(“He took the sword, but she laughed in his face. The dragon attacked her, and she defended the keep with fire, while he ate apples. He swung the lamp at his opponent, who jumped out of his way to shield himself.”)

I want to:

count the male and female pronouns

find out which verbs are associated with said pronouns

find out the direct objects or objects of prepositions associated with said verbs

It’s pretty easy to see what those are in the above example. How does the computer do?

doc=nlp(ttext) output=gendered_pronouns(doc)

He ['Masc'] took sword dobj sword she ['Fem'] laughed face op [face] she ['Fem'] defended keep dobj keep he ['Masc'] ate apples dobj apples He ['Masc'] swung lamp dobj lamp

We get 7 male pronouns, 3 female pronouns, the male actions take, eat, and swing, the female actions laugh and defend, and the objects they interact with (male: sword, apples, lamp female: face, keep).

I’ve got 211 million words to run this on now, from the most popular fics of the biggest anime and manga fandoms on AO3, with more on the way from randomly sampled fics for comparison!

[visualizing-archive-data]

The result of last night’s processing -- maybe I should start a KoFi for the increase in my electric bill 😅

Anyway, have a preview of a different type of WIP. Like in previous posts, this looks at the gendered pronoun usage (Engilsh: he/him/his/himself, she/her/hers/herself) in the top 80 publicly accessible fics by bookmark on AO3. (80 because I forgot the world doesn’t index on 0 ha ha anyway). Also like before, we can take an in-depth look at the verbs associated with he and she: does she go, and he stays? Alternatively, what does he do that she does not, and vice versa? Now using SpaCy’s dependency parse of morphology rules instead of bigrams.

Something else I look at this time are the people or objects she and he interact with. Does she slay the vampire, and he share the bed? Coming soon to Google Data Studio maybe, or Heroku if I get fed-up with limited choices.

P.S. to any writers out there, with or without popular fics: if there was a chance your work would show up in a study like this, would you prefer it to be anonymized? ie title, author, and URL removed from the dataset.

[visualizing-archive-data]

y’all are killing me, why are your favorite fics so long, Fairy Tail fandom?

[destinationtoast]

An Archive of Our Own, a project of the Organization for Transformative Works

AO3 has released a heckuva lot of data about the archive's works and tags, for use by anyone interested in the data!! 🎉🎉🎉

[destinationtoast]

It makes me SO HAPPY that this post got over 100 notes in a very short time... I love that so many people are excited about data, and I cannot wait to see what analyses come from this!

(Also plotting a few analyses of my own... :D :D :D )

[visualizing-archive-data]

Made a few improvements to my personal AO3 web app today, since my Marked for Later was getting too long. Now I’ve got everything in one filterable, sortable table and I can just click the link to take me straight to the fic! My favorite feature? Sort by ‘Version’ to see what’s updated while I’ve been not-reading these fics ;) I could be convinced to push this to production once I figure out how to handle passwords securely... reblog or comment if you’d like a tool like this!

Image ID: Web app displaying properties of fanfics found in my Marked for Later page. The usual categories -- title, author, fandom, rating, category, word count, etc -- are supplemented with other data available in the marked for later work blurb groups, such as version, date visited, etc. although some of these have been deleted from the table for clarity. Selecting a cell updates the features below the main table, printing out the title and summary and then a table displaying characters, relationships, and tags associated with that work.