Fandom Stats

News, info and bloggy things about fandomstats.org

[fandomstatsorg]

A small Christmas update:

After spending a few months in a “conserved” state due to hosting issues, Fandomstats.org is back up and running with full capabilities, graphs and all!

You now (again) can:

See statistics for an AO3 tag, displayed as graphs (breakdown of ratings, most popular categories, ships, warnings, other tags)

Get the same data as CSV or JSON

Use the API directly (and read some basic documentation)

Look up bookmarks of works by a specific author

AND (because I was already tinkering with the code and wanted to build something fun)

Get a random AO3 work from a tag, or any other list of works (including non-filterable searches). For example: a random work from the Yuletide 2021 collection

Merry Christmas or Happy Holidays to anyone who celebrates, and a nice time to everyone else :)

[fandomstatsorg]

hi there! does this return public works only?

In short: Yes, the API counts only public works. 

In not-so-short: The API gets its numbers from a tag’s filtering page (the one with the filter bar on the right). The backend (scraper) is accessing AO3 the same way a non-logged-in user would, and in that case, private works are neither listed, nor included in the values displayed on this page. 

I suppose this might skew the stats for fandoms with a lot of private works, so that’s something to keep in mind I guess. There is currently no way to have the fandomstats API work as a logged-in user.

[fandomstatsorg]

Hi! I was searching for API for AO3 and I found your amazing website! I feel stupid to ask this but how can I use the API endpoint? For example, I wanted to search for Harry Potter and I used this: fandomstats. org/ api /v1.0/stats?tag_id={Harry%20Potter}, but it ends up in error saying "This tag cannot be filtered on". Thanks for replying!!

Hi, I can see the problem! You need to enter the tag without the curly braces, e.g. just tag_id=Harry%20Potter - the {TAG} text is just meant to be a placeholder.

[fandomstatsorg]

PSA, the site is functional again (thanks @salsae for reporting - I got your ask but Tumblr refuses to let me answer). It was yet again a change of markup on AO3.

For anyone who might be using the API: the “warning” filter was renamed to “archive_warning” to match AO3′s markup.

[annathecrow-that-was]

I am not able to search the site fandomstatsorg, the following sentence appears: "This tag was not found on AO3".

Hi, thanks for letting me know - it’s probably a new wave of URL changes at AO3, so hopefuly it will be an easy fix. 

[fandomstatsorg]

Update: unfortunately, it appears that AO3 changed the HTML markup for the filter sidebars, which is where we’re getting our numbers from :( It’s fixable, but it will need a bit more work than I anticipated. 

Generally, this is a common problem when dealing with website scraping. For those who don’t know what “scraping” means: it’s a programmatic way of getting info from a web page by navigating through it’s contents to find specific elements which contain the data we need. It’s not that different from what a user is doing visually (and even more similar to how a screenreader user navigates): “in this div find this list, take every list item; the content is the number we’re looking for”

The catch is, the underlying structure of web pages changes. And unlike a human, the scraping code can’t easily adapt to the new structure (unless it’s written that way of course, but that’s a very different kettle of fish from our humble tools).

tl; dr Things changed, the API is borked, I’ll do my best to get it working soon.  

[fandomstatsorg]

Okay, fixed! Fandomstats.org works again.

[annathecrow-that-was]

I am not able to search the site fandomstatsorg, the following sentence appears: "This tag was not found on AO3".

Hi, thanks for letting me know - it’s probably a new wave of URL changes at AO3, so hopefuly it will be an easy fix. 

[fandomstatsorg]

Update: unfortunately, it appears that AO3 changed the HTML markup for the filter sidebars, which is where we’re getting our numbers from :( It’s fixable, but it will need a bit more work than I anticipated. 

Generally, this is a common problem when dealing with website scraping. For those who don’t know what “scraping” means: it’s a programmatic way of getting info from a web page by navigating through it’s contents to find specific elements which contain the data we need. It’s not that different from what a user is doing visually (and even more similar to how a screenreader user navigates): “in this div find this list, take every list item; the content is the number we’re looking for”

The catch is, the underlying structure of web pages changes. And unlike a human, the scraping code can’t easily adapt to the new structure (unless it’s written that way of course, but that’s a very different kettle of fish from our humble tools).

tl; dr Things changed, the API is borked, I’ll do my best to get it working soon.  

[fandomstatsorg]

Notice: It seems that the bookmark viewer is currently broken due to changes in AO3 URL structure. This  should be an easy fix, I’ll try to get to it as soon as I can. 

Thanks @rikkamaru​ and @brickgrass​ for reporting!

[fandomstatsorg]

Fixed! If it still doesn’t work for you, your browser might be caching the old version - hard reloading with Shift+F5 should sort that out.

[fandomstatsorg]

Notice: It seems that the bookmark viewer is currently broken due to changes in AO3 URL structure. This  should be an easy fix, I’ll try to get to it as soon as I can. 

Thanks @rikkamaru​ and @brickgrass​ for reporting!

[destinationtoast]

I feel stupid having to ask you this question but - I wanted to look at the stats for a small fandom I'm in using fandomstats, but no matter what I enter into the search bar I am told it's a non-canonical tag. Even when I copypaste the canon tag directly from Ao3! I thought maybe my fandom was too small, so I tried entering one of the examples of a canon tag listed on the fandomstats website (Teen Wolf (TV)) and was told it wasn't a canon tag either! What am I doing wrong? ;w;

It’s not just you – it looks like something’s broken!  :(  @fandomstatsorg @annathecrow

[fandomstatsorg]

Yep, it’s broken :/ From what I can tell right now, it seems AO3 now forces https, which breaks an important part of the API logic. I mean, forcing https is good! But, yeah. It broke things for us. 

Unfortunately this might take a while to fix - fetching the data through https is throwing errors and it looks like it might be a platform/library incompatibility issue. *yells at sky*

Thanks for letting us know @everythingbutthekitchensphinx​! Sorry about that. I can’t promise it will be fixed quickly, but I’ll see what I can do. 

[fandomstatsorg]

The bug is fixed now! Thanks for patience. :)

[destinationtoast]

I feel stupid having to ask you this question but - I wanted to look at the stats for a small fandom I'm in using fandomstats, but no matter what I enter into the search bar I am told it's a non-canonical tag. Even when I copypaste the canon tag directly from Ao3! I thought maybe my fandom was too small, so I tried entering one of the examples of a canon tag listed on the fandomstats website (Teen Wolf (TV)) and was told it wasn't a canon tag either! What am I doing wrong? ;w;

It’s not just you – it looks like something’s broken!  :(  @fandomstatsorg @annathecrow

[fandomstatsorg]

Yep, it’s broken :/ From what I can tell right now, it seems AO3 now forces https, which breaks an important part of the API logic. I mean, forcing https is good! But, yeah. It broke things for us. 

Unfortunately this might take a while to fix - fetching the data through https is throwing errors and it looks like it might be a platform/library incompatibility issue. *yells at sky*

Thanks for letting us know @everythingbutthekitchensphinx​! Sorry about that. I can’t promise it will be fixed quickly, but I’ll see what I can do. 

[annathecrow-that-was]

Does your fandomstats page have trouble with tags with quote marks? I tried to run a character who has a nickname and it said "This tag wasn't found on AO3."

You’re right, that doesn’t work - it’s trying to interpret the nickname as a quoted tag. :/

This might take a while to fix because it’s conflicting with the multiple tag logic so badly, but I’ll se what I can do. Thanks for pointing it out Anon!

(also, this is a reminder that Fandom Stats has its own tumblr @fandomstatsorg)

[fandomstatsorg]

This is now fixed! Multiple tags can be searched using commas.

[annathecrow-that-was]

Does your fandomstats page have trouble with tags with quote marks? I tried to run a character who has a nickname and it said "This tag wasn't found on AO3."

You’re right, that doesn’t work - it’s trying to interpret the nickname as a quoted tag. :/

This might take a while to fix because it’s conflicting with the multiple tag logic so badly, but I’ll se what I can do. Thanks for pointing it out Anon!

(also, this is a reminder that Fandom Stats has its own tumblr @fandomstatsorg)

[fandomstatsorg]

Notice: bookmark viewer in 'safe mode'

The bookmark viewer now returns a link to the result page instead of displaying them. This is a temporary fix while we deal with recent AO3 changes that broke the original code. The tool is still usable, if a little unwieldy, and we'll let you know when it's fully fixed.

Details under the cut!

If you tried to search sometime this week, you might have noticed that it returned a blank box instead of the results - this is because AO3 has apparently been hardening their security and the site now can't be displayed in an iframe. Now, better security is good! And there is a number of nasty things that can be done with iframes. But also, yeah, it broke our stuff. 

We have a way to fix it (and make it better, even), but it requires rewriting it from the ground up, which might take a while. Sorry about that.

[fandomstatsorg]

Update: multiple tag search

fandomstats.org can now make graphs for combinations of multiple tags - e.g. searching for al works that use specific two or more tags. This is done by searching with double quotes: for example, searching ‘“Harry Potter” Fluff’ will give you numbers for works using tags Harry Potter and Fluff at the same time.

Some other example searches, so you can glean the logic:

Star Trek - works tagged “Star Trek”

“Star Wars” Fluff Angst - works tagged “Star Wars”, “Fluff” and “Angst”

“Star Wars” Smut “Star Trek” - works tagged “Star Wars”, “Smut” and “Star Trek”

“Angst” “Fluff” - works tagged “Angst” and “Fluff”

[annathecrow-that-was]

Hi! Sorry I didn't feel comfortable emailing about this, just letting you know that on fandom stats you can't search '방탄소년단 | Bangtan Boys | BTS' or other tags using hangul (korean alphabet) - thanks!! this website is super cool and useful xxxxxx

Oh, damn, unicode strikes again. Sorry about that, I thought we’ve fixed it. I’ll look at it as soon as I can.

[annathecrow-that-was]

Searching for tags that use hangul and other non-ascii characters now works.

Sorry again - that was a big goof, considering the amount of K-Pop fandoms on AO3.

[annathecrow-that-was]

Fandom Stats

Fandom Stats is an ongoing project to create open-source tools for "fandom analysis" - data-driven exploration of behavior. You can find a tool visualising data about an AO3 tag, and more.

Damn, sharing a pet project is hard - on one hand you’re like “everyone needs to see this!!” and on the other you don’t want to sound like an advertisement. Oof.

ANYWAY! *makes a show hostess gesture* Voila! We made a thing! “we” being @destinationtoast, @esgibter, and me. Fandom Stats is a website that offers tools for fandom analysis - both for programmers and for non-techie fans. Basically, we’re trying to make “fandom analytics” more accessible.

In less coherrent words: We like graphs! So we made a thing that makes graphs for you!

How does it work: you go to fandomstats.org, and you put a canonical AO3 tag into the search field, and submit. (If you try searching for something that isn’t a canonical tag, it will be sad and tell you to look up the canonical version. We’re working on that. AO3 tag synonymisation is complicated.) And then you’ll get graphs about various things we can gather from AO3.

Let me demonstrate on my last fave, the remake movie Power Rangers. The tag is “Power Rangers (2017)”. I punched it in, and this is what I got:

Keep reading

[destinationtoast]

THANK YOU FOR POSTING THIS, CROW!!  I’ve been meaning to share but wanted to do a big writeup; this one is lovely.  :)  Also, thanks for making awesome graphs out of data!  (Crow did the data viz :D )

As one more demo, say I wanted to do a quick comparison of two popular AUs – coffee shops (Alternate Universe - Coffee Shops & Cafes) and Omegaverse (Alpha/Beta/Omega Dynamics).  I just entered both tags into the site (serially) and screencapped the results (and then I took an extra few seconds to paste the screencaps side by side into a Google Slides presentation, but that’s an unnecessary step unless you’re doing a demo like this one):

(Clearer/bigger versions of images here.)

Go test out your favorite AO3 tags!

[tayefeth]

Can you make graphs based on multiple criteria? For example, coffee shop AUs among works with more than 50 kudos?

[annathecrow-that-was]

Not at the moment, but it’s definitely on the todo list. Hypothetically, anything that gets the filter sidebar on AO3, we can get data for, which includes combined searches. I think the underlying API can already do all that, but the front-end isn’t ready for it.

[annathecrow-that-was]

Fandom Stats

Fandom Stats is an ongoing project to create open-source tools for "fandom analysis" - data-driven exploration of behavior. You can find a tool visualising data about an AO3 tag, and more.

Damn, sharing a pet project is hard - on one hand you’re like “everyone needs to see this!!” and on the other you don’t want to sound like an advertisement. Oof.

ANYWAY! *makes a show hostess gesture* Voila! We made a thing! “we” being @destinationtoast, @esgibter, and me. Fandom Stats is a website that offers tools for fandom analysis - both for programmers and for non-techie fans. Basically, we’re trying to make “fandom analytics” more accessible.

In less coherrent words: We like graphs! So we made a thing that makes graphs for you!

How does it work: you go to fandomstats.org, and you put a canonical AO3 tag into the search field, and submit. (If you try searching for something that isn’t a canonical tag, it will be sad and tell you to look up the canonical version. We’re working on that. AO3 tag synonymisation is complicated.) And then you’ll get graphs about various things we can gather from AO3.

Let me demonstrate on my last fave, the remake movie Power Rangers. The tag is “Power Rangers (2017)”. I punched it in, and this is what I got:

Keep reading

[destinationtoast]

THANK YOU FOR POSTING THIS, CROW!!  I’ve been meaning to share but wanted to do a big writeup; this one is lovely.  :)  Also, thanks for making awesome graphs out of data!  (Crow did the data viz :D )

As one more demo, say I wanted to do a quick comparison of two popular AUs – coffee shops (Alternate Universe - Coffee Shops & Cafes) and Omegaverse (Alpha/Beta/Omega Dynamics).  I just entered both tags into the site (serially) and screencapped the results (and then I took an extra few seconds to paste the screencaps side by side into a Google Slides presentation, but that’s an unnecessary step unless you’re doing a demo like this one):

(Clearer/bigger versions of images here.)

Go test out your favorite AO3 tags!