#computational linguistics

[official-linguistics-post]

A new look at our linguistic roots

Linguists and archaeologists have argued for decades about where, and when, the first Indo-European languages were spoken, and what kind of lives those first speakers led. A controversial new analytic technique offers a fresh answer.

An article from early 2024 on the use of computational phylogenetics for historical linguistics work on Proto-Indo-European. Read at Knowable Magazine (no paywall).

[uneasy-dreams]

studying computational linguistics since 2022 is a complete fever dream because my first welcome-event was all about how we do machine translation and search engines and then a few weeks/months in everyone was like "haha look at this cool new tool ChatGPT" and then suddenly after a year it became "okay we only talk about ChatGPT now" and then it became apparent that all available jobs for the next 5 years are just "can you program us a chatbot into our website it will solve all of our problems" (no and no) and now it seems to be just plain ethically wrong to ever do any actual work in the field because of the environmental impact and how generative ai actively kills critical thinking and creativity and honestly this is not what I signed up for and I deeply regret ever leaving humanities for this even if the job chances might technically be better

[mormorproposal]

TVD & FiF Comparison Using Computational Linguistics

When I opened my inbox back in December 2025 and saw the notification that @pasiphile had published the very first chapter of Fast In Fire, I - like so many others alongside me - was absolutely elated! After all, it had been 12 years since These Violent Delights, the first part of This Life Is A Trip (When You're Psycho In Love), had been published, dutifully offering a gateway into mormor for many, and stealing hearts right, left and center.

As a fan of pasi's writing and with my first semester at uni studying computational linguistics under my belt, the 12 year gap piqued my interest in particular. How would the writing differ? Would you be able to spot differences in regards to the average word or sentence length?

All that accumulated into me asking pasi for permission to analyze their texts, and then went to code. This post now aims to share/document the findings of this endeavor.

Disclaimer: The findings may be a tiny bit inaccurate - while I am absolutely having fun and enjoying this project immensly, I am still very new to this skill set,,

That being said, the findings are as follows:

TVD Word Count: ~168,318

*is slightly inaccurate due to a process of tokenization - which consists of splitting a text into its several parts (e.g. "Sebastian's boss doesn't smoke. -> "Sebastian", "'s", "boss", "does", "n't", "smoke") - beforehand -- I needed to work around having multiple tokens that made up one word (e.g. how "doesn't" is one word but is made up of "does" and "n't"), which I tried my best to account for but some tokens may still slip past

TVD Average Word Length: ~4.25 characters/word

FIF Word Count: ~365,069 words

FIF Average Word Length: ~4.39 characters/word

TVD Sentence Count: ~18,926 sentences

TVD Average Sentence Length: ~8.89 words/sentence

FIF Sentence Count: ~42,250.5 sentences *the .5 possibly stems from a forgotten quotation mark

FIF Average Sentence Length: ~8.64 words/sentence

-> This indicates a slight change in sentence length, with TVD having slightly longer sentences on average than FIF

The fics TVD (11,268 tokens) & FiF (19,031 tokens) have this many tokens in common: 7,493

The fics have an overlap of 24.73%

The Type Token Ratio for TVD (0.05) and for FIF (0.04)

*The Type Token Ratio (TTR) refers to how versatile a text is in its choice of words. It compares the complete Tokens (as already mentioned above) of a text with the overall Types in a text, which basically mean the occurrence of one unique Token [e.g. "Sebastian's boss doesn't like Sebastian's cigarettes" -> the types "Sebastian" and "'s" occur 2x, every other type exactly 1x; the overall sentence has 9 Tokens but 7 Types]. The closer the TTR is to 0, the more variety a text has to offer.

-> The TTR for both TVD and FiF are very close to 0, which is an indicator for their varied choice of words. However, FiF has a minisculy larger TTR, but this can be chalked up to its length being so much bigger than TVD. I'd say TVD and Fif are overall pretty varied with ca. equal levels of different choice of words.

Most frequently used Tokens:

TVD: TOKENS - FREQ | FIF: TOKENS - FREQ

1 you - 9,953 | the - 14,439

2 the - 6,651 | you - 11,791

3 and - 5,205 | to - 8,857

4 he - 4,401 | a - 7,885

5 to - 4,051 | and - 7,564

.. … …

96 how - 268 | head - 661

97 hands - 267 | some - 647

98 way - 264 | looked - 641

99 right - 263 | into - 641

100 want - 263 | Moriarty - 639

-> Note how the ranking of "you" and "the" switches places from TVD to FiF,,, that was so funny to me.

Most frequently used Bigrams:

*A Bigram means a Token Grouping of 2 that occur right next to each other in a text. "Sebastian killed him", for example, would consist out of the Bigrams "Sebastian killed" and "killed him".

TVD: BIGRAMS - FREQ | FIF: BIGRAMS - FREQ

. “ - 3,475 | . “ - 8,179

. ” - 2,220 | . ” - 6,621

You 're - 1,818 | ” “ - 6,178

? ” - 1,480 | ? ” - 4,087

” “ - 1,399 | , ” - 2,490

.. … …

96 again , - 142 | Sherlock said - 294

97 for a - 140 | he 'd - 293

98 there 's - 139 | you 've - 291

99 one of - 139 | ” he - 291

100 back . - 139 | “ Yeah - 291

-> Particularly interesting to me is the frequency of ” “, which marks how often a closing quotation mark was followed by an opening one. Or: how often was a quote directly followed with another quote without any dialoge markers?

Most frequently used pronouns:

TVD: PRONOUN - FREQ | FIF: PRONOUN - FREQ

you - 9,953 | you - 11,791

he - 4,401 | it - 6,257

your - 3,182 | I - 5,898

it - 2,823 | he - 5,263

his - 2,425 | she - 4,533

-> Note how the existance of white hat chapters is clearly noticable in the pronoun frequency - especially with "she" making it into the Top 5 :')

Honorable Mentions:

In the overall Token frequency, in TVD "Jim" came in 26th place (Freq.: 916), which just happens to be lower than the frequency of "Sherlock" in FiF - with 2,087 mentions, it took 25th place… I know Jim's fuming over that somewhere lmao

"men", "sex" & "bedroom" all share one place with 55 mentions, and "fucking", "gun" & "home" all share one place with 54 mentions - if that isn't mormor in a nutshell, I don't know what else is haha

When I grouped the words into word classes, "Jim" took the first place of the TVD nouns with a frequency of 916 occurrences; a mention of "Moriarty", however, occurs 237 times and takes 7th place.

The first 3 most frequent nouns in FiF were Sherlock (Freq.: 2,084), Mary (Freq.: 1,146) and John (Freq.: 1,014).

Moran is mentioned 722 times (-> he's in 5th place) and Moriarty takes the 9th place with being mentioned 632 times.

[lendubsofficial]

finally started uploading an ARG project i’ve been slowly building for a while and i’m honestly really excited about it

it follows a university archivist who discovers a hidden office and an old external hard drive labeled PROJECT_Y, then decides to document the contents before telling anyone else it exist

the story unfolds through:

screen recordings

transcribed files

archive restorations

community posts

and audience participation puzzles!

it’s very slow burn and heavily inspired by old internet mysteries, archival research, obsolete software, and weird academic documents

also i got way too invested in designing fake research papers and archive metadata for this thing

first log is up now :)

[huiopl064]

wait what is your beef with computational linguistics? I'm super curious

i don't have beef with it, in fact I kinda like it; iirc it was the post when everythings a hammer all you have is nails or sumtn and i wanted to write some random thing as like "this is my hammer" and computational linguistics was the first thing that came to my mind?

also it kinda sounds funny like yeagh im computing language

also i think there was an xkcd about hating on computational linguistics? something like "everyone's hating on easy things, but how about hating on computational linguistics, ooh, my field is so ill defined I can have two contradictory statements and still be accepted" or something like that

[ahollowedoutdoll]

ive been having a bit of a crisis lately about the direction I want to go in my education. I’m an undergraduate pursuing two degrees, BA in linguistics and a BS in math. I have loved linguistics for a long time and I have a penchant for solving problems with numbers patterns and abstract reasoning. I’m considering, because I want to do graduate work and go into research, pursuing a master’s in computational linguistics.

this would, honestly, be perfect. a combination of my love for both of these fields and an opportunity to work with computers. all of these are awesome. I did NACLO (north american computational linguistics open competition) in highschool. not only that, but apparently it’s an incredibly high-demand degree, in a growing field, with lots and lots of opportunities! sounds perfect, right?? uhh well

the only problem is that CL in this day and age is kind of just AI. I think. I want to be proven wrong and find some way to be able to apply my interests and skills in a way that doesn’t benefit from and contribute to what I see as a growing environmental and ethical issue. and I have a more nuanced opinion on AI than just “AI bad,” but I know I would be struggling to resolve the cognitive dissonance I’d face helping prop up billionaire’s pet misinformation machines. not to mention i just think a lot of people pursuing AI might be kind of insufferable to me.

CL is an interesting subfield to me and I probably would have pursued it in a heartbeat a decade ago. do I need to accept that this is what it is? could I forge my own path somehow?

I really don’t know, lol

[stellaisnotamermaid]

esl fic writers? 👉👈

I'm doing a term project on native language prediction! I plan for the data to be fanfiction written in English by ESL authors.

For this, I need two things:

author; author's native language

Any help would be greatly appreciated! :)

[mostlysignssomeportents]

Podcasting "How To Think About Scraping"

On September 27, I'll be at Chevalier's Books in Los Angeles with Brian Merchant for a joint launch for my new book The Internet Con and his new book, Blood in the Machine. On October 2, I'll be in Boise to host an event with VE Schwab.

This week on my podcast, I read my recent Medium column, "How To Think About Scraping: In privacy and labor fights, copyright is a clumsy tool at best," which proposes ways to retain the benefits of scraping without the privacy and labor harms that sometimes accompany it:

https://doctorow.medium.com/how-to-think-about-scraping-2db6f69a7e3d?sk=4a1d687171de1a3f3751433bffbb5a96

What are those benefits from scraping? Well, take computational linguistics, a relatively new discipline that is producing the first accounts of how informal language works. Historically, linguists overstudied written language (because it was easy to analyze) and underanalyzed speech (because you had to record speakers and then get grad students to transcribe their dialog).

The thing is, very few of us produce formal, written work, whereas we all engage in casual dialog. But then the internet came along, and for the first time, we had a species of mass-scale, informal dialog that also written, and which was born in machine-readable form.

This ushered in a new era in linguistic study, one that is enthusiastically analyzing and codifying the rules of informal speech, the spread of vernacular, and the regional, racial and class markers of different kinds of speech:

https://memex.craphound.com/2019/07/24/because-internet-the-new-linguistics-of-informal-english/

The people whose speech is scraped and analyzed this way are often unreachable (anonymous or pseudonymous) or impractical to reach (because there's millions of them). The linguists who study this speech will go through institutional review board approvals to make sure that as they produce aggregate accounts of speech, they don't compromise the privacy or integrity of their subjects.

Computational linguistics is an unalloyed good, and while the speakers whose words are scraped to produce the raw material that these scholars study, they probably wouldn't object, either.

But what about entities that explicitly object to being scraped? Sometimes, it's good to scrape them, too.

Since 1996, the Internet Archive has scraped every website it could find, storing snapshots of every page it found in a giant, searchable database called the Wayback Machine. Many of us have used the Wayback Machine to retrieve some long-deleted text, sound, image or video from the internet's memory hole.

For the most part, the Internet Archive limits its scraping to websites that permit it. The robots exclusion protocol (AKA robots.txt) makes it easy for webmasters to tell different kinds of crawlers whether or not they are welcome. If your site has a robots.txt file that tells the Archive's crawler to buzz off, it'll go elsewhere.

Mostly.

Since 2017, the Archive has started ignoring robots.txt files for news services; whether or not the news site wants to be crawled, the Archive crawls it and makes copies of the different versions of the articles the site publishes. That's because news sites – even the so-called "paper of record" – have a nasty habit of making sweeping edits to published material without noting it.

I'm not talking about fixing a typo or a formatting error: I'm talking about making a massive change to a piece, one that completely reverses its meaning, and pretending that it was that way all along:

https://medium.com/@brokenravioli/proof-that-the-new-york-times-isn-t-feeling-the-bern-c74e1109cdf6

This happens all the time, with major news sites from all around the world:

http://newsdiffs.org/examples/

By scraping these sites and retaining the different versions of their article, the Archive both detects and prevents journalistic malpractice. This is canonical fair use, the kind of copying that almost always involves overriding the objections of the site's proprietor. Not all adversarial scraping is good, but this sure is.

There's an argument that scraping the news-sites without permission might piss them off, but it doesn't bring them any real harm. But even when scraping harms the scrapee, it is sometimes legitimate – and necessary.

Austrian technologist Mario Zechner used the API from country's super-concentrated grocery giants to prove that they were colluding to rig prices. By assembling a longitudinal data-set, Zechner exposed the raft of dirty tricks the grocers used to rip off the people of Austria.

From shrinkflation to deceptive price-cycling that disguised price hikes as discounts:

https://mastodon.gamedev.place/@badlogic/111071627182734180

Zechner feared publishing his results at first. The companies whose thefts he'd discovered have enormous power and whole kennelsful of vicious attack-lawyers they can sic on him. But he eventually got the Austrian competition bureaucracy interested in his work, and they published a report that validated his claims and praised his work:

https://mastodon.gamedev.place/@badlogic/111071673594791946

Emboldened, Zechner open-sourced his monitoring tool, and attracted developers from other countries. Soon, they were documenting ripoffs in Germany and Slovenia, too:

https://mastodon.gamedev.place/@badlogic/111071485142332765

Zechner's on a roll, but the grocery cartel could shut him down with a keystroke, simply by blocking his API access. If they do, Zechner could switch to scraping their sites – but only if he can be protected from legal liability for nonconsensually scraping commercially sensitive data in a way that undermines the profits of a powerful corporation.

Zechner's work comes at a crucial time, as grocers around the world turn the screws on both their suppliers and their customers, disguising their greedflation as inflation. In Canada, the grocery cartel – led by the guillotine-friendly hereditary grocery monopolilst Galen Weston – pulled the most Les Mis-ass caper imaginable when they illegally conspired to rig the price of bread:

https://en.wikipedia.org/wiki/Bread_price-fixing_in_Canada

We should scrape all of these looting bastards, even though it will harm their economic interests. We should scrape them because it will harm their economic interests. Scrape 'em and scrape 'em and scrape 'em.

Now, it's one thing to scrape text for scholarly purposes, or for journalistic accountability, or to uncover criminal corporate conspiracies. But what about scraping to train a Large Language Model?

Yes, there are socially beneficial – even vital – uses for LLMs.

Take HRDAG's work on truth and reconciliation in Colombia. The Human Rights Data Analysis Group is a tiny nonprofit that makes an outsized contribution to human rights, by using statistical methods to reveal the full scope of the human rights crimes that take place in the shadows, from East Timor to Serbia, South Africa to the USA:

https://hrdag.org/

HRDAG's latest project is its most ambitious yet. Working with partner org Dejusticia, they've just released the largest data-set in human rights history:

https://hrdag.org/jep-cev-colombia/

What's in that dataset? It's a merger and analysis of more than 100 databases of killings, child soldier recruitments and other crimes during the Colombian civil war. Using a LLM, HRDAG was able to produce an analysis of each killing in each database, estimating the probability that it appeared in more than one database, and the probability that it was carried out by a right-wing militia, by government forces, or by FARC guerrillas.

This work forms the core of ongoing Colombian Truth and Reconciliation proceedings, and has been instrumental in demonstrating that the majority of war crimes were carried out by right-wing militias who operated with the direction and knowledge of the richest, most powerful people in the country. It also showed that the majority of child soldier recruitment was carried out by these CIA-backed, US-funded militias.

This is important work, and it was carried out at a scale and with a precision that would have been impossible without an LLM. As with all of HRDAG's work, this report and the subsequent testimony draw on cutting-edge statistical techniques and skilled science communication to bring technical rigor to some of the most important justice questions in our world.

LLMs need large bodies of text to train them – text that, inevitably, is scraped. Scraping to produce LLMs isn't intrinsically harmful, and neither are LLMs. Admittedly, nonprofits using LLMs to build war crimes databases do not justify even 0.0001% of the valuations that AI hypesters ascribe to the field, but that's their problem.

Scraping is good, sometimes – even when it's done against the wishes of the scraped, even when it harms their interests, and even when it's used to train an LLM.

But.

Scraping to violate peoples' privacy is very bad. Take Clearview AI, the grifty, sleazy facial recognition company that scraped billions of photos in order to train a system that they sell to cops, corporations and authoritarian governments:

https://pluralistic.net/2023/09/20/steal-your-face/#hoan-ton-that

Likewise: scraping to alienate creative workers' labor is very bad. Creators' bosses are ferociously committed to firing us all and replacing us with "generative AI." Like all self-declared "job creators," they constantly fantasize about destroying all of our jobs. Like all capitalists, they hate capitalism, and dream of earning rents from owning things, not from doing things.

The work these AI tools sucks, but that doesn't mean our bosses won't try to fire us and replace us with them. After all, prompting an LLM may produce bad screenplays, but at least the LLM doesn't give you lip when you order to it give you "ET, but the hero is a dog, and there's a love story in the second act and a big shootout in the climax." Studio execs already talk to screenwriters like they're LLMs.

That's true of art directors, newspaper owners, and all the other job-destroyers who can't believe that creative workers want to have a say in the work they do – and worse, get paid for it.

So how do we resolve these conundra? After all, the people who scrape in disgusting, depraved ways insist that we have to take the good with the bad. If you want accountability for newspaper sites, you have to tolerate facial recognition, too.

When critics of these companies repeat these claims, they are doing the companies' work for them. It's not true. There's no reason we couldn't permit scraping for one purpose and ban it for another.

The problem comes when you try to use copyright to manage this nuance. Copyright is a terrible tool for sorting out these uses; the limitations and exceptions to copyright (like fair use) are broad and varied, but so "fact intensive" that it's nearly impossible to say whether a use is or isn't fair before you've gone to court to defend it.

But copyright has become the de facto regulatory default for the internet. When I found someone impersonating me on a dating site and luring people out to dates, the site advised me to make a copyright claim over the profile photo – that was their only tool for dealing with this potentially dangerous behavior.

The reasons that copyright has become our default tool for solving every internet problem are complex and historically contingent, but one important point here is that copyright is alienable, which means you can bargain it away. For that reason, corporations love copyright, because it means that they can force people who have less power than the company to sign away their copyrights.

This is how we got to a place where, after 40 years of expanding copyright (scope, duration, penalties), we have an entertainment sector that's larger and more profitable than ever, even as creative workers' share of the revenues their copyrights generate has fallen, both proportionally and in real terms.

As Rebecca Giblin and I write in our book Chokepoint Capitalism, in a market with five giant publishers, four studios, three labels, two app platforms and one ebook/audiobook company, giving creative workers more copyright is like giving your bullied kid extra lunch money. The more money you give that kid, the more money the bullies will take:

https://chokepointcapitalism.com/

Many creative workers are suing the AI companies for copyright infringement for scraping their data and using it to train a model. If those cases go to trial, it's likely the creators will lose. The questions of whether making temporary copies or subjecting them to mathematical analysis infringe copyright are well-settled:

https://www.eff.org/deeplinks/2023/04/ai-art-generators-and-online-image-market

I'm pretty sure that the lawyers who organized these cases know this, and they're betting that the AI companies did so much sleazy shit while scraping that they'll settle rather than go to court and have it all come out. Which is fine – I relish the thought of hundreds of millions in investor capital being transferred from these giant AI companies to creative workers. But it doesn't actually solve the problem.

Because if we do end up changing copyright law – or the daily practice of the copyright sector – to create exclusive rights over scraping and training, it's not going to get creators paid. If we give individual creators new rights to bargain with, we're just giving them new rights to bargain away. That's already happening: voice actors who record for video games are now required to start their sessions by stating that they assign the rights to use their voice to train a deepfake model:

https://www.vice.com/en/article/5d37za/voice-actors-sign-away-rights-to-artificial-intelligence

But that doesn't mean we have to let the hyperconcentrated entertainment sector alienate creative workers from their labor. As the WGA has shown us, creative workers aren't just LLCs with MFAs, bargaining business-to-business with corporations – they're workers:

https://pluralistic.net/2023/08/20/everything-made-by-an-ai-is-in-the-public-domain/

Workers get a better deal with labor law, not copyright law. Copyright law can augment certain labor disputes, but just as often, it benefits corporations, not workers:

https://locusmag.com/2019/05/cory-doctorow-steering-with-the-windshield-wipers/

Likewise, the problem with Clearview AI isn't that it infringes on photographers' copyrights. If I took a thousand pictures of you and sold them to Clearview AI to train its model, no copyright infringement would take place – and you'd still be screwed. Clearview has a privacy problem, not a copyright problem.

Giving us pseudocopyrights over our faces won't stop Clearview and its competitors from destroying our lives. Creating and enforcing a federal privacy law with a private right action will. It will put Clearview and all of its competitors out of business, instantly and forever:

https://www.eff.org/deeplinks/2019/01/you-should-have-right-sue-companies-violate-your-privacy

AI companies say, "You can't use copyright to fix the problems with AI without creating a lot of collateral damage." They're right. But what they fail to mention is, "You can use labor law to ban certain uses of AI without creating that collateral damage."

Facial recognition companies say, "You can't use copyright to ban scraping without creating a lot of collateral damage." They're right too – but what they don't say is, "On the other hand, a privacy law would put us out of business and leave all the good scraping intact."

Taking entertainment companies and AI vendors and facial recognition creeps at their word is helping them. It's letting them divide and conquer people who value the beneficial elements and those who can't tolerate the harms. We can have the benefits without the harms. We just have to stop thinking about labor and privacy issues as individual matters and treat them as the collective endeavors they really are:

https://pluralistic.net/2023/02/26/united-we-stand/

Here's a link to the podcast:

https://craphound.com/news/2023/09/24/how-to-think-about-scraping/

And here's a direct link to the MP3 (hosting courtesy of the Internet Archive; they'll host your stuff for free, forever):

https://archive.org/download/Cory_Doctorow_Podcast_450/Cory_Doctorow_Podcast_450_-_How_To_Think_About_Scraping.mp3

And here's the RSS feed for my podcast:

http://feeds.feedburner.com/doctorow_podcast

If you'd like an essay-formatted version of this post to read or share, here's a link to it on pluralistic.net, my surveillance-free, ad-free, tracker-free blog:

https://pluralistic.net/2023/09/25/deep-scrape/#steering-with-the-windshield-wipers

Image: syvwlch (modified) https://commons.wikimedia.org/wiki/File:Print_Scraper_(5856642549).jpg

CC BY-SA 2.0 https://creativecommons.org/licenses/by/2.0/deed.en