A paper from the Workshop on the Use of Computational Methods in the Study of Endangered Languages (ComputEL-3) about making better tech tools for linguistic annotation, featuring many ideas that I wish I’d had in grad school!
Abstract:
To reduce the annotation burden placed on linguistic fieldworkers, freeing up time for deeper linguistic analysis and descriptive work, the language documentation community has been working with machine learning researchers to investigate what role technology can play, with promising early results. This paper describes a number of potential follow-up technical projects that we believe would be worthwhile and straightforward to do. We provide examples of the annotation tasks for computer scientists; descriptions of the technological challenges involved and the estimated level of complexity; and pointers to relevant literature. We hope providing a clear overview of what the needs are and what annotation challenges exist will help facilitate the dialogue and collaboration between computer scientists and fieldwork linguists.
Lingthusiasm Episode 24: Making books and tools speak Chatino - Interview with Hilaria Cruz
As English speakers, we take for granted that we have lots of resources available in our language, from children’s books to dictionaries to automated tools like Siri and Google Translate. But for the majority of the world’s languages, this is not the case.
In this episode, your host Gretchen McCulloch interviews Dr Hilaria Cruz, a linguist and native speaker of Chatino, an Indigenous language of Mexico which is spoken by over 40,000 people. Hilaria combines her work as an Assistant Professor of linguistics at the University of Louisville with creating resources for her fellow speakers of Chatino, everything from paperback or cloth children’s books to high-tech speech recognition tools which will make it easier to create more resources like this in the future. And she’s also making these resources available for other underrepresented languages!
Click here for a link to this episode in your podcast player of choice or read the transcript here
Announcements:
There were two big announcements at the top of the episode:
The first is that we have a date for our liveshow in Melbourne! We will be at the State Library of Victoria on Friday the 16th of November. Tickets on sale soon through our EventBrite.
We are also thrilled to announce we’ll be doing a liveshow in Sydney! We’ll be at GiantDwarf on Monday the 12th of November. Tickets available through their website.
We also have new merch!
Thanks to Lucy Maddox for bringing Space Babies to life! Check out the art in this post. A portion of the proceeds from the Space Baby merch will be donated to the Resource Network for Linguistic Diversity.
We also have new scarf colours, and t-shirts that say “I want to be the English schwa. It's never stressed.” Check out our Merch page for more details.
This month’s bonus episode was about hyperforeignisms! We take an international tour through how our minds deal with the interesting edge cases of words that are kinda-English and kinda-other-languages. Support the show on Patreon to get access to this and all 19 bonus episodes.
Here are the links mentioned in this episode:
Chatino language (Wikipedia)
Lengua Chatino resources website (mostly in Spanish)
A video story told aloud in Chatino by Hilaria Cruz
Hilaria Cruz’s page at the University of Kentucky
Hilaria Cruz’s website
Joel Sherzer
Tony Woodbury
Hilaria’s PhD thesis (Linguistic poetic and rhetoric of Eastern Chatino of San Juan Quiahije)
Automatic Speech Recognition (Wikipedia)
Alexis Michaud
Oliver Adams
Tlingit, Ojibwe, Hupa languages (Wikipedia)
Here’s a photo of the children’s books that Hilaria Cruz and her students made! Books 1-6 (from left) are in Chatino. The rightmost book is in Hupa and the second from right book is in Ojibwe. All eight books are available for purchase on Amazon. (More about the book creation process.)
From the description on the ASREL Retreat website (Automatic Speech Recognition for Endangered Languages):
This retreat will foster a dialogue between computer scientists working on Automatic Speech Recognition (ASR) specifically neural networks, native speakers of endangered languages, and linguists doing research on endangered languages to address the issue of the “bottleneck” of language transcription and discuss the use of technology in the transcription of language data.
Tools and technologies to automate and expedite the transcription and translation of oral texts from endangered languages are urgently needed. Most researchers working with endangered languages process their materials manually. Some researchers estimate that it takes roughly from 1 to 50 hours to prepare one hour of spoken text manually.
ASR technologies can significantly reduce the workload of transcribing large collections of speech recordings in these lesser-studied languages. Automating the process will enable the transcriber to become more of an editor, accelerating the overall transcription process. Implementation of ASR technologies could free up time for linguists, language activists, and speakers to create materials for teaching and learning the language, rather than spending countless hours transcribing.
You can listen to this episode via Lingthusiasm.com, Soundcloud, RSS, Apple Podcasts/iTunes, Spotify, YouTube, or wherever you get your podcasts. You can also download an mp3 via the Soundcloud page for offline listening.
To receive an email whenever a new episode drops, sign up for the Lingthusiasm mailing list.
You can help keep Lingthusiasm ad-free, get access to bonus content, and more perks by supporting us on Patreon.
Lingthusiasm is on Twitter, Instagram, Facebook, and Tumblr. Email us at contact [at] lingthusiasm [dot] com
Gretchen is on Twitter as @GretchenAMcC and blogs at All Things Linguistic.
Lauren is on Twitter as @superlinguo and blogs at Superlinguo.
Lingthusiasm is created by Gretchen McCulloch and Lauren Gawne. Our senior producer is Claire Gawne, our editorial producers are Emily Gref and A.E. Prévost, our production assistants are Celine Yoon & Fabianne Anderberg, and our music is ‘Ancient City’ by The Triangles.
This episode of Lingthusiasm is made available under a Creative Commons Attribution Non-Commercial Share Alike license (CC 4.0 BY-NC-SA).
MEGAN: Hi, welcome to the Vocal Fries Podcast, the podcast about linguistic discrimination.
CARRIE: I'm Carrie Gillon.
MEGAN: And I'm Megan Figueroa. And today we are going to be talking about artificial intelligence generally and more specifically automatic speech recognition. We have a guest here with us, because Carrie and I do not know anything about - well, ok, I shouldn't speak about Carrie’s ignorance on the topic - but I don't know
CARRIE: You were correct: I know nothing.
MEGAN: Ok. We are joined by Dr. Rachael Tatman. She is a data preparation analyst at Kaggle, which is, according to its own Twitter account, the world's largest community of data scientists. Rachael has a PhD in linguistics from the University of Washington, where she specialized in computational sociolinguistics. Her dissertation, among other very cool things, showed the ways in which automatic speech recognition falls short when dealing with sociolinguistic variation, like dialects. Welcome Rachael.
RACHAEL: Hi! Thanks for having me.
CARRIE: Hi!
MEGAN: I'm very excited to have you. I feel like, with automatic speech recognition - I don't know if other people feel this way - but I was in the camp where I didn't realize that I should care about what's happening, with how automatic speech recognition is being made or to listen to voices. I didn't know that I had to care, and now I care. Hopefully we’ll show listeners why we should care.
RACHAEL: Yeah! I can share one of my stories about automatic speech recognition. One thing that's really difficult is children's voices, because obviously children are a different size, and they have a lot of acoustic qualities that are different. But also children have a lot of individual variation. If you spend a lot of time with kids, what's a “bink bink”? Is it a blankie? Is it a bottle? I'm Dyslexic, and when I was in grade school, they tried to use automatic speech recognition to like help me type faster, so I could complete assignments and turn them in. And not fail third grade. Yeah: it did not work well. I remember very distinctly that I tried to say “the walls were dark and clammy”. We were doing a creative writing exercise, and it was transcribed as “the wells we're gathered and planning”. Which is kinda close acoustically, but also there's some probably poor language modeling behind that, where they thought that that was a more likely sentence than the one that I'd started with.
CARRIE: Wow.
MEGAN: Lets define automatic speech recognition for the listeners, and for myself. What is automatic speech recognition?
RACHAEL: It's the computational task of taking in an acoustic signal of some kind and rendering it as speech. When I say an acoustic signal, I mean specifically a speech acoustic signal, because also people work with whale song and bird song and stuff. It gets used a lot in especially mobile devices. If you know Google Now or Cortana - I don't know how many people actually use Cortana - or a Bixby, which is Samsung's virtual assistant, or Siri, which is probably the most well-known one, they all rely on automatic speech recognition to sort of understand what you're saying and reply to your tasks. It gets used a lot in virtual assistants, which is Echo or Google Play, or Apple's launching one soon, as well. I don't know how much you guys keep up with tech news, but these are little devices that sit in your home, and you can be like, “hey, Siri”. I guess, I don't know what the Apple one’s gonna be called. Or, “okay, A L E X A”. I don't want to say it, because I don’t want to turn on everybody’s Alexa.
CARRIE: Oh no! I think you just did!
ALEXA: Hmm. I'm not sure what you meant by that question.
RACHAEL: Go back to sleep Alexa. It's everywhere, is the point. People are incorporating into new technologies. They're getting really excited about it. People are talking about incorporating it into testing for schools, for standardized testing. People are talking about incorporating it into medical diagnostic tests. Things like - what's that a semantic one, where you have to name a bunch of things that are similar, before you move on?
CARRIE: I don't know.
RACHAEL: It gets used for diagnosing a lot of things, like schizophrenia and Alzheimer's and specific learning disorders. Semantic coherence test maybe?
CARRIE: Yeah.
RACHAEL: Anyway, people have been working on using speech recognition for that, so incorporating it into this. People are using it for language assessment, for immigration and visas, a lot of very high stakes places.
MEGAN: That's very high stakes! That's very important.
RACHAEL: Probably my favorite thing to be upset about in this realm is people incorporating NLP, which is natural language processing, which is more as text, and also automatic speech recognition, in these algorithms that you put information into and it tells you whether or not you should hire the person.
CARRIE: UGH. Oh my god.
RACHAEL: So very, very high stakes applications. You may not always realize that your voice or your or your language is being used in this way.
MEGAN: You can't see my face, but I'm horrified right now. Okay. It's very important. There's a lot of practical applications that automatic speech recognition is being used for. In all of these realms, there's possibility of discrimination.
RACHAEL: Yeah. As far as I know, no one who has looked at an automatic speech recognition system or a text-based system, specifically looking at performance across different demographic groups on a certain task, has ever found that “nope there's no difference, it doesn't matter, the system is able to deal super well with people of all different backgrounds”. Looking specifically at speech, I've done a number of studies, and by a number I mean two, and I'm working off and on a third, because I am also working full-time - this isn't part of my job. I'm not speaking on behalf of my company or employer. If you're gonna yell at anybody, yell at me personally. This is my private, individual thing. What I found is that there are really, really strong dialectal differences - so differences between people who have different regional origins. Which dialects get recognized more or less accurately seems to be - I'm having a hard time picking it apart, but I think it also is a function of social class. It's fairly difficult to find speech samples that are labeled for the person's dialect and also their social class, and good sociolinguistics sampling methods. It's really hard to find large annotated speech databases that you can do this analysis with, but I found really strong dialectal differences in accuracy, with general American, or mainstream American English, or mainstream US English, or standardized American English - there's a lot of different terms for this “fancy” talk - having the lowest error rate. I found that Caucasian speakers have the lowest error rate. Looking at Caucasian speakers, African American speakers, speakers of mixed race, and the study where I had race information - I only had one Native American speaker, so I had to exclude them, because one data point is not a line. So that's worrying.
MEGAN: Right. What does it mean to have an error? What is the practical result of an error in speech recognition?
RACHAEL: There are three types of errors. One is where a word is substituted, so you say “walls” and it hears “wells” and transcribes that. Another one is deletion, where you say something like “I did not kill that man” and “I did kill that man” is transcribed. I should say people are still using hand stenographers for court cases, as far as I know. I don't think anyone in the legal system is using ASR, but yikes.
CARRIE: Better not.
RACHAEL: There’s also insertion, when you think that you heard a word and it wasn't actually there. A lot of times words that’re inserted are function words like “the” and “of”, things like that.
MEGAN: So deletion, insertion, and hearing it wrong. Doing another word.
RACHAEL: Yeah those are the only three transformations you can do, yes.
MEGAN: Okay.
RACHAEL: Word error rate is just, for all the words, how many of them did you get wrong in one of these ways. Just on a frustration level, if you're using speech recognition as a day-to-day user, and it doesn't work real great, that's annoying. I'm sure if you guys ever use speech recognition, like on your phones, or I have a Google home, and I'll use it for a timer a lot. It's actually gotten better - it used to be really bad at hearing the word “stop” like “stop the timer”. I think that might be because of the [ɑ] [ɔ] merger that some people have. That's my pet theory. But it's gotten a lot better at understanding “stop”. I would have to say “stop” five times while I'm standing at the kitchen with cheese smeared on my arms up to my elbows or whatever.
CARRIE: That's really strange because there isn't a different “stop”. I have the [ɑ] [ɔ] merger, so I can't make the other word, but it doesn't exist anyway.
RACHAEL: Yeah, it may be that the acoustic model is more - so speech recognition, I'm gonna say this generally - because people are futzing around with it a lot and I'm messing it up -generally has two modules. One is the acoustic model, which is “what waveforms map to what sounds” and the other is the language model, which is “what words are more likely”. When you when you put those together, out comes the other end through some fancy math the most likely, for some given set of input parameters, the most likely transcription, ideally. And my guess is that if you're not specifically modeling the fact that some people have two vowels and some people have one vowel in that space, you may be less able to recognize those sounds generally, because you think that there's just a lot of variation there. Especially since there's also the Northern city shift that's muddling that whole area as well. Sorry, should I assume a lot of phonetic backgrounds on the part of your speakers?
CARRIE: Our listeners? Yeah, I was just gonna say: maybe we should describe what the Northern vowel shift is.
RACHAEL: There are a number of vowel shifts in the United States, and if you think of individual vowels as being little swarms of bees that are clustered around flowers, sometimes the swarms of bees move on or the flower moves and the swarm follows after it, and different places have movement in different directions. I don't know, is that a good analogy? I'm using my hands a lot. I know you guys can't see it. Is that clear?
CARRIE: I understand what you're saying but I'm not sure. Good question.
MEGAN: I don’t know. I like the analogy. I feel like that's good.
RACHAEL: I would look up vowel change shifts, if I was listening to this. I’d just google them, and you'll see some nice pictures and arrows. You’ll be like “oh!”
CARRIE: Yeah. We’ll add something to the Tumblr to explain a little bit about vowel shifts, and also the merger we were talking about, because I can't replicate it. I can't do that open o [ɔ].
MEGAN: I can’t either. I don’t have it.
RACHAEL: “cot” [k ɔ t] as in “I caught the ball” and then “caught” [kɑt] - nope, I have it backwards again.
CARRIE: Yep. We haven't asked you yet, but what is computational sociolinguistics?
RACHAEL: I don't think I made up the term, but I'm probably one of the first people to call myself that. Dong Nguyen - she's currently at the Alan Turing Institute - has a fabulous dissertation that has a really nice review chapter that talks about the history of this emerging field. It is approaching sociolinguistic questions using computational methods, and it's also informing computational linguistics and natural image processing and automatic speech recognition with sociolinguistic knowledge. Working on dialect adaptation, I think would fall within that - that's when you take an automatic speech recognition system that works on one dialect and try to make it work good for other dialects as well. I've done some work on modeling variation in textual features by social groups. I've looked at political affiliation and punctuation and capitalization in tweets, and there's pretty robust differences at least in the US between oppositional political identities. I'm trying to think of other people's work, so it's not just: here's a bunch of stuff that I've done!
MEGAN: Basically, everyone's trying to model everything.
RACHAEL: Basically. Or should be, hopefully. I think, historically, there hasn't been a lot of - I think sociolinguists are much better about knowing what's going on in computational linguistics then computational linguists are at knowing about what's going on in sociolinguistics. I'm coming from sociolinguistics and coming to computational linguistics. I'm trying to have a big bag of Labov papers and toss them to people, be like “here you go! Here you go!”
MEGAN: Yes and Labov is a very famous sociolinguist.
RACHAEL: He is, yes. I would call him the founder of variationist sociolinguistics - which is not the only school, but it is the school that I work in mainly.
CARRIE: Yeah, I think that's - well that's the most famous one as far as I know.
MEGAN: Yeah. I didn't know there were other ones. Of course there is.
RACHAEL: Yeah, I'm trying to think of names. Mostly I'll come across it I'll be like “oh”. I guess discourse analysis is a type of sociolinguistics.
MEGAN: Oh, okay.
CARRIE: Yes.
RACHAEL: But different bent.
CARRIE: How is automatic speech recognition trained to understand humans? I think you've already started to answer this, but maybe you can answer it'll be even more, if there is more to say.
RACHAEL: Yeah. I mentioned there are two components: there's the acoustic model and then there the language model. Usually the language model is actually trained on texts. You take a very, very, very large corpus. I think right now - I don't know about the standard, but what I think most people would like to use would be the Google trillion word corpus, which is from scraped web text, or people use the Wall Street Journal corpus, which is several hundred million words long. You know the probability of a certain set of words occurring in a certain order, so it's the poor man's way of getting syntax. I'll tell you about how it's traditionally done. People are replacing both the pronunciation dictionary and the acoustic model, which sometimes includes the pronunciation dictionary with big neural nets. We can talk about that in a little bit, but traditionally the pronunciation dictionary was made by hand. The Carnegie Mellon the pronunciation dictionary, or CMU pronunciation dictionary, is probably the best-known one for American English. People transcribe words, and if there's one that you need that's not transcribed, you add it.
MEGAN: And what’s a pronunciation dictionary?
RACHAEL: It is a list of words and then how they're pronounced. The phones, so “cat” would be [k] [æ] [t] - those three sounds in order. Then the acoustic model takes the waveform and tells you the probability of each of those sounds. So it's like “well I'm pretty sure it's [æ], but I guess it could also be [ɑ]”, through a process of transformations. People recently have been taking a speech corpus - usually one that's labeled, so you know what words are spoken - and then using all of that data and shoving it into a neural net, which is a type of machine learning algorithm - it's a family of machine learning algorithms. People use different types and flavors, and they have different structures. What neural nets are really, really good at is finding patterns in the data, and recognizing those same patterns later, without you having to tell them to do it. They learn it themselves, from just the way that the information is organized. They've been really, really good and useful in image processing, in particular, being able to look at a photo and be like “here is an apple”, “here is an orange” and “I have circled them helpfully for you”. They're really good at that. But as it turns out there is more structure in language than there is in other types of data.
CARRIE: Shocking. [sarcasm]
RACHAEL: It is to some people. I've had a lot of frustrating conversations where people were like “but it works really good on images!” I'm like “yes, but language is different”. If it weren't, we wouldn't need linguistics. People wouldn't need to study language their entire lives, if it was just like images but in sound, basically. Which I think is probably not news to any listeners of this podcast, but definitely it is news to some people. Neural nets are really good at seeing things that they've seen before, or identifying the types of things they've seen before, and if they see new things, they're not so good at it. I think that's really where a lot of the trouble with dialect comes in, because sociolinguistic variation is very systematic between dialect regions. One person can have multiple dialects as well. I don't want to make it sound like you sort people into their dialects and then apply the correct model and then boom everything's correct all the time. Because people have tried that and it works better than not doing anything, but it's still not - I don’t know. There's a lot of work to do, and I don't want to make it sound like speech research engineers are just fluffing around and not knowing about language, because they do. But it's difficult, and it hasn't, I think, been a major focus for a lot of people recently, and I'm hoping that it will become more of a research focus.
MEGAN: You said something in one of your interviews that I wanted to read here that I liked. You say that “generally the people who are doing the training aren't the people whose voices are in the dataset. You'll take a dataset that's out there that has a lot of different people's voices, and it will work well for a large variety of people. I think the people who don't have sociolinguistic knowledge haven't thought about the demographic of people speaking would have an effect. I don't think it's maliciousness. I just think it wasn't considered.”
RACHAEL: Yeah.
MEGAN: I think “it was a considered” part - it's how I felt actually. I obviously very much care that people aren't discriminated against in every aspect of life. But I just didn't think about speech recognition.
RACHAEL: Yeah. I think we have this idea that like “oh a computer’s doing it, so it's not gonna be biased”.
MEGAN: You’re right.
RACHAEL: That’s nice to believe that you have the ethical computer from Star Trek, but bias is built into all machine learning models. It's one of the things you study in a machine learning class. You talk about bias and variance, and it's there in the model, and it's there in the data. Pretending that it can go away if you just keep adding more data is a little bit of a problem for the people who are actually using the system, and it doesn't work as well for them as it should, maybe.
CARRIE: It's also very naïve.
MEGAN: Yeah. Humans are the ones that are doing it, right. We’re behind the machines. Of course there's biases. I was thinking, I've said I've never thought about this before, but I don't use Siri, because Siri does not understand me very well at all. I've given up.
SIRI: I miss you Megan.
MEGAN: I didn't take the next step. I didn't take the next step, and think “oh why is this the case that she's not understanding me very well”.
RACHAEL: Yeah.
CARRIE: She understands me pretty well. I have a pretty standard North American accent.
RACHAEL: A little bit of the Canadian shift.
CARRIE: I do, but it's not enough to trick SIRI, apparently. My accent has shifted somewhat since living in the States for over nine years. I knew that speech recognition did have a problem with at least some dialects, because there's a fairly famous skit from Burnistoun, the Scottish sketch comedy show, where he's just saying “eleven”, and it's one of the words where in a Scottish accent “eleven” is pretty close, so the speech recognition should have been able to pick it up. Most of the sketches is them speaking in a Scottish dialect that I think many Americans would not understand actually.
IAIN CONNELL: You ever tried voice recognition technology?
ROBERT FLORENCE: No.
IAIN CONNELL: They don't do Scottish accents.
ROBERT FLORENCE: Eleven.
ELEVATOR: Could you please repeat that.
ROBERT FLORENCE: Eleven.
IAIN CONNELL: Eleven.
ROBERT FLORENCE: Eleven. Eleven.
IAIN CONNELL: Eleven.
ELEVATOR: Could you please repeat that.
IAIN CONNELL: Eleven. If you don't understand the lingo, away back home your own country. [If you don't underston the lingo, away back hame yer ain country.]
ROBERT FLORENCE: Oohh, is the talk now is it? “Away back home your own country?” [Oh, s'tha talk nae is it? "Away back tae yer ain country"?]
IAIN CONNELL: Oh, don't start Mr Bleeding Heart – how can you be racist to a lift? [how can ye be racist tae a lift?]
ELEVATOR: Please speak slowly and clearly.
CARRIE: Anyway, it's a really funny sketch, if you haven't seen it. I will post it, because I think it's funny.
MEGAN: I don't know what it is about me. I don't know if vocal fry would affect it at all. I'm also kind of mumbly. I try not to be mumbly on the podcast obviously, but in my normal everyday life, I am a mumbler, so that might be it. I expect Siri to understand my mumbles, but she don't, so I gave up.
RACHAEL: But see, that's part of the problem, because - I don't know for sure, but I would be beyond shocked if - because I know that for sure, Google has the ability to - it retains the speech samples that you send them, and I'm sure that they fold them back into their training data, so if you're not using it, because it doesn't understand you, it's pretty much never gonna understand you, is the unfortunate thing. I think that's really part of the reason that there's - I think - pretty strong class effects. This is this is me having a science hunch that I haven't really banged out yet in some experimental work. I think that people who have a higher socioeconomic status and particularly professional class, mobile - not rural the other one.
CARRIE: Urban.
RACHAEL: Urban! Yeah, thank you. Especially professional, mobile, urban people have - I'm almost positive - higher cognition rates, correct word rates.
MEGAN: You mentioned something about how the language model was taking in things like The Wall Street Journal. Wouldn't that affect it too? That's not your acoustic signal, but it's the way you speak? I don't know.
RACHAEL: Yeah. No that's fair. “‘Fiduciary’ seems to be a fairly common word that humans use all the time, so I’m gonna look for that one.”
CARRIE: I would be very surprised if class didn't play a role. It always does. In everything that we talk about, there's something about class going on too. But we don't think about it as much in North America as we should.
MEGAN: We really don't. Especially since it's wrapped in with race and ethnicity so much. I act like I know anything beyond the States. It's just very American.
RACHAEL: I think it's very much the top-level thing that people think about with language variation in the UK, for sure.
MEGAN: Ah, okay.
CARRIE: Yeah. Absolutely.
MEGAN: Interesting.
RACHAEL: There's RP, and then those weird regional dialects that we don't like. As a person not from the UK, that's the judgments that I've gotten from consuming popular media.
CARRIE: It used to be worse. Because the BBC used to only have received pronunciation with their reporters, but now you'll hear regional varieties. Still the most prestigious versions of those varieties, but at least you'll hear Irish dialects now. Things are slightly better.
MEGAN: You'd hope so. ASR is trained to understand humans, so you're feeding in them these datasets, and I didn't know this but I guess, like you said, if I talk to Siri, I'm also feeding into a dataset.
RACHAEL: Yeah. That seems very likely to me. Again, I don't know for sure, and this may be something that's Googlable, you could find using a search engine, and it may be something that you could not find using a search engine. The other thing about neural nets is because they're good at seeing things they seen before, they get really good if you have a lot of data, a lot of data. I have not yet seen the company that would ignore free data that people were giving to it to improve model performance.
MEGAN: Do you have examples of automatic speech recognition failing to understand people that we can give the listeners, so they can see the problem?
RACHAEL: I can give you one from my life, which continues to drive me nuts. I'm from the South and I have a general American professional voice that I use, but especially if I'm relaxing with friends or with my family, I definitely sound more Southern. One of the things that happens in the South and also in African American English is nasal place assimilation. If you have a nasal after a stop, which are sounds like [k] [t] [p] [g] [d] [b], you will change the nasal, [m], [n], or [ŋ], to whatever the thing in front of it was. I would say “beanbag” as “beambag”, especially in an informal setting. Or a “handbag” is “hambag”. Put your things in your handbag. I think it's a fairly common thing. Google used to always, always, always search for “beambag” when I wanted to know about “beanbags”, because I was doing research to get - I currently have one, I just turned to look at it - a really good beanbag chair. They’re very comfy! I like them. It kept telling me about “beambags”, which are not a thing! It just drove me up the wall, because lots of people do this thing. This is a normal speech process.
CARRIE: Yeah. Very common.
MEGAN: Also, a “hambag”, a bag of hams and that might be something people have.
RACHAEL: I guess a Smithfield ham does come in a bag. It comes like a little canvas bag.
MEGAN: I guess that's where it's trying to get you. But that's not what you’re [meaning]. That’s funny. Okay, how do we solve this problem? What should we be thinking about when we develop automatic speech recognition databases and such? Who should be involved?
RACHAEL: Sociolinguists. Definitely hire sociolinguists. That's my general go-to drum. It's a hard problem. I don't want to pretend that a sociolinguist looks at it and they're like “ah! Fix this parameter!” and then suddenly it works great for everyone. Because the fact of minority languages or language varieties, in particular, is that they’re minority because fewer people use them. If you are trying to optimize performance and accuracy for the model as a whole, and you raise it for the people who are from minority groups - whatever those may be - if you are using the one model, that will lower it for your majority language speakers. Just adding more data isn't necessarily going to be the fix. People have been have been working on this for a long time, and it's a very hard problem, and I have nothing but respect for everyone who's working on this. There's a couple of approaches that people are doing. One is to train multiple models on different stable language varieties. In the US I might train one on West Coast generally, and as far as that is a single language variety, I’d probably train one on the Northeast, one on the northern cities, so Chicago, Michigan sort of area - Chicago's in Illinois - Illinois, Michigan sort of area. One on the South. One also for the mid-Atlantic region. And then select one of those models, based on whichever would most accurately represent the person who's speaking. That's one approach. Another approach is to take the model and then change it for every single person's voice. That will capture dialectal variation, but it will also capture individual variation. The reason that your phone doesn't do that automatically is because it is very computationally intensive. These models are very big. They have a lot of information in them. They have a lot of parameters, and to change those, it takes a lot of raw processing power. That's not really feasible to do for individual people, as it stands. I don’t know, maybe in five years it will be completely feasible. We’ll all have GPUs falling out of our pockets everywhere we go. I don't know. That's another approach that some people have taken. I don't know, maybe with some fancy new ensembling - which is where you multiple different types of models and stick them together like - what are those, K’nex? - and they build a pipeline, and then you shove the data all the way through the pipeline, and all the different models that are connected together. Those have been getting really good results lately, so maybe some sort of clever ensembling, where you do something like demographic recognition, and then something like shifting your language model a little bit. I don't know. I don't know. I don't know what people are gonna come up with.
MEGAN: This is the future. This is the future that millennials want or something. I don’t know. This is the future liberals want. If this is the future, I'm thinking about the fact that in 30 years we're gonna be a majority-minority country. We're on our way to this becoming a bigger and bigger problem.
RACHAEL: Yes. Definitely.
MEGAN: The fact that Siri or Alexa - sorry - has trouble understanding people that aren't in this white -
RACHAEL: Super-privileged, small group?
MEGAN: Yeah, right. There's a gender bias too, right? It's males that are understood.
CARRIE: And we're the majority.
RACHAEL: I just want to quickly intercede here - I did some in earlier work finds that it was more accurate - specifically YouTube's automatic captions were more accurate for men than women, but I think, because I couldn't replicate that result, the problem there was actually signal-to-noise ratio. Women tend to be a little bit quieter, because we're a little bit smaller. If you are speaking at the same effort-level in the same environment, there's just gonna be a little bit more noise in the signal for women, because we're not quite as loud. I don't know that clutter signal processing can fix that. I'm gonna keep working on this, and who I might find out that actually there are you know really strong differences, it maybe it can't deal with things that women do more. I was gonna say “vocal fry”, but I've seen no evidence that women fry more than men, which I'm sure you talked about. At length.
CARRIE: Right. That was our first episode. Everybody does it. Leave us alone!
MEGAN: Leave it alone. Get the fuck off my vocal fry! What I'm hearing is this is something that we should all very much care about, because, like Carrie said, everyone else is the majority. If it's best trained on white men that are in higher socio-economic classes, that's not the majority. It sounds like we need to have people in the room, because, like you said, you don't think it was considered when they were making these datasets. We need people in the room that are like “wait, I come from this community where that's not how we talk, this is not gonna work for me or us”.
RACHAEL: Yeah, definitely.
MEGAN: I definitely want to plug a representation too. We need more people in the room.
RACHAEL: Definitely. I've been talking about English, because that's what I know about, and specifically American English. I don't want to get into British dialectology, cuz that's crazy, crazy complex. But this is also a problem in other languages. Arabic dialects are incredibly different from each other.
CARRIE: Right.
MEGAN: Now I'm thinking about people that are bilingual.
RACHAEL: Or bidialectal.
MEGAN: Or bidialectal, for sure. That's gonna be something else that we would want automatic speech recognition to recognize.
RACHAEL: Yeah. Absolutely. I can give people something that you can do right now - is that Mozilla, which is the company that owned the Firefox - continues to own, I think, the Firefox web browser - is currently crowdsourcing a database of voices, and voice samples. You can head over to that website, for which there is a link that I for sure can't find. I think it's called the Mozilla Common Voice Project, but don't quote me on that unless it's right.
MEGAN: We'll put it somewhere.
RACHAEL: Mozilla is doing a collection of voices of people, and they're specifically trying to get people from different demographic backgrounds, for specifically this problem, for knowing demographic information about someone, for having speech samples for the. They're also having people manually check the recording, so if this is something that's interesting, and you want to listen to a lot of voices, I'd recommend heading over there and checking it out.
MEGAN: Ah, so they are crowdsourcing automatic speech recognition. That's a good idea. That’s a tough - how you get the most variation in the people that reply.
RACHAEL: One thing that I found in my own work, and other computational linguists have found as well, is that we know a lot about variation in speech, but a lot of the same variation also exists in text. A lot of the text that you produce in your day-to-day life, especially if it's anywhere online, is getting fed into a lot of natural language processing tools. There are also problems with those. Things like identifying what language someone is using is not as good.
CARRIE: Yeah, I notice that on Twitter a lot. It wants to translate from French all the time.
MEGAN: Yeah.
RACHAEL: Twitter's language ID is a hot mess. A hot mess.
CARRIE: And it's never French. It's never French. In fact, sometimes it's English. I'm like “what is going on?”
MEGAN: I've had Estonian. Translate from Estonian.
RACHAEL: Yeah. Estonian tends to show up a lot. I'm trying to think of - I have started doing some very lackadaisical data collection. I think it seems to work on a character level, so it tends to be fairly good at languages that have a unique character set. It tends to be very good at Thai, but related Germanic languages - pfft - it does not. That's Bing. That's on Microsoft. They're the back end there, so I 100% blame them. Maybe, if they hadn't gutted their research teams, they would be able to do this better.
CARRIE: Hint hint.
MEGAN: That is something that we can do immediately. Do you have something really poignant you want to say about why this is all important? What's the takeaway message? Because we've been talking us this whole time about why it's important, but what do you think is the takeaway?
RACHAEL: It's important to hear people's voices. Both literally and metaphorically.
CARRIE: There we go. There's the money shot.
MEGAN: That’s the money shot. Money, money, money. See that's what we wanted!
CARRIE: Yes. It's important to hear people's voices. I think that's a good place to end.
MEGAN: Yeah, cuz that was it. Unless you have anything else, Rachael?
RACHAEL: Hmm. No, I don't think so. I use my hands a lot, so hopefully a lot of the things that I was saying with my hands I was also saying with my voice.
MEGAN: Yeah, I realized that at our first episode, I was using my hands, and now my hands don't even move. It comes with some experience - of my four episodes that I have done, five episodes.
CARRIE: Five! Five episodes. This is our sixth.
RACHAEL: Ooh! Lucky number 6!
CARRIE: Thank you so much, Rachael, for talking with us today.
RACHAEL: You’re welcome!
CARRIE: That was awesome. I learned a lot.
MEGAN: I know, I learned so much. I was so ignorant on this subject. So thank you. Hopefully this will be of interest to people that have no idea, but also to our listeners that really like speech recognition stuff. I know that I know that they're there. This is very exciting. Alright, cool. I guess we want to leave everyone with one message, which is: don't be a fucking asshole.
CARRIE: Don't be an asshole. Bye!
CARRIE: The Vocal Fries Podcast is produced by Chris Ayers for Halftone Audio. Theme music by Nick Granum. You can find us on Tumblr, Twitter, Facebook and Instagram @vocalfriespod. You can email us at [email protected].
How well do Google and Microsoft and recognize speech across dialect, gender and race?
How well do Google and Microsoft and recognize speech across dialect, gender and race?
If you’ve been following my blog for a while, you may remember that last year I found that YouTube’s automatic captions didn’t work as well for some dialects, or for women. The effects I found were pretty robust, but I wanted to replicate them for a couple of reasons:
I only looked at one system, YouTube’s automatic captions, and even that was over a period of several years instead of at just one…
Transcript Lingthusiasm Episode 24: Making books and tools speak Chatino - Interview with Hilaria Cruz
This is a transcript for Lingthusiasm Episode 24: Making books and tools speak Chatino - Interview with Hilaria Cruz. It’s been lightly edited for readability. Listen to the episode here or wherever you get your podcasts. Links to studies mentioned and further reading can be found on the Episode 24 show notes page.
[Music]
Lauren: Hi Lingthusiasts, Lauren here. Before we get to Gretchen's great interview with Hilaria Cruz today, I have two exciting pieces of news to share with you. The first is that we have a date for our Melbourne live show. We'll be at the State Library of Victoria on Friday the 16th of November. Also, very excited to share with you that we are doing a live show in Sydney as well. We’ll be at GiantDwarf on Monday the 12th of November. For more details and links to tickets, go lingthusiasm.com/show. Our patrons will get a couple of free tickets. We're looking forward to meeting them and all of you as well. We're also super excited to be able to share with you some new Lingthusiasm merchandise that we've been working on, which was another Patreon goal of ours. We are very excited to bring you the space babies and space pigeon from Episode 1 of the show in full and glorious animated colour on a range of merchandise, available through our site. You can see the images, find out more about the illustrations, and our wonderful illustrator, Lucy Maddox, by visiting lingthusiasm.com/merch. And now, over to Gretchen.
[Music]
Gretchen: Welcome to Lingthusiasm, a podcast that's enthusiastic about linguistics. I'm Gretchen McCulloch, and I'm here with Dr. Hilaria Cruz, who is a Neukom Fellow at Dartmouth College and just starting as an assistant professor in linguistics at the University of Louisville, and is a native speaker of Chatino who works with Chatino as well. Welcome, Hilaria.
Hilaria: Well, thank you. Hello, everyone!
Gretchen: Thank you so much for being here!
Hilaria: You are welcome.
Gretchen: I'm here because you invited me down for a workshop at Dartmouth, and so I'm going to talk about that as well. But first, let's start with: How did you get into linguistics?
Hilaria: As a native speaker of Chatino, I grew up in a community where we all spoke Chatino, and then it came time for us to go to school, and then my father says, “Well, I would like you to get an education.” So my father then says, “We're going to go to this other town named Juquila so you guys can go to school.” We came to Juquila and, at a time in the 1970s, the Mexican government wanted indigenous children to study, so they developed these, like, boarding schools – well, it was like a boarding house where indigenous children that came from the outskirts of the Spanish-speaking towns had room and board while they went to public school. So my family came to this, what is called “the houses” there, and I was sent to elementary school not knowing a word of Spanish. It was complete immersion.
Gretchen: Wow.
Hilaria: At the time, there was just one school in that little town, just one elementary school for – I would say, I'm just guessing, 5,000 people. There were many children. There were some children that went to school in the morning. There were some children that were going to school in the evening. Since I did not know that much Spanish, my father took me there and introduced me to this class. The teacher was nice, and then I – just as a warm-up, he let me go there for a few mornings. I would just go, just for a few hours.
Gretchen: How old were you?
Hilaria: I think that I was about seven.
Gretchen: Mm-hmm.
Hilaria: I would just hang out for a few hours, and I would just take off and go back to the boarding house, where my parents were also staying with us. And then the teacher says, “Oh, this is fine. But I think that you are ready to begin your regular classes now. So you are going to come to school from 2:00 to 6:00.” Then I began to get really sad, because I did not want to go to school because I used to get bored, just to sit down there and just not understand what the teacher says. Then I began to go to these evening classes, and I was not happy. So then I decided that I want to go back to the morning class, because it was the same teacher teaching first grade in the morning and then in the evening. I will go back, and he will welcome me. “Aha, yes, come in.” I will go for two, three hours in the morning, as much as I wanted. Then, I will go back again, back home, and, to me – that was the happy medium for me. At some point, then, he stopped me, and he says, “No, no, no, no. You cannot set up your own time. You must come back here, to school.” So I –
Gretchen: You go to school, you play the rules.
Hilaria: Yes. To me school was just horrible. But I guess I persisted, and I got really bored, and I guess I passed, and then I – when we got to sixth grade there was – I guess in that school it was only a middle school, but, actually, my family and I were not happy in that town because that was the first place where I encountered racism against Native people. Because in my community, I was just a member of society, right? But when we got to school, kids began to pinch me, and they will call me “india” and things like that. So I will come back to my father and say, “Why is it that these kids are saying this to me? Why is it that they are pinching me and pounding me?” Because I just did not understand.
Gretchen: Yeah.
Hilaria: Then my father would say, “But you know, every time they tell you that, just be proud of yourself.” But how can a kid to be proud of – how can you be proud if somebody is stopping you, right? That was my experience in that town. It was like a frontier town. There was a lot of racism towards the Chatino people, who live in the outskirts of that town. So then I told my father and my siblings too, “You know what? We're not happy in this town.” Then he told us, “Well, I understand that you're not happy. Let's go to the city.” We went to the city. And there was a more cosmopolitan – we lived in a small area of the city where there were a lot of migrants from indigenous communities, so it was better. I continued my education. My father and I talked, and he encouraged me to continue college because he told me that in college, it'll be a lot of fun. That in college, I will be able to talk to other people, and meet a lot of people, so I was excited about going to college. I continued my education because I wanted to meet interesting people in college. That was the whole goal.
Gretchen: It’s a good goal. I like that goal.
Hilaria: I wanted to have interesting conversations, meet interesting people in college.
Gretchen: Yeah. That's great. I like that.
Hilaria: I think that my father was really smart for doing that.
Gretchen: He knew you very well.
Hilaria: Yeah, I think so. So my goal was to get to college, and have wonderful conversations, and meet interesting people.
Gretchen: Mm-hmm.
Hilaria: I continued going to college. Then, in 1991, I came to the United States. I began to hear conversations about linguists working with Native American languages, reviving these moribund languages, and then I began to think, “You know what? Maybe linguists will be able to help me create an alphabet for the Chatino language.” Because I was very curious about how to represent the Chatino languages, but the only thing that I was familiar with was the Spanish alphabet.
Gretchen: Right.
Hilaria: But since these languages come from such different linguistic families, Spanish does not have all of the symbols to be able to represent a tonal language, let's say like Chatino. We would try to write it down, but when it came time to read it, we could not read it.
Gretchen: It’s kind of unsatisfying.
Hilaria: So there was something missing there. I began to think, “You know what? This sounds very interesting. I think that linguists could help me maybe find a way to write the Chatino language.” I began to write to different linguists. I would write them letters and say, “Yes. Could you please help me develop an alphabet for my language?”
Gretchen: And this is 1991, so you're writing letters.
Hilaria: Ah, well it was –
Gretchen: Or emails maybe?
Hilaria: – letter. It was letter. I was writing emails around 2000, or something like that. It wasn’t in 1991. So I began to write these letters in 2000. My sister, Emiliana, also was on the same path. It was interesting because my sister Emiliana – I would talk about all these things, and I said that I was the first one, but, quietly, she had the same idea. She was more proactive. Well, we were both working on our own ends.
Gretchen: Oh, interesting.
Hilaria: Yeah. So Emiliana was in Oaxaca City then. She had a little coffee shop down there. And there walks in this American guy, whose name is Joel Sherzer. The professor Joel Sherzer, he used to teach at the University of Texas in the anthropology department. Joel Sherzer is a wonderful, very friendly guy. Joel Sherzer began to strike up a conversation with Emiliana, and then Joel asked Emiliana, “Tell me about you. What are you interested in?”
So then Emiliana says, “Well, you know what? I would love to be able to study my language.” And Joel says, “Well, that sounds very interesting. Tell me more about it because we at the University of Texas are very interested in working with native speakers of Mexico. Actually, we're creating a program. Why don't you come and visit us –
Gretchen: Oh my god.
Hilaria: – at the University of Texas?” So Emiliana went to Texas. She joined the anthropology department at the University of Texas. Emiliana began her program at the University of Texas, and we were just all very excited because then we met Anthony Woodbury, who was very interested in working with us with Chatino. And then Emiliana says, “Well, you know, in our studies of Chatino we need linguists. I think that you should join the linguistics department.”
Gretchen: So she recruited you to do the 'stics part?
Hilaria: Yeah! So then I say, “Sure! Yeah, I would love to do that.”
Gretchen: Okay. Is she your older sister?
Hilaria: She’s younger.
Gretchen: Oh, wow!
Hilaria: Well, she always tells me what to do. So that is how I joined the linguistics department. I was doing fieldwork with them. I was not a linguistics student or anything like that. I was just like – I accompanied them because I was just so excited they were studying Chatino, and this is something that I always wanted to do. So I began to do fieldwork. I pay my own way, and I just wait over there.
Gretchen: Oh my god. So you were like the consultant? They were asking you questions about Chatino?
Hilaria: No, no, no, no.
Gretchen: You were just doing it with them for fun?
Hilaria: I was just doing it for fun. No, but they also did – and this was in the summer of 2003 – they did fieldwork. I mean, Emiliana was in school. I was not. I was just like a labourer, someone who was so excited about this, you know? Because this was always what I wanted to do, right? I was just so excited about it. So Emiliana told me, “Hey, we're going to go down there, and we're going to do fieldwork.” And I said, “I’ll come.” I pay my own way. I went there.
Since Emiliana had placed this idea of me that I needed to study linguistics, then I asked Tony, “Hey, do you think that I could join the Linguistics Department?” And then he says to me, “Well, you're going to have to apply, but if you're ready to work hard, we might accept you.”
Gretchen: Did you speak English at this point?
Hilaria: Yes, I did.
Gretchen: Oh, okay.
Hilaria: So that's how I began to study linguistics.
Gretchen: Oh, that's cool. So then you became a grad student at University of Texas.
Hilaria: This is how I began a graduate [degree] in linguistics at the University of Texas in Austin.
Gretchen: Oh, cool. That's really neat. And then you wrote a dissertation about Chatino and learned a lot of stuff, including how to write it?
Hilaria: Yeah. So one of the things that I wanted to do was to describe the poetics of Chatino where, at the time, I would call it poetics. One of the things that I grew up with, and what Joel Sherzer called verbal art, is what he calls it.
Gretchen: Speech...
Hilaria: He wrote a book on speech play and verbal art. This is the title of a book that Sherzer wrote, but basically he used to call it verbal art.
Gretchen: Verbal art. Oh, yeah.
Hilaria: So what he meant by verbal art is just to take into account the different types of speech styles that exist in one community. And one of the things that I saw in growing up in San Juan Quiahije is that there are so many different types of discourse. We have ceremonial discourse. We have political discourse. We have dialogues, you know, exchanges. I wanted to record some of those discourses because some of those – so what gets transmitted in many of those discourses is the need to preserve tradition. For example, there's always a pair of lines that the orator says. This is our tradition. This is what the elders left for us since the foundation of the community, since the foundation of the mountains, and to leave this tradition will be seen as bad. So as a Chatino speaker, every time I hear these ceremonial speeches, they resonate with me a lot. So I wanted to record us. The first assignment that I had in the first moment when I was in graduate school was I proposed to record political speech. I went back to my community. I recorded political speech, and the change in the authority. I did my master's thesis on that. And then for my dissertation, I did an ethnography of speech. I described the different patterns and structures –
Gretchen: Oh, like all the different genres?
Hilaria: The different genres. And it was describing the ecosystem of the different styles –
Gretchen: Oh, that’s interesting!
Hilaria: – of speech in the community. And I worked with very gifted and talented speakers. This is something that I really wanted to do, and it was a lot of work, but it was, I think, very important work. So I have the basis now to be able to continue that kind of work for other people to do the same.
Gretchen: Yeah. So we can we can link to your dissertation. But that's also how you got into, “Oh my gosh, it's really hard to work with audio data.”
Hilaria: That is right! Because it was hard for two reasons to be able to transcribe speech. I was, of course, a native speaker of the language so I knew what they were saying, but the problem was, when it was time to commit this language onto paper, that I was just a beginning writer. I mean, we were just in stages of developing the alphabet for the language, and then also learning linguistics. And then Anthony would worry. He is very meticulous at what he does, so he will say, “Well, what is the alphabet that you want to use?” There were like two or three choices of alphabet, so if you're going to choose one, you're going to have to be consistent. I was just beginning.
Gretchen: That is so hard for a beginner too.
Hilaria: It was just – it was tough. But another thing that I noticed was that it was just very time-consuming to be able to be transcribing these texts. This is something that I began to realise when I began to transcribe this text. In my dissertation, I offer transcriptions of five to six ceremonial texts. All of these are semi-long texts.
Gretchen: You mean long speeches?
Hilaria: Yeah, these are long speeches, and different genres.
Gretchen: Yeah. Because I know when we're doing – for the podcast, we make transcriptions for the podcast. We put the audio onto YouTube, actually, and we use YouTube's automatic speech recognition to create the first draft of the transcript. And then we have a person who goes in and corrects it because there’re all these corrections you need to make. For one thing, YouTube never recognises the name of the podcast, Lingthusiasm, because it's not a real word.
Hilaria: Yeah.
Gretchen: And so it gives us these crazy things about, like, “link Suzy azzam.” Like, who is Suzy? Why is she here? But we're lucky because we have automatic –
Hilaria: You’re lucky!
Gretchen: – transcripts.
Hilaria: At least! At least you have – this is news to me. This is the first time I heard the process by which you do transcription.
Gretchen: But it still takes hours, and we're still paying a human to do hours of detailed work making the transcripts, even though we cut out half of it by having an automatic thing create the first draft that that person can then fix.
Hilaria: That is so interesting. I wish I had a tool like that for Chatino, you know? At least something that could help me – just to give me a little help so I won't get carpal tunnel.
Gretchen: My gosh! Yeah, I bet. And it's probably hard for you to hire Chatino-speaking research assistants here in the US because I don't imagine there are a lot of them.
Hilaria: Well, it's not only that, but since in Mexico, as part of the creation of the nation state, their policy has been to integrate indigenous people into this national language, which is Spanish. So then when students go to school, the language of instruction is Spanish.
Gretchen: Right.
Hilaria: They don't know how to read and write in Chatino.
Gretchen: So even if they're speaking Chatino, you have to teach them how to read and write first?
Hilaria: Yes, that's right. If they can be of, you know, help.
Gretchen: Yeah, absolutely. And that's part of what you're doing this weekend?
Hilaria: That is right, yes. So then what happened, we continued to do research at the University of Texas, and we developed a very strong program of Chatino studies there. We used to call ourselves the Chatino Gang. There have been like eight dissertations on different Chatino languages that came out of the University of Texas from one or two very sporadic works in Chatino. There were like eight very in-depth studies. And one of those works was by Lynn Hou and Kate Mesh. They were studying sign language and gesture. Lynn Hou is a signer herself, and she will use transcribers in any spoken language, whether it’s English, Spanish, or Chatino.
Gretchen: Right.
Hilaria: So she was doing her dissertation on language acquisition in socialisation of deaf children in San Quiahije, in my community. She asked me to transcribe the audio interviews that she was doing with the families. And these were really lengthy interviews. But then I took that very seriously because, like I said, I'm Lynn's ears, and I have to do this transcription really faithfully so she can get access to this language. So in taking that work really seriously to allow her access, I began to do the transcription. But then, at that point, it became to me much more important to be able to have some tool that could help me because it was just a lot of work. So then I made a comment on Facebook, “Hey, you know what? I see that automatic speech recognition, it's just very developed in English and all of these languages, how can we get a tool to be able to transcribe this text in Chatino?” I really don’t care. I would love to just have a tool that says things in Chatino because they were repeating these things, “cha, cha,” all the time, and it was just like, “Oh my god. I just want to have something that could at least recognize a few words so that I don't have to type all of these words.”
Gretchen: I mean, because the estimates that I've seen for how long it takes to do a transcript are like one hour of transcriber work for one minute of audio.
Hilaria: Yes.
Gretchen: And that's the kind of work – so if someone has an hour of audio data, that's 60 hours of work to try to transcribe that one hour.
Hilaria: That's right. No.
Gretchen: Which is ridiculous!
Hilaria: Yeah, it’s very labourious. So then I began to ask people. In talking with some linguists, they will say, “Well, it's very difficult to do speech recognition in small languages,” because the models such as forced alignment, which is a model that they had been using at the time, needed hundreds if not thousands of hours of text, and we did not have that.
Gretchen: That’s the whole point of it being a small language, you don't have those kinds of resources.
Hilaria: Yes. So then I began to think, “Well, how can we make it – how can we speakers of minority languages make it, or facilitate, or invite these people who are doing this automatic speech recognition research to be able to do collaborations and to help us create tools?” So then I went to several meetings, and I met the people who ran Linguist List, Damir Cavar and Gosia Cavar. It seems like they have some interest in doing ASR, and it seems like when I talked to them, and I told them about the problem, that they said, “Oh yes, I think that this could be possible.” It seems like it wasn't a challenge for them. They invited me to IU, Indiana University, and one of the interesting things that we did with Damir and Gosia there, which I did not encounter before, was that Damir thought that we could entice people who were doing computational linguistics if we offer some data in open access.
Gretchen: Okay.
Hilaria: So then what I did there was that we had a little recording, and then I re-spoke many of the texts that I had transcribed for my dissertation first. I re-read them. Then we put them again into ELAN, and then we put all their – we annotated them with parts of speech, and cut and paste.
Gretchen: So you re-spoke them like in an audio booth so you'd have higher sound quality? Or was it just slower?
Hilaria: Well, we didn't have an audio booth. It was just a nice recorder.
Gretchen: Like a nice quiet room compared to being outside where they weren’t even recorded the first time?
Hilaria: Yes, it was a nice recording. And we had a good tape recorder basically.
Gretchen: Oh, okay, okay.
Hilaria: So I re- spoke them in a –
Gretchen: Like high quality, slow...
Hilaria: Yeah, something like that. I tried to re-read the text. And so we compiled a corpus of 3.5 hours, which we put in this program called GORILLA, where people can just download it, and they can use it to do any type of research that they want to. I thought that that was very clever and – because Damir says, “Well, we need to allow people to have a nice corpus so that they can use it if they wish to add a different language into their models.”
Gretchen: And so do people start using it?
Hilaria: This is how I came into contact with the people that I'm working with right now. At some point – also, Alexis Michaud, who works on a group of languages called Yongning Na, he was also asking the same question. He's a linguist, he's a phonetician, and he was working with these languages, and he also wanted to do some automatic speech recognition for the languages that he was working with.
Gretchen: Where are they spoken?
Hilaria: In China.
Gretchen: In China. Okay.
Hilaria: Yes. The Na languages are spoken in China, so he also put some high-quality data out –
Gretchen: Out there on the internet, yeah.
Hilaria: Yeah, out there in the internet. And that is how he got connected with Oliver Adams, who is one of the co-organizers for this conference that I'm doing right now. So Oliver Adams got in touch with...
Gretchen: Alexis.
Hilaria: With Alexis. So they began to do this collaboration, but then it came time when they wanted to fit the model with another language that was also a tonal language. We had this corpus that we had developed with Linguist List, which was –
Gretchen: Chatino.
Hilaria: – Chatino with open access with one speaker, me –
Gretchen: Which is also a tonal language.
Hilaria: Which is also a tonal language.
Gretchen: It’s completely unrelated to this language in China.
Hilaria: Yeah, and actually it’s spoken by a comparable size of population, like 40,000 people, kind of like that, 40-50,000 people. So that is how we began this collaboration.
Gretchen: And so is the idea to make tools that could work regardless of what the language is? Or you have to kind of – so it'll work on Yongning Na, it'll work on Chatino, it'll work on some other language, it doesn't matter? Or is it to figure out how much data you need to train a very small amount of data, and then it works specifically on the language?
Hilaria: Yes. Well, actually the methods that Oliver Adams is using is neural networks.
Gretchen: Oh, okay.
Hilaria: Yes. So he developed this software called Persephone. With Persephone, then, you can input data on – I guess in this case he was interested in tonal languages, so maybe he developed some tools so that the model could recognise tonal languages. That's why he fed two tonal languages into the model, to see what kind of outputs they had. It seems like with the corpus that Alexis was working with, the output was just excellent, because he used more data. But the output in Chatino was also very good. It's very promising.
Gretchen: So it's useful for you to take a first draft of a transcript or something?
Hilaria: I think that it’ll be very useful. I have not used it to transcribe new data, and this is the reason why at the retreat we're going to find out how can we, who are not technologically savvy people, start using and training these models with new data.
Gretchen: So at the retreat the goal is to bring together the automatic speech recognition people and the minority language documentation people and say, “Okay. How can we help each other? How can we make these tools that’ll work for everything?”
Hilaria: How can we collaborate? How can we make tools for language documentation? Yes. Because on the one hand, we linguists are not – we don't know how to operate these models, and the engineers, they know how to work these systems. So the two of us are going to come together, and we're going to have an honest conversation. We linguists will say, “How would you like us to prepare our data so you can use it for your models?” They will tell us and vice versa, “This is what we need.”
Gretchen: And you have people working on multiple different languages, and multiple different technology-type things, all together?
Hilaria: Yeah, that's right. In my conversations with Oliver Adams, right now our tools for major languages are very advanced. A lot of the problems have been solved. Actually, there are many sub-specialties within that field. For example, one of the interests that Oliver Adams has is multilingualism in ASR. So for him, this is so interesting because we're going to have different speakers. We're going to have speakers of Chatino languages, speakers of Mayan languages. Basically, what Oliver Adam says is that many of the differences sometimes could even be anatomical. He should explain what he means more, but...
Gretchen: So for multilingual automatic speech recognition, is that an automatic speech recognition tool that works for multiple languages at the same time?
Hilaria: Yeah, I think so.
Gretchen: So if you're speaking in Chatino one minute and Spanish another minute, and let's say you also happen to speak a Mayan language, you could speak to it in any of those languages and it would be able to pick up, correctly, whatever you were doing?
Hilaria: You know, I’m really new in this field, so I really cannot speak –
Gretchen: This is the hope, maybe.
Hilaria: Yeah, yeah. I think we need to ask the ASR people these particular questions.
Gretchen: Yeah. But it would be great if it would work for multiple languages. That would be really cool.
Hilaria: Yes, yes. Actually, this is the new frontier.
Gretchen: Yeah, that’s right. Because there's six, seven thousand languages in the world, and there's what, maybe ten that you have really good automatic speech recognition tools for right now?
Hilaria: That’s right, yes. Yes. But the thing is that, still, for minority languages, there are certain requirements that need to be there in place first. Like with Chatino, it was easy to do this because I have prepared the corpus. There is an alphabet we have for Chatino. So we have research in Chatino now, but many of those languages do not have this research available. So even if you have a sound file that is not transcribed in one language, it will not be useful for someone to –
Gretchen: Because you do need some training data.
Hilaria: Yeah, you do need training data, and also, you need a person to evaluate the output of the model.
Gretchen: Right. Because then you can't fix it if it’s...
Hilaria: That's right. For example, in the way we work with Oliver, it was that he put this data in, and then I as a person, as a speaker, I went out and just evaluated the output of the system. It has to be reciprocal.
Gretchen: And so what's – in 20 years when this is amazing and everything works great, what's the vision for how this works? Is it so people who speak Chatino can say, “Okay, Google,” to their phones in Chatino and it will reply back?
Hilaria: Well, I mean, people can give it many uses, right? I don't know if I can say what kind of uses people can give it if we were able to get to that point. But on a personal level, I would love to be able to have a tool that could help me transcribe text that I have. Because, actually, we have hundreds of hours of recordings of Chatino text, and it'll be wonderful to be able to have these transcriptions. And the results of these transcriptions could be fed into ongoing dictionaries to study the syntax of the language, to study morphology, or all the different aspects of the grammar.
Gretchen: Or to make books or these kinds of things in the language.
Hilaria: Or to make books in the language. For example, we have recorded many stories that – they're sitting there. We haven't been able to transcribe them. It would be nice if we had a nice transcription with the story, so then we can work with our artists and make children's books, and develop all of these materials to promote the language.
Gretchen: Yeah. Because you've made some books already, right?
Hilaria: That is right! We just had some books published with the help of many people. And I'm just so proud of this because this is one of the first times that I have seen children's books in Chatino. They are so beautiful, colorful.
Gretchen: They're really beautiful. You were showing them to me earlier and they’re really lovely.
Hilaria: This is a project that I did with my students in our language revitalization class that I taught in the winter 2018 at Dartmouth College. So one of the first things that we did was to do the drawings on cloth books. Each student developed their own theme, and they put it on cloth books. Then we had an exhibition, and then the exhibition was a success. It was really beautiful. People loved it.
Gretchen: The cloth books look so cool! They're soft and you can – you know, a baby couldn't destroy them.
Hilaria: That's right. And then I got some funding from the Neukom Foundation to do the publication in a different format of these books. One of the students had to draw pictures for many of the books because the originals were just images that students pulled out –
Gretchen: From the internet somewhere.
Hilaria: – from the internet somewhere, because we were not thinking forward about publications and things like that. But when we realised that we needed to publish them, and that Neukom was offering some funding to publish them, we realised that we did not want to get –
Gretchen: You didn’t want to get sued.
Hilaria: Yeah, sued! So now we have these new books with completely new images, and they are –
Gretchen: And they're lovely. And they're Creative Commons, and they're Open License.
Hilaria: That's right.
Gretchen: So you had a few of those books be translated into other languages that don't have enough children's books?
Hilaria: That's right. Because I had native speakers of North American languages in my class. I had my student who spoke Tlingit. There was another student who spoke Hupa, and Ojibwe. And when they saw this, they realised that they wanted to do the same version in their own languages.
Gretchen: That’s so cool.
Hilaria: It was just really amazing. So all of these books just came out.
Gretchen: So now there's this little link between the Tlingit speakers, and the Hupa speakers, and the Ojibwe speakers, and the Chatino speakers. They'll all have the same pictures in their books with the words in their own language.
Hilaria: It is just so amazing, you know? I went back to Mexico, and I took the cloth books down to Oaxaca, and there was this friend from my community who came to visit. I was visiting my mom in Oaxaca. He came to visit. And then I sat down with him, and I read him one of the children's books. And then at the end he says to me, “It is so sad,” he says, “that our language is getting lost.” That is – so really, the books really bring these conversations about the importance of language.
Gretchen: And if the kids are – because, probably, a lot of the kids still kind of speak the language at home, but then when they go to school and the only language they see written down is Spanish, whereas if they could see also written versions of Chatino so they could be bilingual, and know that there's people who care about the language, and give it more prestige, and these kinds of things.
Hilaria: That is right. I grew up in Mexico hearing that indigenous languages were not languages because they did not have a writing system. That is why I wanted to develop an alphabet to show that this is a legitimate language. By having these cute little books, it's –
Gretchen: And they look very professional, too. Like, they’re shiny. And they look very professional.
Hilaria: That’s right. Yeah. We wanted to make Chatino look good. So in this conversation that I had with this person from my community, I said to him, “One of the worries that I have,” I said, “if I distribute this book in the community, is that many of the books that I see, like textbooks that the schools give for free, they all end up in the toilet." So then I said, “One of the worries that I have is that my book will end up in the toilet.”
Gretchen: Yeah.
Hilaria: He said to me very seriously, “You know what? I'm going to tell you one thing.” He says, “I read the Bible. I do not take the Bible in the toilet. The Bible in my house has a special place. This book will be next to the Bible.”
Gretchen: What a compliment!
Hilaria: Yeah.
Gretchen: That is so meaningful.
Hilaria: It was just really beautiful, yes.
Gretchen: Yeah.
Hilaria: So I want to use these books to promote the language. One of the things that I would like to do, since this is a personal endeavour, and I don't have the backing of the state, I don’t have unlimited resources.
Gretchen: Yeah.
Hilaria: I would like to enlist families in the community to read the books, and then take videos of them interacting with the books and reading them with their children, take videos, and then, with their permission, upload them on social media, and in this way promote reading.
Gretchen: And they can see it. Because I think this is the thing is the technology space seems like it's so dominated by just a few languages, and to say, okay, this can be a language of technology, and this can be a language of writing and of the future that you can keep passing on to your kids.
Hilaria: That is right. Yeah.
Gretchen: Yeah.
Hilaria: I sometimes put little videos saying little phrases in Chatino. There are a lot of Chatinos who have migrated to the States. And they have children, and some of them are teaching Chatino to their children. Apparently I have some toddlers that follow my little videos.
Gretchen: Oh my gosh!
Hilaria: They just watch it over and over, and they repeat the words.
Gretchen: Oh my god, you’re like their teacher, or their grandma.
Hilaria: Yes. But I wish I could do more. It's just very sporadic.
Gretchen: Yeah. But that's still so cool. So if you can get other people making videos as well, maybe that helps.
Hilaria: Yes. Yeah, just make these books in different forms, like make little –
Gretchen: Or digital versions of them or something.
Hilaria: – animation, or things like that, yeah.
Gretchen: Yeah, that's very cool. I've taken a photo of the books already. So we will share a photo of the books, and we'll also link to whatever website or something you have set up for those. People can go see them, and you can see what they look like.
Hilaria: And you know what? One of the most important things about this is that this – as you say, these books have a Creative Commons License. So if someone out there would like to create children's books, they can use the same images, and just put their own text, and use the same things to publish their own books for their own language.
Gretchen: Yeah, that's really great. Hopefully you'll get photos being sent in from around the world of people doing that.
Hilaria: That would be amazing.
Gretchen: That would be amazing. Send Hilaria your photos if you end up using them.
[Music]
Gretchen: For more Lingthusiasm and links to all the things mentioned in this episode, go to lingthusiasm.com. You can listen to us on iTunes, Apple Podcasts, Google Play Music, SoundCloud, or wherever else you get your podcasts, and you can follow @Lingthusiasm on Twitter, Facebook, Instagram, and Tumblr. You can get IPA scarves and other Lingthusiasm merch at lingthusiasm.com/merch.
I can be found as @GretchenAMcC on Twitter, and my blog is AllThingsLingthustic.com. Lauren tweets and blogs as Superlinguo.
To listen to bonus episodes, ask us your linguistics questions, and help keep the show ad-free, go to patreon.com/lingthusiasm, or follow the links from our website. Can't afford to pledge? That's okay too. We also really appreciate it if you could rate us on iTunes, or recommend Lingthusiasm to anyone who needs a little more linguistics in their life.
Lingthusiasm is created and produced by Gretchen McCulloch and Lauren Gawne. Our audio producer is Claire Gawne, our editorial producers are Emily Gref and A.E. Prévost, and our production assistants are Celine Yoon and Fabianne Anderberg, and our music is by The Triangles.
Hilaria: Stay Lingthusiastic!
[Music]
This work is licensed under a Creative Commons Attribution- NonCommercial-ShareAlike 4.0 International License.
Do Siri or Alexa have problems understanding you? Learn why, with Carrie and Megan. They talk to Dr. Rachael Tatman about Automatic Speech Recognition (the t...
If you found our episode on ASR interesting, we have more information below. If you haven’t listened yet, find it here.
I’m interested in Rachael’s work on ASR and bias. Can you tell me more? Sure can: you can go to her own website or blog, or read some media about her work on regional dialects, race or gender. You can also find her on Twitter: @rctatman.
Where can I donate my voice if I want ASR to do a better job with other varieties of English? Mozilla. You can also read about it at the New Scientist.
What is the vowel merger you talk about in the episode? This is the ɔ/ɑ merger. Many English speakers make a distinction between caught [kɔt] and cot [kɑt]. Neither of us do. (In my version of Canadian English, both are [kɒt], and in Megan’s dialect they are [kɑt]. The difference is, my low back vowel is a bit rounded, and hers isn’t.) If you pronounce caught and cot the same, you have this merger too! (And if you distinguish them, you don’t.)
What are vowel shifts? Vowel shifts involve changes in many different vowel pronunciations. It’s a type of chain shift, where whole systems undergo a change together. The most famous shift is probably the Great Vowel Shift (around Shakespeare’s time), which is one reason the English spelling system is such a disaster.
What is the Northern Cities Shift? It’s a vowel shift found mainly in the Great Lakes region, including Syracuse, Rochester, Buffalo, Detroit and Chicago. There are 6 vowels that undergo the shift: [æ] (as in ‘cat’), [ɑ] (as in ‘cot’), [ɔ] (as in ‘caught’), [ɛ] (as in ‘bet’), [ʌ] (as in ‘cut’), and [ɪ] (as in ‘kit’). ‘Cat’ sounds like ‘kyet’, ‘cot’ sounds like ‘cat’, ‘caught’ becomes ‘cot’, ‘bet’ sort of sounds like ‘bot’, ‘bus’ sort of sounds like ‘boss’, and ‘bit’ sounds like ‘bet’. It’s confusing for people without this shift.
The “normal” vowel space for North American English:
The Northern Cities Vowel shift moves the vowels around a little:
The Northern Cities shift as a vowel chart (Labov, Ash & Boberg 1997)
(For those who’ve never seen vowel spaces before: imagine the front of your mouth is at the left, the back at the right, the top of your mouth is at the top and bottom, bottom. If you pronounce ‘bee’ [bi], your tongue is highest and frontest it can be.)
What is a waveform? A waveform is a two dimensional representation of a sound. The two dimensions in a waveform display are time (horizontal) and intensity (vertical). There’s a fun example here.
Carrie
——————————————————————————————-
If you have any questions or comments - especially if something isn’t clear - please let us know here, or by sending us an email at [email protected]. You can also follow us on twitter @VocalFriesPod or instagram @vocalfriespodor on facebook at Vocal Fries Pod. Send us your questions or suggestions for possible topics!
