#underrepresented languages

[allthingslinguistic]

You, a person who's currently on the English-speaking internet in The Year of The Pandemic, have definitely seen public service information about Covid-19. You've probably been unable to escape seeing quite a lot of it, both online and offline, from handwashing posters to social distancing tape to instructional videos for face covering. But if we want to avoid a pandemic spreading to all the humans in the world, this information also has to reach all the humans of the world—and that means translating Covid PSAs into as many languages as possible, in ways that are accurate and culturally appropriate. It's easy to overlook how important language is for health if you're on the English-speaking internet, where "is this headache actually something to worry about?" is only a quick Wikipedia article or WebMD search away. For over half of the world's population, people can't expect to Google their symptoms, nor even necessarily get a pamphlet from their doctor explaining their diagnosis, because it's not available in a language they can understand. [...] According to a regularly updated list maintained by the Endangered Languages Project, Covid information from reputable sources (such as governments, nonprofits, and volunteer groups that clearly cite the sources of their health advice) has been created in over 500 languages and counting, including over 400 videos in more than 150 languages. A few of these projects are shorter, more standardized information in a larger variety of global languages, such as translating the five WHO guidelines into posters in more than 220 languages or translating the WHO's mythbuster fact sheets into over 60 languages. But many of them, especially the ones in languages that aren't as well represented on the global stage, are created by individual, local groups who feel a responsibility to a particular area, including governments, nonprofits, and volunteer translators with a little more education or internet access.

Covid-19 Is History’s Biggest Translation Challenge (Gretchen McCulloch, Wired)

[allthingslinguistic]

The many languages missing from the internet

English and a handful of other languages dominate the internet, but this is leaving indigenous cultures without a voice online. Now they are fighting back.

An article in BBC Future about linguistic representation on the internet. 

The Kaqchikel Mayan community from Guatemala includes more than half a million speakers. Miguel Ángel Oxlaj Kumez is part of it and was one of the organisers of the first Latin American Festival of Indigenous Languages on the Internet, held in 2019.

“When I get on the internet I find more than 90% of the content in English and hence a significant percentage in Spanish and other languages,” he says. “So what I have to do is to move to another language, and that favours the displacement of my own language.”

“It discredits my own language, because – as it is not on the internet – then it is not valid, then it does not work, therefore why am I going to continue learning it? Why am I going to teach it to my children if, when I turn on the internet or television, I cannot find it there?”

Oxlaj Kumez is working with other activists to create a version of Wikipedia in Kaqchikel Mayan, as well as a translated version of Mozilla’s Firefox web browser. His dream is to be able to have a “digital life in my own language, and when I decide to move to another language that it will be my decision”.

He is not the only one with that dream. In 2003, Unesco adopted a recommendation to promote the use of multilingualism online. Ever since, the organisation has been pushing for universality on the internet, with a special focus on indigenous languages.

Read the whole thing. 

[allthingslinguistic]

Two recorded talks from typography conferences about increased cross-linguistic font support, from Zachary Scheuren, one of the people working on the Google Noto font which aims to support all the writing systems of the world. Description of the first talk, Invisible Letters, Invisible Languages: 

If we can’t see a language, does it exist? Of course it does, but in an increasingly computerized world, many languages remain invisible. Some languages have no writing system to be seen. Others are not necessarily accessible in the digital realm. What are the implications for these languages in light of today’s technologies? Computing in general is grounded in the assumption that spoken language can be transcribed and manipulated. But who are we leaving behind and why should we care? The benefits of digital representation aren’t limited to the locale of a particular language, script, or culture. The possibilities range from linguistic analysis to all forms of cross-referencing across disciplines like linguistics, history, archaeology, literature, sociology, economics, anthropology, and more. This presentation will look at a number of historic and minority writing systems and consider what can be gained by digitizing them and making them accessible to the world.

Watch on youtube. 

Here’s what the second talk (Writing Systems of the World) is about: 

There are hundreds of writing systems in the world, some no longer in use, some in use by small groups, and some (like the words you are reading now) in use by larger groups. Many writing systems have been encoded in various ways for use in computing, but encoding is just the first part of the process. Without fonts and proper support in applications a writing system might not be usable at all. The current state of technology allows us to do so many amazing things yet many writing systems are not adequately implemented for use in computing. How can it be that some scripts are so difficult to use? What is standing in the way? Why are shaping engines so complicated? (They aren’t). Why is dealing with text encodings so hard? (It shouldn’t be). Why are there little to no fonts for some writing systems? What can we do to move forward? 

 This presentation will provide an overview of world scripts and the common problems that keep some scripts from being used. We will then look at what needs to be done to remedy this and how everyone can help.

Watch the whole thing on youtube.

[allthingslinguistic]

Celebrating “Zero to Digital” | PanLex

Translation Commons has published "Indigenous Languages: Zero to Digital", a guide for creating the digital infrastructure to bring a language online.

A post on the PanLex blog about the steps involved in creating digital language tools for under-supported languages, including the Zero to Digital guide that breaks down in detail exactly what steps are involved. Excerpt: 

Many of us who use a well-supported language online may not realize how many layers of technology underpin the implementation of language on our devices. When we buy a new phone or laptop, we barely notice these layers, because our language usually functions and displays seamlessly on the device without hiccup. But that’s not the experience for speakers of the under-resourced languages, even including some with millions of speakers. For example, Lahnda (Pakistan) with 93M speakers and Wu Chinese with 81M are not well supported.

Here are some of the foundational layers that must be implemented behind the scenes for a language to be digitally supported.

 - A writing system (such as Latin, Devanagari, or a newly created system) and orthography (spelling rules) must be identified, chosen, or developed for the language.  - Each letter or character of the writing system (like capital B, lowercase c, or !) must exist in Unicode, giving each one a standard numeric code point.  - A font that displays all letters and characters in the writing system must be developed.  - Using typography design software, font designers identify and create all necessary glyphs, or graphic shapes that represent the letters and characters of the writing system.  - The font designers write rules to handle any required ligatures and other complex cases in order to combine, stack, or connect the glyphs properly, in accordance with the writing system’s conventions. (See graphic below.) This step can be labor intensive for complex writing systems.  - The font must be available on a device at the moment the user needs it. Keyboards must be created so that text can be input in a language. These can be either on-screen keyboards or mappings from common physical keyboard layouts such as QWERTY.

Read the whole post or view the Indigenous Languages: Zero to Digital guide directly.  

[allthingslinguistic]

Wikipedia Is Helping Keep Welsh Alive Online

Welsh and other smaller language movements on Wikimedia projects suggest there may be ways to train technology to allow for cultural differences.

An article in Slate about the role of Wikipedia in creating language tools. Excerpt: 

Although Alexa still does not speak or understand Welsh, the Celtic language’s presence in tech has increased dramatically within a short period. Google announced in February that it had expanded its offerings in Docs, Sheets, Slides, and Drive to include Welsh. And Google Translate—infamous since 2009 for its Scymraeg, or scummy Welsh—has, according to the BBC, recently taken a great leap forward in terms of the accuracy and quality of its Welsh translations. Morlais and others attribute this in part to the fact that there are now more than 100,000 articles on the Welsh version of Wikipedia, known as Wicipedia.

Like other language editions, Wicipedia is a separate website with its own content, not simply a translation of English Wikipedia, a distinction that matters for both users and big tech companies. Back in 2017, Morlais observed, “There appears to be an indication that there is a link between the languages with the most Wikipedia articles or pages and the languages that are supported by the digital giants.” Google Translate and other technologies use artificial neural networks to learn from example, training themselves with language data from rich internet sources like Welsh Wikipedia.

The Welsh community is not alone in using wiki-technology to promote its language. This year’s Celtic Knot conference in Cornwall, England, included several indigenous languages with their own Wikipedia editions. The original idea, as the name suggests, was to focus on Celtic languages, including Irish, Scots, Breton, Welsh, and Cornish, which was declared extinct merely a decade ago. But as word got out about a Wikipedia minority language conference, others began to join, representing, for example, the Sámi language spoken in parts of Norway, Finland, Sweden, and Russia; the Berber family of languages spoken in Northern Africa; and the Basque and Catalan communities. (In his 2017 presentation, Morlais noted that Catalan was one of the few minority languages supported by Google search, an accomplishment he linked to the fact that Catalan already had more than 500,000 articles on its language edition of Wikipedia.)

Read the whole thing.

[allthingslinguistic]

A Focus on Machine Translation for African Languages

Masakhane is an interesting project to make machine translation more available in languages of Africa. From the description: 

We need African researchers from ACROSS the continent to join our effort in building translation models for African languages. Masakhane means "We Build Together" in isiZulu and was inspired by the Deep Learning Indaba theme for 2018.

Phase 1

We want to develop baseline machine translation models from English-to-Target African languages, with publicly available code and data. This problem is part data gathering, part developing the translation model, and part error analysis to understand what issues the models have

Phase 2

Once we, as a collaborative African NLP team, have trained baseline translation models for many of our languages, we combine our datasets and do transfer learning with fine-tuning across the languages.

Phase 3

Write & submit a paper, with all of our work, to a top-tier NLP conference and in doing so, once and for all put Africa on the NLP map

Read more about the project. 

[allthingslinguistic]

Why do Greek, Czech, Hungarian, and Swedish, with their 8 to 13 million speakers, have Google Translate support and robust Wikipedia presences, while languages the same size or larger, like Bhojpuri (51 million), Fula (24 million), Sylheti (11 million), Quechua (9 million), and Kirundi (9 million) languish in technological obscurity? Swedish, Greek, Hungarian, and Czech have a wealth of language resources, created one human at a time over centuries. They're the languages of entire nation-states, with national TV and radio recordings that can be used as the foundation for text-to-speech models. Their speakers have the kind of disposable income that makes media companies translate popular novels and subtitle foreign movies and TV shows. They're found in countries that tech companies imagine their customers might be living in or might at least visit on holiday, meaning it's worth localizing interfaces and adding them as translation options. They have regularized spelling systems and dictionaries that can be rolled into spellcheckers and predictive text models. They have highly literate speakers with internet access who can contribute to projects like Wikipedia. (Speakers who can even, in the case of Swedish, create a bot to automatically make basic Wikipedia articles for rivers, mountains, and other natural features.) Language resources don't just appear. People have to decide to create them, and those people need to be fed and watered and educated and housed and supported, whether that's by governments or by companies or by the kind of personal wealth that lets individuals take on time-consuming intellectual hobbies. Creating parallel corpora and other language resources takes years, if it happens at all, and cost tens of millions of dollars per language.

Gretchen McCulloch, The widely-spoken languages we still can’t translate online. (My latest article as Wired’s Resident Linguist.)

[allthingslinguistic]

A paper from the Workshop on the Use of Computational Methods in the Study of Endangered Languages (ComputEL-3) about making better tech tools for linguistic annotation, featuring many ideas that I wish I’d had in grad school! 

Abstract: 

To reduce the annotation burden placed on linguistic fieldworkers, freeing up time for deeper linguistic analysis and descriptive work, the language documentation community has been working with machine learning researchers to investigate what role technology can play, with promising early results. This paper describes a number of potential follow-up technical projects that we believe would be worthwhile and straightforward to do. We provide examples of the annotation tasks for computer scientists; descriptions of the technological challenges involved and the estimated level of complexity; and pointers to relevant literature. We hope providing a clear overview of what the needs are and what annotation challenges exist will help facilitate the dialogue and collaboration between computer scientists and fieldwork linguists.

Read the whole thing.