#word recognition

[lessonplancurator]

[fortunejoy]

This video is private.

[appliedfunctions]

I was searching for Star Wars memes, and I saw this beauty. The asymmetry between recognizing speech and talking sets off so many associations for me. 

It reminds me of the difference between receptive and expressive vocabulary. The number of words that we speak or write on a day-to-day basis is much smaller than the words we can read or comprehend. Which is the truer measure of vocabulary knowledge? Or maybe that question is a red herring, and it doesn’t make sense to partition our words into the ones we know well enough to use in a sentence versus the ones we know well enough to understand in a sentence. 

The meme also makes me think of Hinton’s work in deep learning: “To recognize shapes, first learn to generate images”. In a digit-recognition network, you train your model to recognize (to correctly label) a digit from a pixel image. Traditionally, you use supervised learning, so that the model starts guessing labels for words immediately and learns by minimizing its error rate. Hinton instead suggests that the model undergo unsupervised “generative pretraining” first and learn to generate those images. With this training, it can develop discover a useful set of general features and structure in the images. Once those features are in place, learning to label images become a matter of “discriminative fine-tuning” for the network. I still haven’t wrapped my head around the recognition/generation asymmetry in this domain, admittedly, but the meme reminds me of it. Maybe it’s a difference between seeing and imagining.

Lastly, the meme reminds me of P vs. NP, the question in computer science about different kinds of computational problems. The problems in P can be solved efficiently (P for polynomial time) like sorting a list or multiplying numbers. The problems in NP can have their answers verified efficiently (NP for nondeterministic polynomial time) like Sudoku, Minesweeper or subset sum. You don’t need to solve the Sudoku again to check the answer; just check each row, column and cell for duplicate numbers. The problems in P can have their solutions checked efficiently because they can be solved efficiently: Just solve the problem again and see if the solution matches the new answer. The million-dollar question is whether the family of problems that can be solved efficiently equals the family of problems that be verified efficiently. (The answer is probably not.) Here the production/recognition asymmetry translates into a difference between solving and checking and the question of whether there are problems that are easy to check but inherently hard to solve.

So, that Star Wars meme--it’s a real tapestry of meaning if you ask me.

[nicolausen]

[appliedfunctions]

Word Recognition Notecards

These are some "notecards" for Magnuson, Mirman, and Myers (2013).

Gating Incrementally play longer and longer chunks of a word. Listeners guess the word that finishes the token. For example:

b

ba

ban

bani

banis

banist

banister

Listeners provide a variety of guesses in the early gates (-> activation of many words in parallel). Gate 5 is the uniqueness point; there is only one completion of the word. The recognition point (gate of correct guessing) often precedes the uniqueness point (-> incremental activation of words). The amount that recognition precedes uniqueness is probably determined by the target's word-frequency (more frequent words guessed earlier). 

Lexical Decision Play a wordlike stimulus. Listener decides whether stimulus is a word or not. Measure accuracy and response latency. Sensitive to word frequency (more false rejects of less common words) and neighborhood density (probably, slower to make decision when stimulus overlaps with a lot of words). Unlike real recognition because it's a post-perceptual judgment about stimulus. Also a listener can make a lexical decision without knowing the meaning of a word (I've heard people say tamp but I don't know what it means).

Cross-Modal Semantic Priming (e.g., Castle-Candy-Sweet Priming) Listeners perform lexical decision on printed words. Play a spoken stimulus before some trials. For example, play castle then show sweet. Listeners perform faster on these trials. Huh? Because castle activates a neighborhood of ca- cohorts (cat, cabin, candle, candy), and those cohorts in turn activate semantic relatives (candy partially activates sweet). Marslen-Wilson and Zwitserlood (1989) showed cross CSMP effects for phonological cohorts (castle-candy-sweet, yes) but not for rhymes (castle-hassle-bother, no), supporting Marslen-Wilson's Cohort Theory of recognition.

Naming / Repetition / Shadowing Play a wordlike stimulus. Listener repeats word. Measure accuracy and response latency. As with lexical decision, word frequency and neighborhood density influence outcome measures. Unlike real recognition because it's post-perceptual and a listener may pay really close attention to the speech sounds (without semantic processing).

Similarity Metrics Speech signal activates a set of words based on their similarity to signal, and activated words compete with each other. Words with relatively sparse activation neighborhoods will undergo less competition than words from denser neighborhoods. But how do we define similarity?

We could use cohorts (words which share the first 1-2 phonemes) as the similarity metric to define the set of competitors. We could use segmental neighbors. The DAS metric includes all words that differ by Deletion, Addition, or Substitution of just one phoneme. The Graded Similarity metric measures similarity using phonetic features. For cat, words differing in just one feature are most similar (cap, pat), and words differing by two features are less similar (cab, bat), and so on. This metric would conclude that veer and bull are neighbors because they differ in just three features even though don't share any specific phonemes. (I would use the graded similarity metric but weight changes by word position and also adjust resting activation based on word frequency.)

My diagram above illustrates that the similarity metrics overlap and disjoin at different locations. For example, the segmental and phonetic metrics limit the neighborhood to words that are about the same length.

Frequency-Weighted Neighborhood Probability (FWNP) Calculate the similarity of target word t to the utterance x, and weight that score (multiply) by t's log-frequency. Call this the frequency-weighted similarity score (FWSS). Compute the FWNP by dividing t's similarity score by the sum of the FWSSs for every word in the lexicon.

$$FWNP=\frac{f_{t}S_{tx}}{\sum{f_w{S_{wx}}}}$$

If we use the DAS metric, then similarity is a binary measure. Words that differ in more than one phoneme have a FWSS equal to 0. All words that differ by up to one phoneme have a score equal to 1. This adjustment simplifies the equation to a ratio of a word's frequency over the summed frequencies of its neighbors. I think of it as a fish-pond metric.

$$FWNP=\frac{f_{t}}{\sum{f_n}}$$

Ganong Effect Lexical context influences perception of a phonetic category boundary. For an intermediate /k/-/g/ sound X, listeners perceive a /k/ is Xiss and a /g/ in Xift. Evidence for top-down lexical effects. (Ganong, 1980)

Compensation for Coarticulation Phonetic category boundaries shift when listeners try to compensate for coarticulation. For an intermediate /t/-/k/ sound X, listeners are more likely to perceive a /t/ in [ʃXa] but a /k/ in [sXa]. (See Elman & McClelland, 1988)

[appliedfunctions]

Last winter, I wrote an implementation of the TRACE model of word recognition in pure R. I developed the package as my final project in my course on parallel distributed processing (taught by Tim Rogers). Yesterday, I finally got around to putting my report on the project online.

The above figure is from the Ganong effect demo in that paper. It shows the activation of phonemes and word units over time when the network was presented with “Xlug” where “X” is an intermediate sound between /p/ and /b/. The network initially guesses “blood” and “plug”, but decides on “plug” once /g/ arrives. Afterwards, top-down connections from “plug” to /p/ cause activation for /p/ to rise above /b/.

Programming postmortem

I wrote a naive implementation using message-passing object-oriented programming. Each network node is a bundle of data, and we send instructions (messages) to these nodes to tell them to collect input from neighboring nodes or to update their activation values.

This implementation was partly inspired by Jeremy Kun’s implementation of backpropagation in Python. His implementation convinced me to start with a few nodes, make sure they behave as expected, and bootstrap from there. And that's how this implementation developed: I first created toy nodes to implement generic functionality, then figured out input nodes and feature detector nodes, wrote some functions to assemble a layer of those nodes, and then iterated to phoneme and word nodes/layers. Very interactive and organic, bootstrapped from the bottom up.

For the #rstats people out there, I create the nodes as R6 objects. R objects normally have copy-on-modify semantics so that when I update x$value, I get a new copy of the object x. R6 objects have reference semantics, meaning that they seal off a chunk of memory. I'm not sure why I decided on R6 last year, but I probably reasoned that node$update() made more sense than node <- update(node). I also convinced myself that R6 objects would yield better performance because thousands of nodes wouldn't have to be copied on every network tick.

Unfortunately, the naive implementation doesn't scale. When I tried to regenerate the plots yesterday, it took an hour for one network to complete 60 ticks. (I don't remember it being that slow last year, but oh well.) I could try to optimize the pain away, but that would require profiling to find the pain points. Alternatively, I could ditch naive implementation and use a couple of data-frames: Store the nodes in a giant data-frame, and use split-apply-combine operations to e.g. summarise all the incoming activations to each node.

Oh and another thing

My re-implementation was not a direct port of any other implementations. That is, I didn't translate some source code into another language. Instead, I based my implementation on the description in the original paper. This top-down approach led to the most annoying part of the project.

Acoustic features like voiced, consonantal, vocalic, etc. are implemented in TRACE using values on a continuum from 1 to 8. A fully vocalic sound /a/ get an 8 for vocalic, but a continuant sound like /s/ is kinda vocalic so it gets a value of 4. The original TRACE paper leads one to believe that each sound has just one value for each feature:

Based on the text, I implemented /k/'s acute feature as a vector [0, 0, 1, 0, 0, 0, 0, 0]. WRONG! That's not what they did in 1986. The original C-TRACE code defined the acute feature for /k/ as [.1, .3, 1, .3, .1, 0, 0, 0]. These smeared feature values popped up somewhere in the specs of each phoneme, so all of my phoneme definitions were wrong. This mismatch between the description in the paper and the original code caused a lot of headaches as my simulations failed to obtain expected results. That was the most frustrating part of the project, illustrating a disadvantage of my paper-first strategy of implementating TRACE. A write-up can gloss over details, but the code doesn't lie.