#sensory augmentation

[mindumedia]

The application of entrainment

Many people feel uneasy at the thought of Nature manifesting creation according to irrefutable laws. Whereas it doesn’t take a Rocket Scientist to relate the clear and present evidence of Nature’s expression, and observing the universal principles of Nature’s cocreative confluence is self evident to even a child’s point of view, it does take a leap of faith for us adults to look beyond the…

View On WordPress

[quicktotheratcave]

How To Plug In A New Sense: Sensory Substitution With Android and Wearables

One of the coolest phenomena in sensory neuroscience is sensory substitution, or the brain’s ability to make sense out of non-natural sensory information. For instance, when an algorithm is used to convert video into an audio signal, people can learn to use the generated audio to navigate and pick up objects—and even activate the visual cortex while doing so! This ability is likely owed to the brain’s remarkable ability to detect statistical patterns, which allows it to quickly work out how patterns of tones correspond to the spatial features of the environment. Amazingly, this ability can extend even to senses that we don’t have; for instance one team found that people were able to improve their navigation performance when fitted with a device that communicated the direction of magnetic north through vibration. After six weeks of training, people reported not only learning to interpret the vibration signals but also developing a new intuitive sense of spatial orientation; tests of eye movements also showed that some subjects integrated information from the vibration signals into their automatic eye-stabilization reflexes.

The possibility of augmenting our sensory systems has become particularly exciting now that most people carry around a phone constantly connected to the internet and loaded with sensors capable of detecting everything from infrasound to ultraviolet. I recently started experimenting with this technology using a smartphone and a Wear OS smartwatch to actually deliver output (vibration) to my wrist. As a first “sense” I wanted to add, I developed a program that used the phone’s radio to detect the beacons broadcast by Bluetooth devices and translate them into specific vibration patterns that were played on the watch. While using the system initially just felt like weird buzzing on my wrist, after using the system for a few weeks I began to notice something like what the subjects in the magnetic-north trial did. I started to develop an intuitive ability to attribute the signals to specific device; to notice people moving behind me based on their Bluetooth transmissions, and to recognize specific people from specific Bluetooth patterns. Building a navigation app

While this was cool, it wasn't necessarily useful. To put together something more practical, I decided to build a smartwatch-based version of Feelspace’s original navigation app. The app uses the heading derived from GPS measurements to generate a vibration signal which corresponds to the cardinal direction you are moving, where the vibration frequency decreases as the travel direction goes from south to west to north to east.

If you want to try this app, you can download it here! You’ll also need to download Sixth Sense which provides the utilities to actually generate the vibration.

I also decided to convert the code that handles communicating with wearable devices and generating vibrations to a separate app (called Sixth Sense), which provides services that can be used by any app that wants to uses sensory substitution functions. Interfacing with wearable devices can be complicated and annoying, so having a dedicated “output” app allows many “sensing” apps to easily generate and execute vibration patterns without needing to know about the lower-level details. In addition to permitting other developers to easily write apps, this means the Sixth Sense system is platform agnostic—for instance, it can generate vibrations on a Wear OS smartwatch if one is present, but also use the phone’s internal vibration motor.

Building new sensory substitution apps

This section talks about developing apps that work with Sixth Sense. If you’re not interested in that it’s probably not worth it.

Sixth Sense allows any app to generate vibration patterns on a wearable simply by sending a broadcast as shown below

Intent intent = new Intent(); intent.setAction("neurelectrics.sixthsense.SEND_PACKET"); intent.putExtra("sequence", "50,0,50,0"); intent.putExtra("priority", 1); intent.putExtra("duration", 100); intent.putExtra("expires",System.currentTimeMillis() + 1500); sendBroadcast(intent);

This command will cause whatever device the user has selected to output vibrations to vibrate according to the parameters passed in extras described below. All parameters are required. Sequence: A String of arbitrary length, containing the vibration intensity at each time point in a comma-separated list. Vibration intensity can range from 0 (motor off) to 255 (maximum). Not all output devices support variable intensity, and these devices interpret any value above 0 as “on”. Therefore it’s important to make sure that important info is not encoded in sequences like “50,100,50,100”, which will just produce a continuous vibration on devices without intensity control. Priority: An int larger than 0 describing how time-critical this output is. Outputs with a smaller value will preempt outputs with a higher value. Because multiple apps can be sending data to SixthSense at the same time, an output can be preempted by an output from another app.

Duration: An int specifying how long each time point in the sequence lasts. This value must be less than 255. Longer point durations can be achieved simply by duplicating each point (i.e. a 500 ms vibration could be generated with a sequence of “100,100” and a duration of 250)

Expires: long value describing at what time this output should be discarded if it has not been transmitted, with the time specified in the format of System.currentTimeMillis(). Because outputs can be delayed due to system lag or preemption by higher priority outputs, the expires value allows you to control when the data is considered “stale” and should not be transmitted.

There are also a few other things to keep in mind when building sensory substitution apps.

Minimize memory load: While it’s possible to encode information in complex ways (i.e. intensity ratio between two pulses), these are difficult for users to learning because they require the user to remember the length of the first pulse while waiting for the second). Simple manipulations (like changing the frequency) are easier to learn.

Provide an option to control the speed of vibration sequences. Sequences can be easier to learn when they are initially presented slower.

Be cautious of high frequencies: The mechanical properties of many devices act like a low pass filter, so encoding critical information above 20 Hz is not recommended. Be mindful of power use. Continuously monitoring many sensors can drain the phone battery. The impact can be greatly reduced by optimizing for the circumstances when the app is used (i.e. NavSense turns on GPS only when the accelerator detects motion).

[those-difficult-things]

The Prosthetic Pigeon

About week ago, I assembled a haptic compass. It’s a strap that goes around your ankle with a little (~5cm x 5cm x 2cm) box of electronics velcroed onto it:

In the strap are 8 vibration motors like the one in a cell phone, and the north-most motor is constantly vibrating. In effect, the device gives you an extra sense: a coarse-grained knowledge of your absolute orientation. 

Or at least that’s the idea. The small study [pdf] that inspired it showed mixed results, with some subjects able to integrate the tactile information into their spatial processing much more thoroughly than others. The subjects whose performance on navigation tasks improved the most also reported some surprising new sensory experiences:

I was intuitively aware of the direction of my home or of my office. For example, I would wait in line in the cafeteria and spontaneously think: I’m living over there.

Interestingly,  when I take off the belt at night I still feel the vibration: When I turn to the other side, the vibration is moving too--this is a fascinating feeling!

So far I haven’t experienced anything dramatic, though I do get some phantom vibrations after taking the device off. Unlike the subjects in the study, I haven’t been wearing mine continuously; it’s a little loud (I should glue or tape the corners of the acrylic case and the battery wire, which both sometimes buzz) and the vibration can get a little too intense.

Still, being able to cast Know Direction at will is a pretty neat superpower.

[futuristech-info]

Thync can augment your mood - Induces a Red Bull like effect or relaxed few beers deep sensation

[quicktotheratcave]

Sensory Augmentation and the Pebble Smartwatch

Last Christmas I got a Pebble smartwatch. It's basically a watch with a microcontroller and some neat devices (bluetooth module, accelerometer, etc) that can run apps and communicate with your phone. Unfortunately, I didn't have much of a chance to really dig into it for the next few months due to being busy with finishing up my thesis research project, but I recently took a crack at writing some software for sensory substitution on it.

Sensory substitution is an incredibly cool technology. The Wikipedia article explains it much more comprehensively than I do here, but it's essentially a way of using a computer system to transduce information from one sensory modality into another one. (For instance, researchers have restored some sight in blind people using input from a video camera connected to a grid of electrodes on the tongue, and found that their brains adapted so that information from the tongue array was processed in the visual cortex.)

The wild thing about this is that there's no reason why the sense that you're adding has to be one that people normally posses. In other words, the same technology could be used to give people an artificial sense of magnetic north, or the ability to “see” in infrared, given appropriate sensors and “transducers” to connect the sensors to normal sensory inputs such as touch.

Unfortunately, the transduction step is in many ways the most difficult. Devices like tongue electrodes aren't really suitable for wide adoption, and interfacing a sensory device to the skin brings with it other problems like power consumption, maintaining a reliable connection, and making the device sufficiently unobtrusive and durable. Currently, the best method for creating a reliable “input port” for these types of experiments requires surgically implanting a small magnet within the hand which can then be controlled by using an oscillating magnetic field generated by a coil over the skin. That's cool, but hardly a practical solution for most of us.

In many ways, the Pebble is actually a pretty good platform for creating a non-invasive version of this technology. It's energy efficient, capable of communicating with other devices to get data, contains a motor capable of creating tactile sensations, and is something that people wear everyday already. Could I create a Pebble-based sensory substitution device? It seemed simple enough.

The first step was deciding exactly what I wanted the device to substitute. The Pebble contains a magnetic field sensor (similar to the ones in smartphones), which I initially wanted to use, but unfortunately the API does not yet support accessing it. While feeling magnetic fields would have been cool, I ultimately decided on something easier to measure: the network connection quality between my phone and a remote dedicated server. Because while on campus my phone usually jumps between wifi hotspots of wildly varying quality, I figured it would be interesting to be able to perceive the zones of weak and strong signal.

The next step was designing how that information could be communicated. While an obvious choice would be to make the strength of the vibration simply depend on the connection quality, this suffered from an unexpected drawback: the Pebble's vibration motor makes a loud and quite distracting “rrrrrrrrrrrrr” sound when it's fully spun up. Because this sound can be almost completely avoided by running the motor in short bursts (less than 100 milliseconds), I chose a different coding scheme: every second the watch gives two very short vibration bursts. The intensity of the bursts is determined by the current network speed (measured by downloading a file with a known size), while the interval between them is determined by the network latency (this time to establish a connection with a server, which reflects primarily the delays from switching and wireless interference)

Did it work as a sensory substitution system? I've only been using it for a few days, but so far it's very interesting. There's a noticeable subjective difference between being on a 3G network and a wifi net (the 3G has higher latency but is more consistent, the wifi feels faster and more “spongy” for lack of a better word) and a good connection and a poor one, and I've spent a lot of time walking around noticing things like the “radio shadow” of my campus's brick buildings. While using the motor constantly does drain the Pebble's battery much faster than normal (I went from about a 6-day battery life to about 1.5 days), it's not an enormous problem given that I usually recharge it overnight anyway. I'm planning to try it out for a while and see to what degree it becomes

If you'd like to try the app, its available on the Pebble app stores here.

[guyfellow-blog]

Eidos

[thinlyspread]

The Ape in Me

I was pulling out of a parking space in the lot behind Octane a few days ago, when I looked to my left, where there was a large, silvered (mirror-like) window. Looking in the window, I could see the reflection of my entire car, but not myself in the car. Of course, I could see my car respond to every turn of the steering wheel I made. I had a moment of extreme cognitive dissonance, where I suddenly felt like my "self" included the car.

As children, we partially identify our sense of self by seeing (directly or through mirrors) what aspects of our environment change when we direct our body to move. Even as adults, if we're not certain that a reflection is ours, we'll wave our hands or shift our heads in a distinct way in order to determine if the reflection matches our movements.

If we begin to integrate sensory inputs from remote sensors (for example, cameras on multiple UAVs) directly into our normal complement of sensory input, will our brain alter our sense of self to be more abstract? Direct integration of alternate sensory input doesn't seem all that far in the future, considering the technology BrainPort is developing. By mapping the video from a camera to a grid of electrodes stimulating the tongue, BrainPort has been able to augment the visual capabilities of blind people. Considering the plasticity of the brain, it doesn't seem too far-fetched to consider humans as becoming clouds of sensory input.

Thoughts?