#computer input

[awesomeassistivetechnology]

How to Write with Your Eyes

Nope, not kidding. I have actually been writing computer programs with my eyes (and a few voice commands thrown in here and there).

More specifically, I'm using an eye tracking device called the Tobii EyeX. It's a consumer-level eye tracker which mounts to the bottom of my computer monitor and connects to my computer via USB 3.0. (I've written about this type of eye tracking device before, and you can also look at Tobii's website.)

Okay, so how?...

Good question! And I'm going to answer it with the rest of this post. I'm going to write this in the general style of a how-to guide, in case any of my readers actually want to copy my set up. (To my knowledge, no one else has documented how to do this.) However, even if you have no desire to write Python with your eyes, you might still want to skim the rest of this. Several of the tools that I'm using have potentially much broader applications.

Hardware requirements: -Tobii EyeX sensor bar (link, < $150) -external monitor, preferably at least 20 inches but no more than 24 inches (You probably could make this work with a laptop, but having a larger screen area makes the eye tracker much easier to use. 24 inches is the size limit for the EyeX sensor.) -Modern computer with USB 3.0 support (I have a quad core processor at 3.6 GHz, 8 GB of RAM. I wouldn't recommend much less than that, unless you want to eliminate the speech commands from the project and use keystrokes instead.)

Software requirements: -Windows 7 64-bit (should work on Windows 8 or 10, as well. But you may have to make some adjustments.) -EyeX drivers and associated software (included with the device) -Dasher (free and open source) -FreePIE (free and open source) -Compiler and editor for the language you wish to program in (for this example, I'll be using Sage and its browser-based interface)

You will also need some sample code from the language you want to program in (so that Dasher can build an appropriate language model). You can ignore this if you only want to write in English (or another natural language).

Got all that? Good, let's get started.

1. Install the EyeX drivers and mount the sensor according to the manufacturer's instructions. Run the configuration to get EyeX to work properly with your monitor, and calibrate it to recognize your eyes. All of this is standard for any user of the EyeX, and you should be able to follow instructions from the manufacturer. When you're finished, you can test your calibration using the "test calibration" option in the EyeX software.

2. Download and install Dasher. We will make some configuration changes to Dasher in the next step, but for now just take a minute to familiarize yourself with Dasher if you haven't used it before. With the default settings, you can click the mouse inside of Dasher's window to start writing, and drag the mouse towards the letters you want in order to write. Your text will appear in the box at the top. For more details, see Dasher's website. (Obviously, skip this experimentation/exploration step if you aren't physically able to use the mouse. You can experiment with Dasher once we get it working with the eye tracker. I just included this step so that, as much as possible, new users can get used to one new technology at a time.)

3. (Skip this step if you just want to write in English, not computer code.) Close Dasher, and locate its program files on your hard drive. On Windows 7, I have a Dasher folder in Program Files (x86). Inside of that is another folder for my specific version, in this case Dasher 4.11. In your Dasher folder, find the folder labeled system.rc. We are going to add two new plaintext files to this folder. First, an alphabet file. This defines the all of the valid symbols for your language. I would recommend starting with an existing file, so make a copy of alphabet.englishC.xml and put it in the same folder. Name your copy alphabet.<yourlanguage>.xml, and open the file in a plain text editor. Make the following changes : -Change line 5 so that the text in double quotes is the name of your language (this will be shown in that the user interface). -Change line 9 by replacing 'training_english_GB.txt' with the filename for your training text. (I used 'training_sage.txt') -Remove the section labeled "Combining accents" (lines 13 through 27) -(optional and language-dependent) I additionally removed line 127 (the tab character) because my editor does auto-formatting with tab characters, and unexpected results were generated if Dasher inserted a tab character and then attempted to delete that character with the backspace key (which it regularly does if you take a "shortcut" to the character you want through another box). I decided that it was simpler to leave the autoformatting on in my editor (I really didn't want to manually insert six or seven tab characters before a line of code, as can happen in Python!) and insert my tabs manually with voice commands. You will have to decide how to handle these sorts of formatting and autoformatting issues yourself, based on the language and editor that you choose.

It is also at this point that you can reorder or regroup any of the characters if you want, in a way that makes sense for the language you are writing. Save your changes, and now you have an alphabet file. Now, let's specify the training text. Create a plain text file titled training_<yourlanguage>.txt (or whatever you called it in the alphabet file). Paste all of your sample code into that file. (More is better! But Dasher will also learn from what you write over time.) Save your training text file in the system.rc folder, just like the alphabet file.

Now, open Dasher and make sure that you can access your new language. Click on the Prefs button, and look in the alphabet selection box. You should be able to locate the language that you just specified. Click on it, click okay, and try writing in your programming language. If everything is working, proceed to the next step. Otherwise, go back and check that you got all of the syntax and file names/locations correct.  (A few troubleshooting details: If it works, but Dasher warns you that you have no training text, then your alphabet file is correct but you need to check the training text file (or file specified for training in the alphabet file). If it works, but the predictions seem bad, then you probably need more training text. If it works, but Dasher takes too long to load, try deleting some of your training text.)

4. Okay, now let's configure the Dasher user interface. Click on the Prefs button. Under alphabet selection, make sure your desired language is selected. On the control tab, select 'Eyetracker Mode' for 'Control Style'. Then, select 'Mouse Input' for the 'Input Device'. You can select the speed and choose whether or not you want Dasher to automatically adjust its speed. (I like this feature, but it's up to you.) Finally, choose the method(s) for starting and stopping. Personally, I like the "Start with mouse position – Centre circle" option for with the eye tracker. You might also want to enable the mouse button control and/or spacebar, especially until you get comfortable with the eye tracker (obviously only useful if you have a way to push the spacebar or click the mouse button). I would not recommend using the "stop outside of canvas" option, because it makes the motion feel extremely jerky when the eye tracker briefly leaves the active area. All of that is personal preference, though. There's no harm in experimenting to see what you like best.

5. More Dasher configuration. In the preferences window, click on the Application tab. Under "Application style:" select "Direct entry". This means that Dasher will automatically send keystrokes to the currently active window. This lets you write directly into your code editor of choice. Let's test again. With Dasher open, open Microsoft Notepad. Click in Notepad so that your cursor is in the text entry field. Then go to Dasher and (without clicking on anything besides the canvas – you don't want to change focus) start writing with the mouse. Your text should appear in Notepad. Once that's working, give it a try with your desired code editor.

6. Download and install FreePIE.

7. Let’s actually use this eyetracker! At the end of this post, I have pasted the code for a simple mouse control script. Take this script and paste it into a new script in FreePIE. Click on script menu and then run script (or press F5). (Tumblr doesn’t support any sort of code environment, so, if you have trouble getting the script to work, let me know and I can email you the actual file.) Then press the Z key to start the eye-controlled mouse movements. If everything is working correctly the mouse cursor should now follow your gaze around the screen. I've also set up this script so that you can start or stop the eye mouse using speech. If you have a decent microphone (preferably headset), make sure that that device is selected as your default recording device in Windows. Then, with the script running, you can say "start eye mouse" or "stop eye mouse" to start or stop the mouse movement. (If the computer has a hard time recognizing your voice, you can improve this by doing some of the training in Windows speech recognition. You can find it in the control panel.)

8. Now all that's left to do is chain all the pieces together. Get FreePIE open with your mouse script running. Make sure that Dasher is running, and also open your code editor. Make sure that the focus is in the text entry field of the code editor, press the or use speech to start the eye mouse, and finally start Dasher (by whatever you selected in Dasher's configuration). You should now be able to enter code with only your eyes.

9. (Optional) Depending on your needs, you may want to add some additional commands to the FreePIE script. For example, I have written some code that lets me enter tab characters by voice, and also navigate with in text by voice (e.g. "go to end of line", "go to beginning of line", "go up line", "go down line", etc.).  Each of these is just a couple of lines in FreePIE. Feel free to reach out to me if you need some help with adding these sorts of additional commands.

That's a lot of text, so let's finish off with a screenshot. Here's what the whole thing looks like when it's up and running. On the left-hand side of the screen you see Dasher. On the right-hand side is Firefox showing Sage. FreePIE is running but minimized.

[Image description: the image shows a screenshot of a Windows 7 desktop with two visible applications. On the left-hand side of the screen is Dasher. The majority of that window is filled with a collection of different colored squares and typable characters. There is a cross in the center of the window, and a red line shows where the user is currently pointing. On the right-hand side of the screen is a Firefox web browser window. The active tab is labeled Palindromes, and most of the window is occupied by text fields showing Sage code and output.]

The script for FreePIE:

#Use Z or speech to toggle on/off import ctypes import math def update():     global prevX     global prevY          #Settings     smoothingConstant = 0.2 #between 0 and 1; 0 means no cursor movement, 1 means use raw data               rawxcord = tobiiEyeX.gazePointInPixelsX     rawycord = tobiiEyeX.gazePointInPixelsY     diagnostics.watch(rawxcord)     diagnostics.watch(rawycord)          xcord = prevX + (rawxcord - prevX) * smoothingConstant     ycord = prevY + (rawycord - prevY) * smoothingConstant     if enabled:         ctypes.windll.user32.SetCursorPos(int(xcord),int(ycord))         global enabled          #enabled = False     prevX = xcord     prevY = ycord      if starting:     global enabled     global prevX     global prevY          enabled = False     prevX = 0     prevY = 0     tobiiEyeX.update += update toggle = keyboard.getPressed(Key.Z) start = speech.said("start eye mouse") stop = speech.said("stop eye mouse") if toggle:     global enabled     enabled = not enabled      if start:     global enabled     enabled = True      if stop:     global enabled     enabled = False

[awesomeassistivetechnology]

Low cost eye tracking enters the world!

Within the past year, multiple companies have released eyetrackers costing hundreds, instead of (tens of) thousands, of dollars. These new devices are aimed at gaming and multimedia applications, but they are still amazingly useful as assistive technology. The two main competitors at this point seem to be Eye Tribe and Tobii EyeX.

I purchased an EyeX unit this March, and it has significantly expanded what I can do with my computer. I will write later about the specifics of my setup, but I want to today just survey some of the development happening on the EyeX and other low-cost eye tracking devices (much software works across multiple devices) which is relevant to disability. (There is also significant gaming development on the platform, but it is largely using multiple input devices and would not be accessible to many physically disabled people. Plus,Tobii will tell you all about that.)

To be clear, I’m saying that these devices have great accessibility potential, not that they are currently ready made computer access solutions. At present, using these devices as an accessibility tool requires at least moderate technical skills.

Okay, without further ado, here are some software projects I find interesting:

OptiKey is an on-screen keyboard and mouse emulator specifically designed for eye tracking input. Free and open source. Windows only.

Project Iris uses eye tracking to control the mouse. It also allows you to define active screen regions and triggers actions when you look at them. Many potential uses for accessible gaming. Paid. Windows only.

Gaze Speaker is a self contained environment for speech generation by eye movements. Also includes other tools like an email client, a simple web browser, and a file browser, all designed to be accessed by eye alone. Free and open source.

FreePIE lets you write Python scripts that take input from many, many different devices, including the EyeX. You can use that input to trigger just about any action you can imagine, provided you can figure out how to write the code. Probably best for geeks, but extremely powerful for meeting custom needs. Free and open source.

[awesomeassistivetechnology]

Pointing Devices Part 2: Specifically Adaptive Options

Well, I can't call this soon, but I have finally written part two.  Part one can be found here.

In part one, I covered pointing devices that may be useful to people with disabilities but also reasonably easy to find, often because abled people use them, too.  Today I want to tackle the less commonly seen (and generally more expensive) domain of devices used almost exclusively by people with disabilities.  Please note this is not a complete list.  It's rather difficult to find good information on these topics, and new ideas constantly emerge from both researchers and assistive tech users.  I have included all of the relevant information that I have discovered in several years of researching assistive technology, but please do not hesitate to inform me if I'm missing something.

Head tracking systems: These systems track the position of the user's head and space, usually by optical means.  The head movement data is then used to position the cursor on the screen.  There are two main technologies currently in use.  The first uses a source of infrared light, which is projected at the user's head.  The user wears one or more reflective dots which reflect the light back towards an infrared camera.  Software is used to analyze the data from the camera and determine how the dot is moving. FreeTrack is notable as a free and open-source project doing this sort of head tracking.  (The project is designed to work with a web cam equipped with a homemade IR filter.)  The second approach to head tracking is based on visual light.  This technology uses an ordinary camera to record the user's head movements in real time and then uses image processing software to extract movement data.  A nice (free) example of this technology is CameraMouse.

Eye tracking systems: Similar in spirit to head tracking, eye tracking uses the movements of the eyes to position the mouse cursor.  Many different technologies have been used to track eye motion, but the most common technologies in use today are optical, based on reflection of infrared light.  This is very similar to head tracking, except that, instead of tracking a reflective dot, the system tracks the reflection of infrared light off of the various pieces of the eye.  Though there are some efforts at free and open source eye tracking systems, most currently available systems are very expensive.

Foot operated mice: These devices usually take the form of a boot or slipper worn on one foot.  As the foot is slid across the floor, the system registers the movement and translates it to cursor movement.  Such systems generally also include a set of buttons to trigger mouse clicks and possibly other actions.

Voice control: Developers of speech recognition software have often included some sort of mouse control feature in their technology.  Common variants include saying "move mouse right" and similar commands to control the mouse, or using a numbered grid to position the mouse on the screen.  While these systems do work in the sense that they do in fact move the mouse, they are not true pointing devices as they rely on discrete input events.  (Personally, I also find that they are barely usable from a practical standpoint!)  A different approach was taken by the developers of Vocal Joystick, which should still be considered a research project, though an alpha release of the software is available.  Vocal Joystick uses combinations of vowels sounds to continuously move the mouse cursor around the screen.  It's a very interesting concept, but usability must ultimately be determined by the end-users.

Mouse operated joysticks: While I did mention joysticks in part 1, I want to point out that there are some joysticks specifically designed for control with the tongue, lips, or teeth.  These can be quite useful for people with limited motor control and different variants are used for both computer work and gaming.

Continuous button-based systems: In these systems, one or more buttons are held down to move the mouse in the cardinal directions.  The most familiar implementation of this principle is in the mouse keys feature found in Microsoft Windows.  This accessibility feature allows the user to move the mouse by holding down arrow keys on the keyboard: as long as the right arrow key is held down, the mouse cursor moves to the right.  Similar systems exist using various modified button and switch types (different sizes, different activation forces, proximity switches, sensors that detect muscle activity, etc.).

Note that I elected not to include in my list of pointing devices various systems that I would call "pointing device emulators".  These systems use various methods of translating discrete events into mouse movements.  (The distinction is in the difference between continuous motions and discrete events.  All of the systems that I have discussed involve some sort of continuous motion on the part of the user which is translated as a continuous motion on the screen.)  While these systems are very useful for some people, they are beyond the scope of this post.

I hope these two posts have been helpful.  I've not been able to find this type of compiled information anywhere else on the Internet, so I hope this information is empowering to people who want to understand their computer input options.  Please feel free to send any further questions my way.

[awesomeassistivetechnology]

Dictation is a free online speech recognition software powered by Google Chrome. You can use Dictation to write emails and documents in the browser with your voice

This is a cool website which allows anyone with a microphone and internet access to use dictation software developed by Google.  It’s not perfect, but the recognition accuracy is pretty impressive.  I was seriously surprised at how well it worked! 

Unfortunately, there are no tools for correcting mistakes or editing your writing.

It only works with the Google Chrome browser (or the chromium browser).

I’ll be sticking with Dragon for now, but this may come in handy when I’m on the go and don’t have access to my desktop computer.

[awesomeassistivetechnology]

The AlphaGrip Ergonomic Keyboard and Trackball for gaming and computing; an all-in-one ergonomic keyboard, gaming keyboard, and ergonomic trackball enables the world's most comfortable PC gaming and computing

Has anyone tried this? It looks cool and I’d love to hear your experiences!

[awesomeassistivetechnology]

Pointing Devices Part 1: Fairly Common Options

[Images: Two trackballs.  The first features a grey sloping surface with a red ball set in one end.  The ball is approximately 1 inch in diameter.  To either side of the ball is a pair of buttons: one large button and one small button.  A cord emerges from the "top" end of the device.  The second trackball is much larger with an approximately circular plastic base about 6 inch diameter.  In the center is a yellow ball about 3 inches in diameter.  Two blue 1 inch circular buttons are set above the ball, and a cord emerges between the two buttons.]   Trackball: Trackballs essentially turn an old-fashioned mechanical mouse upside down.  Instead of pushing a mouse along the desktop, you roll a ball with your thumb, fingers, or palm.  (Many trackballs can also be operated with a variety of other body parts, for example a foot or forearm.) Trackballs are available in a wide variety of sizes and designs.  They are also available in both mechanical and optical varieties.  In general, optical varieties feel smoother and allow for more precise control, though they may also be more expensive.  I personally find trackballs far less painful than mice for regular use.  They can also be helpful for people with limited motor control or who need to use body parts other than their hands to operate a pointing device.  The picture above shows the Logitech Marble Mouse and a large trackball from infogrip.  I am especially fond of the Marble Mouse for desktop use.   Touchpads: Touchpads (or trackpads) are very familiar from their widespread use on laptop computers.  They generally work via an electromagnetic interaction with the fingertips and hence require very little pressure to operate.  Touchpads of a variety of sizes and designs are available and can be used with practically any computer.  Modern touchpads also have the advantage of supporting multi-touch and gestures, such as scrolling by swiping with two fingers.  The touch screens found on many phones and tablets are essentially touchpads layered with a screen.   Joysticks: Though joysticks are far more familiar in the videogame world, they can also be quite functional computer input devices.  The options here are quite varied, from thumb joysticks on videogame controllers to arcade style joystick operated with the whole hand.  (Some older laptops featured small joysticks in the center of their keyboards as pointing devices.  These only required very small motions, and similar devices can be purchased today and plugged into your USB port.)  For some people, combining a USB videogame controller with a free software utility (to map the joystick and buttons onto mouse controls) may be an awesome assistive technology solution.  For other people, buying a joystick explicitly intended as assistive tech which is designed to emulate a mouse may be a simpler solution.  

[Image: a Bamboo Wacom drawing tablet featuring a rectangular black surface and a smooth black pen with one rocking switch.]   Digitizer Tablets: Also known as drawing tablets, artists' tablets, or just tablets, digitizer tablets are most commonly used by graphic artists as input devices.  Unlike most pointing devices, which register relative motion, a digitizer tablet assigns each point on the tablet surface to a point on your screen.  (If you place the pen in the upper right-hand corner of the tablet then your mouse cursor will jump to the upper right-hand corner of your screen.)  Because of this behavior, digitizer tablets usually have a slower learning curve than other mouse replacements.  However, they can be extremely effective for some types of computer tasks, such as photo editing, text editing, drawing, and some types of games.  In general, this sort of input device will be good for people who are comfortable using pencils and pens but rather difficult otherwise.  (Though, it is worth noting that no pressure is required to operate the device.  The pen interacts with the tablet via electromagnetic interaction.  I have difficulties writing for extended periods but can use a digitizer tablet just fine.)   Part two coming soon!