You Can Now Write by Blinking Your Eyes!
Introduction The title is absolutely correct, we’ve developed a prototype for a software application that enables you to write text just by blinking your eyes. Not only that, but it also puts that text on a skype conversation and sends it all by it self! At this point you, the reader, must be wondering -“Why??”. Well the answer is quite simple really: fear. Have you ever imagined what it must be to find ourselves immobile, unable to properly articulate ourselves and communicate with others, to move and to perform the simplest of actions? To be practically trapped inside our own minds! It is a kind of situation that a healthy person cannot even begin to understand or evaluate. The fear of ever being on such a condition provided enough motivation to do a “state of the art” and figure out what solutions are already available to enable communication and access to today’s modern internet and social technologies. The internet is a de-facto gateway to the world, to other people, to entertainment, culture, news, art, etc., so it makes sense that any technological interaction solution takes that into consideration. The good news is that there are plenty of system available, using blinks, eye movement and even facial nerve movement to provide interaction. The bad news is that there is a direct proportion between quality (accuracy, precision, reliability) and price. In a nutshell the best systems are expensive (tens of thousands). This situation of course didn’t put our minds at ease. Then we noticed one of our brand new toys sitting around on the table and realized we had the technology and the know-how to built something that may actually help fixing this issue, or at least provide a new angle from where we can approach the problem. The toy in question is Neurosky’s Midwave Mobile EEG headset, a rather inexpensive and quite accurate device (for it’s price tag), that can be plugged in to almost everything (Pc, Mac, iOS, Android) via Bluetooth. We’ve bought this device a few months ago for recreational and scientific purposes, in fact we grew quite fond of it’s blink detection feature, since it is rather reliable. Now we had a new usage for it. The question Our problem was to devise a way, using cheap and established technological resources, to help severely handicapped or paralyzed individuals to communicate with world and use the internet. Our solution was to be something available to everyone (in terms of technology and price), something reliable, easy to setup and to use. The answer The solution to our problem was something we believe to be simple and effective: an software application that uses a EEG device (such as Mindwave mobile) to detect eye blinks. Those blinks would be translated into actions by the software, ranging from writing a word, to placing that word on a skype window, a full text on Gmail or a search query on Google. We also decided that software such as this shouldn’t have a price tag, it’s something that can make a difference on someone’s life, not a luxury or a recreational item. You can’t put a price on something like that, so our application’s source code will be released to the community (on GITHub) and hopefully will grow into a fully-featured opensource software tool. Our goal on the end of this project is to have developed a tool that answers our question, that actually helps people to communicate and that is free to use and to evolve. Aside from Neurosky’s SDK elements, we have also used Brian Peek’s awesome ThinkGearNet .Net library to handle the communication between the Mindwave headset and the application. Enter BlinkWrite The tool’s name is BlinkWrite, since it’s primary focus is to enable people to write using blinks. I must remind the reader that at this point, the available version of the tool is prototype/proof-of-concept not a release version, so some features are indeed missing and some bugs may be lurking around. For now it is a simple windows application without any design effort (except on terms of usability). At this moment BlinkWrite enables the user to perform the following actions:
Write text in English
Copy-Paste that text into a Skype Window
Spellchecker (very basic)
It also brings the following tools:
Blink strength setup
Blink testing tool
Device connection setup (Com port, baud rate)
Device connection testing
General interface configuration settings
During this prototype stage we decided to focus our attention on two different aspects of the application: adaptability and user interaction. We believe that this type of tool isn’t a “one-size-fits-all” kind of thing, it needs adjusting and tweaking to respond better to it’s user, so we built an interface where all the core-aspects of the application can be configured, making it more responsive without making it overly complex. This matter will be addressed in greater detail later on. The user-interaction is the key to the whole concept. The author of the book “The Diving Bell and the Butterfly" took about two minutes to compose a word using only the blinks on his left eye. Our goal with this tool is for users to be able to compose a whole sentence (a complex one)…under two minutes. This can be achieved through usability, on our first take on the problem we used aphabet-splitting and an autocomplete box, this granted us sentence writing times of about 1.5 to 2 minutes, for two to three word sentences. This time however can be improved if alphabet sorting is applied (e.g ordering it by the most used characters) and the autocomplete mechanism refined. With proper tuning over the application’s controls, the writing experience will be faster and more natural for the user. You can see it in action right here during a full test run. Enjoy! The Goods The current version of BlinkWrite is 0.1 [Proto] and you can get it here The source code will be available soon! How does it work? Our modern days text processing tools offer a whole lot of functionalities and to use them, one must perform several different actions. On BlinkWrite we’ve mapped those actions into states, each being triggered by a specific user action over the application or by the application itself. The trick to achieve that was to identify all the actions and the sequence of those actions on the process of writing and sharing text, basically we’ve set a basic workflow for text-writing. To assist on that interaction process, the interface also features some interesting capabilities, such as control iteration timers, that enable the user to interact with any control on the UI. The process is very simple:
We define which controls are available during each step of the workflow
We set a time frame for the application to iterate over them (such as change control at each 1000 milliseconds)
The user just has to blink to activate the desired control/option
That action triggers the next step of the process
This may become clearer if we look at the application’s UI:
So, on a typical using session, the user will start with the screen on this state:
We have divided the alphabet/application functions into sections, the application starts by iterating over those main sections. If the user wants to type in “Cat”, he starts by selecting the section where the letter ‘C’ is. This triggers a state change and now the application knows that the users wants a letter on that row, so the iteration focus on those five characters alone:
After the desired character is selected, the application automatically places it on the autocomplete box. The user is then taken to the main iteration state, where he must choose the block for the next char. This process must be repeated three times (to have 3 chars). After we have 3 chars on the autocomplete box, a new state is triggered and the user can iterate over the list of words that start by those three letters to choose the desired one. After a word is selected (by blinking) on the autocomplete box, the application places it on the main text area and adds a white space:
It then goes back to the main iteration, where the user can write a new word or use “special functions”, such as “delete previous char”, which is equivalent to a backspace, “undo last word”, which deletes the last inputted word from the main text area, or “finish”, that copies all the text on the text area and opens the “choose destination” screen:
This screen behaves like a popup here the user can see all the open applications at the moment and choose the destination for his text. At the moment only skype is available. When the application is selected, the text is pasted there, on skype’s case, it is also “sent” on the chat window for the other user to see. Now, although skype is the active window on the screen, with one blink from the user, BlinkWrite’s window shows up and the application is ready to go again. This may seem like a long and complicated workflow, but having the process narrowed down like this is in reality a pathway to optimization. Having all these stages working allows us to know where we can automate more, refine and tweak to make the whole thing faster and more natural. In fact we already have some changes for the next version that we will share with you, but more on that later! What about settings? Configuring and using BlinkWrite is very easy! Let’s take a look at the settings screen:
To setup a connection to a device, all you need is set a COM Port, the baud rate for your Mindwave device and click “Connect”. By default the baud rate for the Mindwave mobile is selected (56700). One can also setup the iteration speed for the several UI iterators (autocomplete, alphabet, character), set and test the intentional blink strength threshold, which is based on Neurosky’s blink intensity measurement, and a couple other parameters. The intended use of this screen on the final version of the software is that these settings are set once, to adjust it to the end-user. Even so, the whole process is simple and it shouldn’t take more than a couple of test-runs to complete. A complete user-manual is present under .pdf format on the .zip file. It covers several “How-to’s”, from writing to testing blink detection. What are the requirements for running the software? To run this prototype and maybe a few of it’s future versions there are a couple of specs to take into consideration:
The prototype is only compatible with Neurosky devices. More specifically Mindwave Mobile, if you test it with any other Neurosky headset, please let us know about the results.
A Windows OS is required. We developed/tested it under Windows 7
.Net framework 3.5
Bluetooth (either internal or USB adapter)*
Skype (for copying-pasting-sending text)
Other than this, any ordinary PC/Laptop should be able to run the prototype and some of it’s future versions *This is required to connect the Mindwave Mobile headset, for other models, check your headset’s connectivity requirements Why is it a prototype? We’ve decided to unveil this project at such an early stage because we want to see if it’s something that matters or not. Developing and perfecting this software will be a long and hard path, with many work-hours involved. So before we invest all that time “pro-bono”, we wanted to know if this is actually relevant for the end-users and the community alike. Also, since this is intended to be a opensource project, we figured that it would be best if everyone got in early in the game, not after we have a behemoth of a project takes requires a month’s worth of training just to understand. We consider it to be a prototype, since it lacks many basic, but from a development standpoint, “superficial” features. Things such as punctuation and numeric characters are important for the application’s function, but rather irrelevant during this stage, since implementing them wouldn’t bring anything new to the codebase, just more of the same. We also have some details of the application that are planned but not solved/executed, things such as 1 char and 2 char words for instance aren’t supported on this version, but they will be on the next. And many more items (there is a list further ahead on the article), that will require some attention and that will make a world of difference for the end-user. Up until now our focus was to see if an application such as this was useful and viable. Now that we know it to be viable, adding features will surely make it more useful. What’s next? There is still some work do be done before we can call this anything but a prototype. Our top priority is to get the code base in good enough shape to release it to the community. The main concern is more related to architecture than to code beauty. We feel that the tool will grow easier and quicker if it has solid architectural base. If elements are well organized and defined, it’s easier to expand or build on top of them. So we will be refactoring some of the code in order it can be released on GITHub for everyone to use. We also have a “feature-set” for upcoming versions, we didn’t established a road map or priority chart for them yet, but we will after the refactor. So in terms of features we will have:
Punctuation support
Pronouns (all) support
Numeric characters support
Save As
Save
Send Email on Gmail
Send Tweet
As for improvements/optimizations:
Intelligent autocomplete box: recognize/suggest 2 and 3 character words
Write pattern autocomplete box: order suggestion words based on user’s word usage (improves current autocomplete box)
Select “final action” button: the user will be able to select at any time what activity he intends to perform (e.g write on skype, send Gmail email). Allows greater workflow optimization and removes one step/screen from the workflow
Sort alphabet by character usage: keep the current groups, but organize the chars by their frequency on words
Dynamic alphabet groups: re-organizes the characters list, showing only the next possible chars for the word (e.g on the first char it shows the whole alphabet, on the second char it narrows it down to the second letters of all the words that start by char1, and so on).
Improve the start time for the interface iterators
Implement an "action buider" where users can create new actions for the application (such as "open website x", "scroll down x points on blink", etc.)
And what about more languages? The application is designed to accommodate new alphabets and dictionaries. In fact the whole UI is agnostic about those elements, so it is trivial to load new dictionaries for the autocomplete box and new characters for the keyboard. Further down the road, a “select language” feature may be implemented on the application’s settings and even a “load language” feature to allow users/community to load their own files (they will have to comply with a preset format of course). Want to get involved? At this point we will be doing our best to spread the word about this project, but unfortunately our resources are very limited, so we might not reach everyone. It would be of capital importance to know how relevant is the path we are taking and if we can do anything to make it more relevant. It’s also important to make the dev community aware of this project, so that people can get it, use it and build on top of it. It would be vital for this work to reach it’s intended recipients: the end-users that will benefit from it. But it is also important to reach people that are related and/or have experience/work on the fields of healthcare, palliative care and even research, that have a professional or personal connection with our intended end-users and that above all can help us to determine the relevance of this work and even tailor it to better suit the needs of the persons we are trying to help. Our grasp on that end is very limited, so if you represent a company/entity or know someone that is involved on this tool’s field of application, know any academics or professionals that have a similar goal to ours, or have a friend/relative that can benefit from our work, please drop us a line via email. You can re-tweet or reshare our twitter/facebook announcements or you can grab this article’s link and share it as you like. If you have some skills that can be useful for this project and want to help out, or want to get in touch with the author about this project, please email Rui to rui.guedes[at]wecamefrommars.com Please start your mail’s subject with “BlinkWrite:” so that it won’t get lost in the crowd :) Remember this is a non-profit opensource project, so all work contributions/crediting will be handled either via GITHub (soon) or on the case of theoretical work/articles, posted here on the blog (by direct article submission, re-blogging, etc.).
Your humble minion Rui Guedes. ( Follow Rui on Twitter )
Good luck and godspeed!