INB386

Advanced Multimedia Systems

[jtinb386-blog]

1-Page Blog Summary (Assessment 3)

Blog Summary

User Interface Re-Design

After the feedback of Assessment 2, we decided to focus on substantiating our design decisions before moving forward.  I researched papers by Doug Bowman and came up with two articles that changed the direction of our project.

The first article talks about the 6 stages of design methodology: observe real world performance; model cognitive, perceptual and motor demands to create interaction metaphors; design/develop sensors; implement design systems; and test usability.

I expanded on the idea of a map to include real-world behaviours that could be used as design metaphors.  A Refedex became our new metaphor – search through a list in the index (vertical movement) and flip to a page (horizontal movement).

The second article discusses the three critical tasks of 3DUI navigation: travel; selection; and manipulation.  Our original prototype focussed too much on natural selection: find a link, and then click on it to activate.  Using the Refedex metaphor, we were able to take the natural selection of searching through a book, and apply it the gestural PC/Kinect interface, resulting in a hypernatural experience.  The natural metaphor remains constant, while the gestural input enhances the user’s power with magic techniques (such as target-based travel).

User testing

In one of the tutorials, we conducted user testing by setting up our prototype and getting users to fill out a questionnaire.  The resulting feedback led to our biggest overhaul from GHD2.0 to GHD3.0.  I observed the user interaction in each test and prompted fellow students to fill out questionnaires and express their opinion about GHD.

Feedback from the surveys was mixed; some people appeared to be more favourable than others.  The strongest area appeared to be visuals, while the weakest area was gestures and navigation.  On reflection, a lot of people appeared to be struggling to use GHD, since they were unsure whether or not they were in range of the Kinect, and received minimal feedback from the user interface.  At the time, there were also one or two technical issues which caused GHD to spontaneously jump between pages.

In the week following testing, we changed our UI to address these issues.  Using the magic selection technique of target-based selection, we made it so the user was always targeting an item, so long as their hand was raised to the screen.  We also addressed feedback issues by making menu item selection more responsive; items now pop out and fade colour in/out as you get closer to their y-axis centre.

Content Management

Jordie created a Javascript/HTML template and a list of instructions on how to use them.  I took these assets and generated content using a map of Capalaba Shopping Centre.  This involved inserting written content; creating and inserting icons; tagging locations on the map; and creating menus, submenus and all store pages.

Other Contributions

Video/photo documentation; and group meeting coordination.

[jtinb386-blog]

(Note: can only be accessed with QUT login)

From the early days of 3D UI research, there seemed to be two design approaches for 3D interaction techniques: one that tried to design techniques with as much interaction fidelity as possible, and another that tried to enhance usability and performance through the use of “magic” techniques. Magic techniques might be intentionally less natural, or they might enhance natural interactions to make them more powerful. Below, we discuss both natural and magic techniques for three of the so-called “universal” 3D UI tasks: travel, selection, and manipulation… Travel, the task of moving the viewpoint (or avatar) through the virtual 3D environment, is conceptually very simple, but the design space for travel interaction techniques is surprisingly large…Magic travel techniques exhibit great diversity…  Another set of magic travel techniques called “target-based travel” only require the user to specify a point of interest, after which the viewpoint is moved smoothly to the new location (for example, Hachet et al.14). Target-based techniques reduce mental load for the user, but may not be as flexible or as natural for all applications. Manipulation-based travel is based on hand movements rather than body movements or button presses. For example, the user can “grab the air” with both hands to move through the virtual world as if pulling on an invisible rope. Though fatiguing, these techniques allow physical movement to be mapped to virtual travel without the need for large tracking areas or complex locomotion devices. Selection, the task of picking one or more objects from the environment, is also a fundamental task in many 3D UIs. It has been studied extensively, and the number of different 3D selection techniques is very large… Techniques based on pointing can be used from a distance, while still being considered natural. The canonical pointing technique in 3D UIs is raycasting, in which the user controls the direction of a virtual light ray or laser beam with physical hand movements, and intersects an object with the ray to select it. This technique works well in many situations, but it can be difficult to select very small objects, since small hand rotations can result in large movements of the end of the ray, and only objects that are at least partially visible can be selected. Most magic selection techniques are also based on touching or pointing metaphors, but are “hyper-natural” (that is, they use natural movements but make them more powerful by giving the user new abilities or intelligent guidance)… Manipulation. Closely related to selection is the task of manipulation, in which the user modifies the position, orientation, scale, or shape of virtual objects… Typically, the virtual hand is considered to be rigid, which makes certain rotations very difficult to perform; allowing fingertip manipulation is technically challenging, but makes the technique even more natural… Evaluations of selection and manipulation techniques (for example, Poupyrev et al.27) have generally shown that magic, hyper-natural techniques outperform their more natural counterparts. 3D UIs of this sort can make performing tasks in the virtual world easier than in the real world, which is a strong argument for the magic approach.

*****

This research paper has governed a lot of the decisions we made when re-designing our UI.  We went back to the drawing board and focussed on the abovementioned universal tasks: travel; selection; and manipulation.  Our original design focussed too much on natural selection, where the user selected options by pointing and clicking.  The ‘magical’ and ‘hypernatural’ techniques discussed here proved to be a superior alternative, where interactive elements of the UI give power to the user. In terms of travel, users were struggling to navigate in our original prototype, since there was no on-screen representation of their hand, forcing them to trial-and-error.  We addressed this using target-based travel, where a selection is constantly highlighted, as long as the user has their hand raised. Selection became a lot easier when we implemented these changes; however, we still needed to update and replace UI elements in order to make best use of our new travel solution.  The new side-menu allows selection through vertical hand movement, and manipulation through a swiping horizontal movement.  As mentioned in the research, selecting small objects can be difficult due to the limits of raycasting.  We addressed this by removing text menu items, and replacing them with large symbolic icons. Manipulation is an extension of selection, and is the focus of our new hypernatural techniques.  Whereas before we focussed on natural point-and-click navigation, interaction now takes place through search-and-select using the book metaphor.  Natural techniques of searching through a list and flipping to a page (such as searching a phone book) have been given magical qualities when transferred to Kinect/PC interface.  The result is hypernatural selection – searching and swiping as if you were using a phone book or Refedex, applied to a gestural search function.

[jtinb386-blog]

An over-the-shoulder perspective of our latest iteration - GHD3.0. After some serious code optimisation by Jordie, and a new system which means you don't have to hold your hand to the right to select; GHD3.0 is starting to look polished. Content has been added based on the Capalaba store map.

This video marks the last of our major changes before submission of the final assessment piece, and will most likely be the final video of GHD.

[jtinb386-blog]

Closer look at how we set up Gestural Help Desk:

1. Laptop runs HTML/CSS/JavaScript.

2. VGA cable connects laptop to TV and USB cable connects Kinect to laptop.

3. TV runs GHD at 1920x1200 resolution.

4. Kinect captures hand movement and facilitates input for the user interface.

[jtinb386-blog]

Week 12 Tutorial

This week we had desk critiques.  Jordie and I set up GHD3.0 to show early; Hehato turned up for a brief minute and Brian was there later to help out.  Our demonstration went smoothly, and our tutor appeared to be pleased with the progress we had made since last week.  There were a couple of minor issues which we were advised to address.  Icons animated in slowly, one by one, which was a problem we resolved in five minutes.  The new tweak makes all icons animate on at the same time; an important consideration for anyone who is using this in a hurry.  Another issue was the shopping cart icon.  The meaning behind it was not entirely clear, so we need to either icons with words, or find a more suitable icon to represent “search stores”.  My input was to change the shopping cart icon to a magnifying glass (the common symbol for search). The only other thing to note was that the interface wasn’t quite as smooth as the tests we ran at our last group meeting.  The effect wasn’t too great, however it would be good to see a return of the smooth transitions for our formal critique next week.

[jtinb386-blog]

At the most recent group meeting, Jordie spent the entire session optimising the code for GHD3.0 and making template files.  With these template files and a list of instructions, I got to work filling in the content of the 'website'.  Over the course of a day, I created all of the store pages, filling out icons, written descriptions and tagged locations on the map.  I also tagged locations for parking, ATMs and toilets.  The final thing I did was to create submenus and organise shops into categories (clothing, electronics, food, etc.).

We also came up with a new name for our shopping centre: Capalindale (we have been referring to Carindale for so long, yet since we used the map for Capalaba, we decided to merge the two suburbs into something far superior).

[jtinb386-blog]

Our project has gone through multiple iterations. 3DUI was our original concept and brought the idea of interacting with a 2D interface using gestures in a 3D environment. GHD1.0 was the creation of Gestural Help Desk, with an interface operated by Kinect hand-tracking and mouse/glove combo for the clicking/grabbing gesture. GHD2.0 removed the mouse and operated by moving up/down for a selection, then choosing an option by swiping your hand to the left. GHD3.0 features optimised code for a much more responsive and user-friendly interface, where the user no longer has to wave their hand around in search of feedback.

[jtinb386-blog]

GHD3.0 Developer's Mode.  This shot shows the bare bones of our new UI, and how pages now transition.  The green zone is controlled by your hands; if you move your hand fast enough, it will skip over into the blue zone.  This movement can be activated by a swiping motion to the left.  The sensitivity of the swipe can be adjusted by changing the width of the green (detection) zone.  If the Kinect registers your hand, then it will highlight an option menu for you to select, instead of having to hover over the link.

[jtinb386-blog]

Group Meeting 8

Penultimate week before submission, we held our eighth official group meeting on Tuesday 16th October, between 10am and 4pm.  Brian was unable to attend, Hehato stayed until 1.30 and Jordie did most (if not all) of the work at this meeting. The original plan was to start putting content into the framework of our new UI.  However, we found that the HTML5/CSS/Javascript combo wasn’t working too well; framerates slowed down to a point where the user experience was jarring.  Jordie spent a few hours optimising the code to the point that it was smooth and transitions were seamless.  He also showed us the “developer’s mode” he had created, bypassing the problems we had with our original design.  Instead of having to search for an item by waving your hand around, menu items are always selected (depending on the position of your hand) and are activated by swiping to the left. Jordie’s new design has made vast improvements on previous iterations, and is proving to be a substantial step forward to completion.  The rest of the meeting, Jordie created template pages and instructions so that we could create the ‘website’ for our UI.

[jtinb386-blog]

This recent development brings a mixture of emotions.  On the one hand, it justifies our entire project and is proving to be successful in the real world.  On the other hand, they stole our idea.  Or perhaps they beat us to it.  Either way, the use of gestural input as a hygienic practice (such as dentistry/surgery) was the intent our project.  Perhaps we could still commercialise this with a Western audience now that our design ideas have been put into practice elsewhere.

[jtinb386-blog]

A quick demonstration of GHD2.0; this version shows our new user interface based on the book metaphor.  Selection occurs by moving your hand up/down a list and activation occurs by flicking your hand across. This video also shows some of the difficulties users are experiencing with gestures.  We need to integrate an on-screen representation of the user’s hand and fix the J-Query bug that causes the navigation to spontaneously jump between links.

[jtinb386-blog]

Through user testing and surveys, we deduced a number of issues that need to be fixed over the next week.  The first issue was technical, and we knew about it before testing (we are still working on a fix).  Essentially, links activate prematurely after selecting other links, causing navigation to be jumpy and erratic.  Fixing this should be a simple matter of analysing J-Query for the bug and changing a few lines of code. The second issue was brought up by users during testing and in the surveys.  Users struggled to use gestures since they were unsure of their interaction with the screen.  Our new design does not feature a cursor and the limited range of the Kinect made users have to wave their hand around in search of feedback.  This issue can be resolved by including an icon (possibly hand gestures) to indicate where the user’s hand is. A few other issues were also brought up, including: fatigue; lack of content; and difficulty reading text.  Fatigue from arm movement will be minimised once the other changes are implemented (creating ease of navigation and reducing time/arm movement required).  Lack of content will be addressed over the next week as shop information is added.  Difficulty reading will be eliminated when we recreate the UI on the native 1920x1080 resolution.

[jtinb386-blog]

This was the first survey we put together for testing in Week 11 Tutorial.  Twenty copies were printed out and left beside the prototypes, and people were encouraged to fill them out upon completion of testing.

The survey was not intended for deep user testing; more to gauge first impressions and address major issues.  A new survey will be written for when deep user testing is conducted in the next tutorial.

[jtinb386-blog]

Week 11 Tutorial

This tutorial was a turning point for the project.  We turned up with two separate designs – our original mouse/glove-operated UI, and our new gesture-exclusive UI.  Research and justification was also brought to substantiate our decision to overhaul our UI.  The plan was to get one of the two UI’s approved and build upon the most successful.  During the tutorial, we set up our equipment and prepared both prototypes for testing.  With the encouragement of lollies and surveys for user feedback, we got a dozen of our fellow class members to test each prototype.

The most popular choice appeared to be our new research-based user interface.  There are a few design and technical issues which still need to be resolved (discussed in future posts), but the feedback for GHD2.0 has been positive.  We will continue to build on this design and attempt to complete the final prototype for the next tutorial.

[jtinb386-blog]

We have started to create a new UI based on our “book” metaphor.  This screenshot show the progressive development of our gestural menus.  Using the book metaphor, we search through menus by moving our hand up and down through the list, and selecting an option by swiping the page.  Hovering over a menu item highlights it green and an arrow pops out, prompting the user to move their hand to the left.  We will include a map in the blank space in our next version of the prototype.

[jtinb386-blog]

Another Bowman article justifying the application of 3DUI’s with 2D content follows*:

3D UIs might take into account the user’s head motion, hand motion, and potentially full-body input to provide powerful interaction techniques— not only for dealing with 3D content but also for 2D information spaces. For working with 2D content at the desktop, constraining input and output to planar surfaces appears effective. However, getting up from the office chair and stepping back from the screen doesn’t necessarily imply that the user is no longer interested in the displayed information. The user might want to gain an overview of the data or invite colleagues for an informal presentation, which involves direct pointing from a distance—a truly spatial operation.

The research substantiates physical interaction within the context of 2D information.  Our starting reference was the touchscreen interface from Carindale; this 2D space has provided the foundation around which we have developed our 3DUI to create GHD.  The essence of GHD is to provide (to quote Bowman) “an overview of the data” regarding the shopping centre.

*This document was taken from QUT online library and requires student/staff access.

[jtinb386-blog]

This research article was created in association with leading 3DUI researcher Doug Bowman.  One of the key statements which is driving our new design follows*:

Our research at the Bauhaus-Universität Weimar proceeds in six main steps:

1.       Observe the cognitive, perceptual, and motor performance of humans interacting in the physical world.

2.       Model the cognitive, perceptual, and motor demands of a certain task to create interaction metaphors.

3.       Develop sensors, low-level interfaces, and device drivers to record human actions as input for computer applications.

4.       Design input devices—a combination of sensors assembled ergonomically.

5.       Implement the designed interaction systems in prototype applications to involve users in the development process.

6.       Examine usability and adjust the design.

Because these aspects are interrelated, the whole design process is iterative.

We have re-worked our alternative prototype based on this design model.  Our ‘design metaphor’ is a book.  Information can be searched for in books such a Refedexes and Phone Books.  We apply the page turning metaphor to our model by flicking left/right to move through our ‘pages’.  Searching through lists on a page also rolls over – selection can be made by moving your hand down a list to select which option you want (using the phonebook metaphor, you would slide your finger down a list of numbers to find the one you were after).

*This document was taken from QUT online library and requires student/staff access.