#leapjs

[thingsforrandomaccess]

Sample demo of just the camera rotation and zooming working. (I took this video around a week ago, but due to my phone having problems uploading, I wasn't able to get this up til now).

[thingsforrandomaccess]

Primitive mesh creationg and Improving upon the camera.

This week I added three additional primitive creation  modes (sphere, cube and plane). Switching to primitive creation mode currently isn't detected by the Leap (just using buttons for now). When I figure out a better way to handle switching modes, I'll pull it over to it.

For the time being, I'll be removing panning functionality from the camera controls while I work on creating a less limiting camera rotation function. For pitch and yaw, I'll be using the upward and downward hand movements to independently control each rotation. 

My current implementation only allows for yaw movement due to the vector information I am using to project the mouse movement onto the ball (given horizontal palm swipes). I added in detection of horizontal vs vertical hand movement to differentiate between axis of rotations. Due to small variabilities in hand swipe motions, to detect an entirely discrete motion that is purely horizontal in motion is impossible so I do some approximation to figure out whether a motion reaches a certain threshold to be considered horizontal, vertical, or neither motion. The leap controller can extract pre-calibrated motions such as circular gestures and swipe motions and also the unit directional vector of that swipe. 

[thingsforrandomaccess]

Interesting Article

So this isn't really a progress update but I found an interesting article here about how gestural ui's really aren't that intuitive to users due to their inability to convey to the user visually the capabilities of an application. In other words, it's difficult to create a truly natural user interface using gestures simply because it's hard to figure out how to interact with a computer gesturally without the visual cues. 

Gestural interfaces have been historically more difficult to grasp and learn due to the fact that there really are no set discrete motions to accomplish tasks on the computer. The traditional keyboard and mouse method of interaction is relatively straightforward and provides a set of discrete actions a user can provide to the computer. With a more manageable toolset of input methods and good graphical feedback in the form of menus and dashboards, the user can more easily grasp how to use a new application using just the mouse and keyboard to interact.

This really lent some perspective about the direction I'm going with MotionSculpt. Although I have been focusing on gaining a good understanding of the leap javascript library and Three.js WebGL library, I also have spent a good amount of time this week trying to reevaluate the best way to create an intuitive and natural user interface. This article leads me to believe that I need to insert more visual cues for the user to easily pick up the gestures required to manipulate 3D objects in virtual space. I also need to narrow down my gestural library into a set of easy to learn, and relatively intuitive motions, both discrete and mixed.

So far, all my implementation has been focused on getting the two libraries to work together robustly (which needs more work with the camera... - although the camera controls work to a certain, they really aren't user friendly yet). I am going to change (yet again) how mode switching will happen with the camera. Instead of using finger counting - which is still a little bit shaky due to the tendency of the Leap controller to sense extra fingers - I want to create a simple menu that a user can simply point and tap to select a mode. The method I'm using right now is really too finicky for a user to interact with a scene effectively. I've tried different approaches and parameters to make a smoother transition but it still isn't the best due to the controller's limitations. Additionally, the purpose of the project isn't to create a gestural ui for camera, but also for mesh manipulation. So I don't want to focus too much time on that.

That's it for now. Here's a screenshot of how things look:

Might upload footage of what I've done in action so people can better understand what I'm talking about.

[thingsforrandomaccess]

Camera Mode Integration

For this week, I finished getting the camera controls to work with the leap motion controller through a series of simple gestures. Originally I had planned for the workspace to have a series of modes that the user can cycle through using a circular gesture. With a clockwise/counterclockwise gesture, the user can go through rotation, translation, and zoom modes of the camera and then use their hand position to adjust the camera location.

This approach unfortunately was quickly scrapped due to multiple circular motion gestures being registered at once despite the hand gesture. This makes going from one mode to another highly difficult and frustrating. I needed to approach this with a gesture that is more binary in nature, with less noise in the input.

With this in mind, I decided to use a left hand signaling mode: to change between modes of the camera, the user would simply signal with their left - or right - hand to change the camera into either translation, rotation, or zoom mode. I had the choice of creating a very robust method for detecting left/right hand, but after a bit of testing and research I ultimately decided to go with a simple method: given two hands are detected by the leap motion controller, figure out which hand has the smaller/larger x-coordinate. That will dictate the left/right hand. This only works given that the left and right hands do not flip orientations and that the leap motion controller is in the correct orientation. I chose to use this way over wasting too much time being nit-picky over smaller implementation details - especially at this stage of the project.

This week, I will narrow down my gesture library and figure out what I want to focus the rest of this semester doing with this project (given all the information I have gained so far from implementation and research) and hopefully start gaining some ground with what I decide to do next.

[thingsforrandomaccess]

Weekly Progress Update

Been having fun with the gestures library and implementing things like gesture created drawings, etc. I've managed to get my stuff integrated into the main webgl frame. So it draws gestures on the canvas with the cube. Like this:

It's kinda messy, but I've been working on integrating the progress I've had with gesture sensing with camera controls, which should be done tomorrow night. Not as much progress this week as I would've liked, but it was mainly due to busy-ness with another paper and project that was due this week. I should be able to make up the progress this week. 

[thingsforrandomaccess]

Fall Break Progress

This week has been mostly about trying to get the camera control working with the Threejs framework, as well as figuring out how to get the 3D locations for fingers and hands. I've also calibrated size of the 3D coordinate space of the leap motion device to match up with the size of the canvas on the window. This allows for more intuitive interaction with the screen.

For camera controls. I've gotten the leap motion to recognize a few gestures, which will allow it to change the mode of camera movement - translation, zoom and rotation. I've also refactored the camera mode to switch according to the gesture that the leap motion device has registered. Next, I will implement actual live finger tracking to move the camera instead of mouse click and drags.

In addition, I've tested and wrote up a small demo HTML page that parses and draws the positions of the fingers and hands as sensed by the leap motion device. This little demo file helped me understand how to visualize what the leap motion is capable of sensing (on a 2D scale - since the camera controls are largely still limited to 2D movement within a 3D interface).

Next week I hope to get the mode change working, as well as figure out which gesture will be the best for changing camera mode. Right now all it does is sense circular and swiping motions. I hope to expand on that later. 

[double-are-blog]

My Leap Motion has shipped. Wednesday is fun day. Leap.js (a JavaScript API) is up for grabs so I'll be adding air gestures to "Dashboard" in the next week or so. Leap is... Do almost anything without touching anything. You can only do so much on a computer with clicks and taps. But with the Leap Motion Controller and our apps, nothing’s holding hands and fingers back. So almost anything's possible.