random things i need random access to.

[thingsforrandomaccess]

I’ve managed to come up with a mode picking/switching method.

The video quality isn’t too good, so here’s a screenshot to see exactly what’s going on when the mode menu is activated:

Essentially, the user can activate mode picking by clenching their hand into a fist, bringing up dial with four different modes the user can select from: Rotate, Zoom, Sculpt, and Select. I’m still working on the sculpting functionality, which isn’t yet ready to be shown as a video. To pick a mode on the dial, the user keeps their hand clenched and moves their fist in the direction of the subsector of the dial that they want to change modes to. A line will extend from the center of fist when the modemenu was called up out to the sectors. To select the mode, the user must unclench their hand. The pointer reappears in the color of the selected mode (red for rotate, green for sculpt, yellow for zoom, or purple for selection/null mode. The colors help differentiate the mode currently. 

Besides this, I’ve also been playing around with materials in WebGL. (This is partially why the cube in the video is very oddly colored with hard to differentiate faces. I would like to achieve a nice glossy white floor (which at the moment is black and hence invisible because I am still playing around with reflections in Three.js).

Oh, and I've fixed the problem with the mesh wireframe display. It turns on with pressing the "t" key and returns to normal with pressing the "f" key. Here's the transparent view:

Before the demo on Thursday, I will be meeting one or two of my test subjects again to see how my developments are received by them. 

[thingsforrandomaccess]

Sample demo of just the camera rotation and zooming working. (I took this video around a week ago, but due to my phone having problems uploading, I wasn't able to get this up til now).

[thingsforrandomaccess]

So I've spent the last hour or so trying to figure out how to get transparency to work for my wireframe. Finally figured out it was a bug with the canvas renderer. Apparently with WebGL renderers you can render the obscured edges too. The solution is too hack-y if I want to be able to interact with multiple primitives, so I'm still considering whether or not to proceed with transparency.

[thingsforrandomaccess]

Updates

This week I was able to debug and figure out what was wrong with my camera rotation. Apparently, it was a mix of two problems, one of which was pretty silly and involved some bad mix-up with my three.js vector objects and array-represented vectors for my palm position coordinates and mouse projection calculation. After fixing this and adjusting some rotation-speed coefficients, camera is now working very smoothly. The arcball rotations work pretty well without a need to differentiate rotation axes.

Additionally, I've also reconvened with my little test subject and it seems intuitive enough to not have to differentiate axes. I wasn't really able to get him to understand axes rotation very well. His reactions seemed delayed and he had to think a lot more about how he wanted to place the cube when given three axes of rotation. Hence I've settled on this current implementation.

I'm also currently working on developing some deformation capabilities for the cube. When I questioned him a month back, I realized that my test child treats 3D objects like they are somewhat moldable (i.e. clay). When asked to interact gesturally as if he were flattening out an object, he tends to make back-and-forth rolling motions with the flat palm of his hands, as if flattening out the object with a rolling pin. This and additional motions demonstrated that the child senses the 3D object as having physical properties such as malleability and elasticity. Another interesting gesture he used was to ball up the imaginary 3D cube with his hands to smooth it out. Like this:

I'm still in the process of implementing these functions into the application.

Also added a "ground" in 3D space since many people get confused by the rotation and zoom - they tend to think these are transformations happening to the object rather than the eye/camera of the scene. I also added a trigger transparency button since it's a lot more helpful to modifying the mesh when you can see the full geometry of the object.

Since the end of the semester is fast approaching, I would also like to touch upon what I would like to do after this semester is over to continue the project (since at this rate I don't see myself being able to complete everything I've set out to do and I've also come up with more ideas for the future during this semester):

- Bring in physically-based deformations to objects such as denting and bunching up.

- Interacting with multiple objects in the space - not just one.

-  Loading in obj meshes and saving them for later.

- Effective object picking.

- More research for a good balance between using gestures and the mouse/keyboard.

[thingsforrandomaccess]

Updates

I've implemented two ideas for exiting out of camera modes this week. One is by swiping underneath the controlling hand to interrupt and exit the current mode, and the other is by simply closing the controlling hand into a fist. After some input from other people, I decided to use the fist-making method to interrupt camera modes.

I am still struggling to get rotation to work correctly. My current implementation is through the usage of the TrackBallControls base code from Threejs. This is a quaternion arcball based camera api was primarily used for camera controls dictated by the user's mouse or touch. 

Because of the variability in the movement of the hand and fine-ness of motion detection from the hardware, integrating hand movements with mouse controlled rotation results in much more stuttering movement in the camera controls. Additionally, for some reason that I am still trying to figure out, after rotation, the object tends to start zooming in/out on its own even without user input. Even by completely disabling zooming capabilities, this phenomenon still occurs. I am still trying to debug this problem by looking more carefully into my camera rotation implementation.

I've also starting looking into ways to implement basic shape editing, such as stretching and pulling out faces in polygons through matching the normals. I'd like to get the rotation fully working before working on this though. 

[thingsforrandomaccess]

Primitive mesh creationg and Improving upon the camera.

This week I added three additional primitive creation  modes (sphere, cube and plane). Switching to primitive creation mode currently isn't detected by the Leap (just using buttons for now). When I figure out a better way to handle switching modes, I'll pull it over to it.

For the time being, I'll be removing panning functionality from the camera controls while I work on creating a less limiting camera rotation function. For pitch and yaw, I'll be using the upward and downward hand movements to independently control each rotation. 

My current implementation only allows for yaw movement due to the vector information I am using to project the mouse movement onto the ball (given horizontal palm swipes). I added in detection of horizontal vs vertical hand movement to differentiate between axis of rotations. Due to small variabilities in hand swipe motions, to detect an entirely discrete motion that is purely horizontal in motion is impossible so I do some approximation to figure out whether a motion reaches a certain threshold to be considered horizontal, vertical, or neither motion. The leap controller can extract pre-calibrated motions such as circular gestures and swipe motions and also the unit directional vector of that swipe. 

[thingsforrandomaccess]

This is late I know....

So this week, I've fixed up some bugs regarding the camera movement. The rotation still needs some work, but I've implemented color change and pointer finger tracking.

The reasoning behind drawing a pointer to track the position of the palm/pointer finger is that I find that the user cannot get a good grasp of the space that the Leap Motion Controller senses without good finger tracking. Hence when in camera mode (two hands) the pointer is tracking the palm. The pointer changes color depending on the mode that its currently in. More visual cues in the application really helped with the user being able to determine what the Leap Motion is sensing and what it isn't.

I've also started looking into good ways to approach sculpting with WebGL. Ultimately, I want to be able to achieve something like this (mainly the sculpting components). This means I'll need to use a sphere or a more complex mesh to work with. I'll look into this this week.

[thingsforrandomaccess]

Interesting Article

So this isn't really a progress update but I found an interesting article here about how gestural ui's really aren't that intuitive to users due to their inability to convey to the user visually the capabilities of an application. In other words, it's difficult to create a truly natural user interface using gestures simply because it's hard to figure out how to interact with a computer gesturally without the visual cues. 

Gestural interfaces have been historically more difficult to grasp and learn due to the fact that there really are no set discrete motions to accomplish tasks on the computer. The traditional keyboard and mouse method of interaction is relatively straightforward and provides a set of discrete actions a user can provide to the computer. With a more manageable toolset of input methods and good graphical feedback in the form of menus and dashboards, the user can more easily grasp how to use a new application using just the mouse and keyboard to interact.

This really lent some perspective about the direction I'm going with MotionSculpt. Although I have been focusing on gaining a good understanding of the leap javascript library and Three.js WebGL library, I also have spent a good amount of time this week trying to reevaluate the best way to create an intuitive and natural user interface. This article leads me to believe that I need to insert more visual cues for the user to easily pick up the gestures required to manipulate 3D objects in virtual space. I also need to narrow down my gestural library into a set of easy to learn, and relatively intuitive motions, both discrete and mixed.

So far, all my implementation has been focused on getting the two libraries to work together robustly (which needs more work with the camera... - although the camera controls work to a certain, they really aren't user friendly yet). I am going to change (yet again) how mode switching will happen with the camera. Instead of using finger counting - which is still a little bit shaky due to the tendency of the Leap controller to sense extra fingers - I want to create a simple menu that a user can simply point and tap to select a mode. The method I'm using right now is really too finicky for a user to interact with a scene effectively. I've tried different approaches and parameters to make a smoother transition but it still isn't the best due to the controller's limitations. Additionally, the purpose of the project isn't to create a gestural ui for camera, but also for mesh manipulation. So I don't want to focus too much time on that.

That's it for now. Here's a screenshot of how things look:

Might upload footage of what I've done in action so people can better understand what I'm talking about.

[thingsforrandomaccess]

Camera Mode Integration

For this week, I finished getting the camera controls to work with the leap motion controller through a series of simple gestures. Originally I had planned for the workspace to have a series of modes that the user can cycle through using a circular gesture. With a clockwise/counterclockwise gesture, the user can go through rotation, translation, and zoom modes of the camera and then use their hand position to adjust the camera location.

This approach unfortunately was quickly scrapped due to multiple circular motion gestures being registered at once despite the hand gesture. This makes going from one mode to another highly difficult and frustrating. I needed to approach this with a gesture that is more binary in nature, with less noise in the input.

With this in mind, I decided to use a left hand signaling mode: to change between modes of the camera, the user would simply signal with their left - or right - hand to change the camera into either translation, rotation, or zoom mode. I had the choice of creating a very robust method for detecting left/right hand, but after a bit of testing and research I ultimately decided to go with a simple method: given two hands are detected by the leap motion controller, figure out which hand has the smaller/larger x-coordinate. That will dictate the left/right hand. This only works given that the left and right hands do not flip orientations and that the leap motion controller is in the correct orientation. I chose to use this way over wasting too much time being nit-picky over smaller implementation details - especially at this stage of the project.

This week, I will narrow down my gesture library and figure out what I want to focus the rest of this semester doing with this project (given all the information I have gained so far from implementation and research) and hopefully start gaining some ground with what I decide to do next.

[thingsforrandomaccess]

Weekly Progress Update

Been having fun with the gestures library and implementing things like gesture created drawings, etc. I've managed to get my stuff integrated into the main webgl frame. So it draws gestures on the canvas with the cube. Like this:

It's kinda messy, but I've been working on integrating the progress I've had with gesture sensing with camera controls, which should be done tomorrow night. Not as much progress this week as I would've liked, but it was mainly due to busy-ness with another paper and project that was due this week. I should be able to make up the progress this week. 

[thingsforrandomaccess]

Fall Break Progress

This week has been mostly about trying to get the camera control working with the Threejs framework, as well as figuring out how to get the 3D locations for fingers and hands. I've also calibrated size of the 3D coordinate space of the leap motion device to match up with the size of the canvas on the window. This allows for more intuitive interaction with the screen.

For camera controls. I've gotten the leap motion to recognize a few gestures, which will allow it to change the mode of camera movement - translation, zoom and rotation. I've also refactored the camera mode to switch according to the gesture that the leap motion device has registered. Next, I will implement actual live finger tracking to move the camera instead of mouse click and drags.

In addition, I've tested and wrote up a small demo HTML page that parses and draws the positions of the fingers and hands as sensed by the leap motion device. This little demo file helped me understand how to visualize what the leap motion is capable of sensing (on a 2D scale - since the camera controls are largely still limited to 2D movement within a 3D interface).

Next week I hope to get the mode change working, as well as figure out which gesture will be the best for changing camera mode. Right now all it does is sense circular and swiping motions. I hope to expand on that later. 

[thingsforrandomaccess]

Progress for the Alpha Review

This week I was able to hook up the leap motion controller to my threejs framework and also made changes to the framework itself(mainly a lot of re-architecting the code since some of the choices I made in implementing mouse controls weren't that great, in retrospect).

I wasn't really able to implement new functionality for the Threejs framework that I've built but I did modify the camera movement (throught mouse dragging) to use quaternions instead. This makes the movement of the primitive a lot smoother and less buggy when doing camera rotations.

For the leap motion controller, I've been using the leap motion javascript library to see what kind of data the leap device can give me. from that, I plan on attempting to implement camera movement with the leap device. 

I have some good gestures pertaining to camera movement that I hopefully can start implementing in the next few days.

The look hasn't really changed much from last time, but here's a screenshot of how it looks: (I can't really show the leap motion parts without recording video, will probably attempt to do that next week if i get the gesture detection working).

[thingsforrandomaccess]

Progress...

This week I've continued with working with Three.js to create a good 3D environment for the MotionSculpt application. I've gotten basic camera controls through mouse dragging (which includes zoom, translation and rotation motion). 

In the next week, I will be working on bringing mouse controls to dynamic primitive creation (for now, just a sphere, cube and plane will do).

Here is a screenshot of how it looks:

In addition, I've begun playing around with the Leap Motion controller and SDK. I haven't made as much progress as I'd  have liked to, as I spent most of my time implementing mouse controls and designing the main Three.js powered part of the application. I should be able to make more progress next week.

In regards to the gestural interface research, I've already talked to a few kids (and peers) and gathered a good amount of information for designing a clear, intuitive gesture-based user interface. Depending on the performance of the Leap Motion controller in detecting certain kinds of motions (which I plan on figuring out next week), I will have a good basic library of gestures for the functionality I've implemented above (Camera and primitive control) by next week's update. 

[thingsforrandomaccess]

Exploring Three.js

The majority of this week I've been looking into the Three.js library. After playing around with some code and reading some documentation, I managed to create a simple primitive maker. The user input so far is just a few rudimentary sliders in a controller bar.

In addition, I've also done a few experiments with continuous rendering with the requestAnimationFrame function. This particular function prompts the renderer to redraw the scene over and over. The advantage it has over a setTimeout() javascript call is that this function is hardware supported, meaning that if the user minimizes the window or something similar, the app is not continuously consuming the CPU. Through the creation of this sample demo file I've been able to gain a lot of practical knowledge about what I can do with the Three.js library as well as discovered a good hardware supported method of rendering in real-time.

I've also found quite a few interesting examples online of what people have been able to do with three.js. 

This particular example online i found of a glass molding simulation is especially interesting due to its use in mesh deformations - something that I want to use to use for mesh sculpting.

Another interesting example that demo's the capability of the leap motion controller for web painting applications.

I will be beginning actual experimentation with the leap motion device sometime next week as well as beginning development of a good gestural user interface. 

Til next week!

[thingsforrandomaccess]

Hello, World!

This post will be the first of several posts documenting my progress as I work on my senior DMD design project, which is called MotionSculpt (as stated quite plainly above). 

In a nutshell, MotionSculpt is going to be a web-centric 3D design tool with a gesture based user interface. My 3D hand motion input will be given through the Leap Motion Controller, which has an SDK for app development. I hope to use this as well as WebGL to create this application.

In terms of my progress so far, I've just submitted the initial project proposal and design document this past Friday, so now I'm getting around to doing some more research reading about the Leap Motion SDK and also trying to get warmed up with WebGL (since I've never used it before). I am considering using a WebGL library called three.js which simplifies and abstracts a lot of the complicated work with just WebGL. 

So yea, that's about it for now. I will probably start conducting gestural user experience and interface research with various other people within the next week. Some main features I would like to have interactive gestures for include selecting primitive creation and having interactive custom shape creation, as well as camera/viewport controls (rotational vs. translational).

This might be a little vague right now, but I'll clear it up when I make more progress on it as the weeks progress.