Designing Interactive Artifacts

A blog framing my ongoing reflective practicum during the course "Designing Interactive Artifacts" at ITU.

[frederikhojlund]

Closing Thoughts on Learnings From This Course

Following Donald A. Schön’s ideas of the reflective practitioner this course has emphasized the value of sketching and prototyping. Schön describes design practice as a conversation with the materials of the design situation (Schön, 1992). Here the designer performs certain moves and “listens” for the situation’s reaction. From that he adjusts his understanding of the situation and thereby the design, he is creating. This is a process of reflection- and knowing-in-action. When engaging with this conversation prototyping and sketching are very important design moves, which allow the designer to explore, affect and understand the design situation. Inquiry of the design situation can be done through prototyping and sketching. Therefore this is not tools that only belong to the final stages of the process.

This course has given me a better understanding of Schön’s processual concept and most importantly, how it applies to practice. It is not just a matter of making and refining prototypes. Design is about being able to articulate the arguments behind design decisions and understand the results of a prototype put into test. In the context of interaction this requires an understanding of the different aspects, attributes and relations, that constitute interaction. Here I have been introduced to different vocabularies and frameworks for approaching and articulating specific interaction attributes.

Compared to other design courses, that I have followed, this course has focused on the practical aspects of interaction design. I have obtained competencies within three areas: Processual experience, articulation of concepts and technical skills.

References:

Schön, Donald A. (1992): “Designing as reflective conversation with the materials of a design situation.” Research in Engineering Design 3.3: 131-147.

[frederikhojlund]

Our final concept was presented to our class, teachers and CEO of digital design agency Yoke. This is a short video from the presentation.

[frederikhojlund]

The final concept

After testing the different concepts we have decided on one final as our solution to the brief. We have kept the interactional qualities from “Tag” and combined it with the conceptual conflict between reality and computer mediation from “Spotlight”.

Our final concept called “Ghost” extends on the explorative qualities from “Spotlight” by providing options for interaction between visitors. This is done through feedback communicated in the same way, as we did in “Tag”. Furthermore Ghost enhances the conflict between the expected behavior and the actual behavior of the system by showing persons on the screen, that are not present in reality. Instead of having a static image as background like in “Spotlight”, we are now using a pre-recorded video of people walking around at the entrance area.

With the concept “Ghost” we are trying to encourage curiosity by letting users explore through interaction. This interaction holds a conflict between users’ expectation and users’ experience, as it draws a connection between the digital and the real world. Together with the peephole principle for engagement (Dalsgaard & Dindler, 2009) this conflict is what excites users’ curiosity and encourage them to explore the system’s perception and reaction. The idea is, that puzzling visitors provoke reflection on the computer as a medium for representing the real world.

The concept as a solution to the brief Our goal was to demonstrate a novel interactive experience, that provoke visitors to reflect on digital technology as a medium for perceiving reality - focusing on the relation between human and computer.

The field of IT is about representing and enhancing human reality and everyday life through information technology. ITU as a university seeks to explore, understand and develop this field. It is a process driven by curiosity and knowledge generation. Some main focuses within this area are the relation between humans and digital technology, and technology as a medium for representing reality. Our concept invites visitors at ITU to explore this relation between humans and digital technology.

References:

Dalsgaard, Peter & Dindler, Christian (2009): ”Peepholes as Means of Engagement in Interaction Design” in Nordes 2009 - Engaging Artifacts, Oslo.

[frederikhojlund]

Exploring and evaluating concepts

We want to explore different conceptual possibilities before settling with one concept. The technology of motion tracked have proved to be very flexible and suitable for the brief and situation, but we still have to decide on a final concept. We have spend some time converging on the concept of “tag” with one circle following persons nearby, but it is still lacking inquisitive use. Therefore we want to diverge and look into other conceptual ideas. We might be inspired by other ideas and be able to combine them.

To decide which conceptual ideas to pursue, we have evaluated each idea’s interactional properties.

“ALL CIRCLES”

Our initial test sketch, where all tracked persons are marked by a circle. When two circles collide they become one. The main interaction attributes are here fast, instant and stepwise (Lenz et al., 2003). The system reacts instantly as intended and there are no fluent states in-between - the user is either being tracked or not. This makes the correlation between action and function too obvious, which leaves nothing left to be explored.

“VIDEO MASK”

Inspired by the approach of using peepholes as means of engagement (Dalsgaard & Dindler, 2009) this sketch requires social collaboration to reveal hidden information - a video masked between users. This sketch includes an interesting visual aesthetic of layers being controlled by collective motion. But the user engagement is quite passive, because the goal of revealing the video is not challenging at all. Users just spread out their positions to expand the canvas. Furthermore the sketch don’t encourage motion. When testing it we tended to stop moving and watch the video.

“TAG”

The strength of the "tag" concept is the way it encourage high intensity of interaction - also socially. But it is very obvious conceptually and includes no associations to IT - apart from the motion tracking technology.

“SPOTLIGHT”

"Spotlight" is a combination of "tag" and "video mask" in the sense, that it focuses on one user at a time and masks out everything else. A static image of the entrance at ITU is shown in the background with no people. On top of that the person in focus is shown, but other people a hidden, even though they are present i reality. This creates a conflict between, what the user expect to see on the flat screen, and what is actually shown. This conflict encourages inquiry (Dalsgaard, 2008, p. 4) and it provokes reflection. The user has to explore the installation actively to understand it. Conceptually this is very interesting, but the interaction lacks the visibility and apparent feedback from "tag".

References:

Dalsgaard, Peter (2008): "Designing for Inquisitive use" in Designing Interactive Systems '08, ACM, South Africa.

Dalsgaard, Peter & Dindler, Christian (2009): ”Peepholes as Means of Engagement in Interaction Design” in Nordes 2009 - Engaging Artifacts, Oslo.

Lenz, Eva, Diefenbach, Sarah & Hassenzahl, Marc (2003): “Exploring Relationships between Interaction Attributes and Experience” in DPPI 2003, ACM, Newcastle upon Tyne, UK.

Wensveen, S. A. G., Djajadiningrat, J. P. & Overbeeke, C. J. (2004): “Interaction Frogger: A Design Framework to Couple Action and Function through Feedback and Feedforward” in DIS2004, ACM, Massachusetts, USA.

[frederikhojlund]

Using feedback to emphasize personality and make interaction visible

We found the single following circle to be an interesting interaction, that creates a relation among users themselves as well as the system. When only one user is tracked, it creates an unbalanced power structure between users, because the tracked user possesses the spotlight, and others don't.

Also the circle is given a touch of personality as a sticky creature or object. To further emphasize this personality and make the possible interactions clearer, we made the circle look like its mass was attracted by persons nearby. We started by making an initial sketch, where the mouse position represents the location of a tracked person.

This was a coupling between the user's action direction and the circle's direction (Wensveen et al., 2004, p. 178). It act's as augmented feedback showing the user, that the circle responds to his movement. But it also shows functionality by suggesting, what will happen, if the user gets nearer the circle. When the user gets too close to the circle, it will move towards his position and start following him. This is functional feedback, because the ultimate result of the user's actions is the circle following him.

We implemented the augmented feedback into our motion tracking program. The output worked as intended, but was lagging a little.

The feedback made the circle's "intentions" a lot more transparent. The impression of the circle is an object, that is eager to follow everyone and therefore always look for another person to follow.

References:

Wensveen, S. A. G., Djajadiningrat, J. P. & Overbeeke, C. J. (2004): "Interaction Frogger: A Design Framework to Couple Action and Function through Feedback and Feedforward" in DIS2004, ACM, Massachusetts, USA

[frederikhojlund]

Let’s play tag: Catch the marker

When we tested out our initial motion tracking sketch, we found it interesting to play with the behaviour of the marking circles: How well the circles follow movement, and the way circles merge, when to persons get close to each other. But because all persons being tracked have their own circles, the invitation for engaging socially is very limited. We needed to insert more interesting relations between people being tracked.

The user needed a goal to pursue, and the goal could be provided by the system. We decided to explore, what would happen if we only showed one circle at time. We modified our sketch to have one marking circle, that follows one moving person at a time. If another person comes near the circle changes target and follows the new person.

We needed the circle to change target fluently instead of stepwise (Lenz et al., 2003). This was done by implementing physics of velocity and acceleration into the circle object and an algorithm to steer the transition.

This turned out to offer an abstract but clear goal for the user: To either have the circle or avoid it. This way the circle became a spotlight connoting the system's focus. The concept was similar to the classic children's game "tag".

References:

Lenz, Eva, Diefenbach, Sarah & Hassenzahl, Marc (2003): “Exploring Relationships between Interaction Attributes and Experience” in DPPI 2003, ACM, Newcastle upon Tyne, UK.

[frederikhojlund]

Results from testing: Not on the floor - on the wall

After live testing we have realised, that it is practically (almost) impossible to project our tracking output onto the floor in Atrium at ITU. There is simply too much light coming from the glass facades. Instead we are now exploring a traditional flat screen or projection onto a white sheet of fabric. This changes the interactional situation slightly from “direct” to “mapped” (Norman, 1998, p. 23), which invites for different kinds of concepts.

The wall is less direct, but at the other hand it gives the user a better overview of the whole situation. He can more easily assess, what is going on in other places of the motion tracking area.

A traditional screen will have to show the video of people walking around - what the computer sees - along with the overlaid graphics. Otherwise it is hard for the user to figure out, that the graphics respond to his motion.

By showing the captured video of the users themselves the aesthetic of the experience changes. It becomes clearer, how the interaction system perceives and interprets actions. Suddenly the output connotes surveillance and puts emphasis on the feeling of being watched. Surveillance could be an interesting theme for the installation, but it implies negative associations, that don’t match the idea of welcoming visitors. Therefore we needed to find a balance in our concept.

References:

Norman, Donald A. (1998) [1988]: “The Design of Everyday Things”, The MIT Press, London, England.

[frederikhojlund]

Joining circles and social interaction

When coding our initial sketch for motion tracking the output turned out quite interesting. It is not intended, but if two persons passes close to each other, the computer perceives them as one object. When marking objects with circles the result looks like the two circles are joined into one. The "bug" is due to the way the moving objects are tracked. The computer is simply not able to distinguise between the two.

The result presents an opportunity for interaction! It has inspired us to design for social interaction. Our vision to further work from is to create a space, that encourages social interaction among visitors at ITU. Next we are going to explore this interaction live by our self and test, what it feels like.

[frederikhojlund]

Getting started with motion tracking

We have started coding some initial sketches to test, whether we are actual capable of making a motion tracking prototype. We have found an open source coding library called Open Computer Vision (OpenCV) originally developed by Intel. It provides the necessary tools for doing some very interesting analysis of photos and videos. Fortunately it has also been ported to Processing, which is a language, that I know very well.

The technology of making computers analyse and “understand”, what goes on in a photo or video, is called “computer vision”, and motion tracking is a subject within this area. I am very excited about making the computer perceive images in the “same” way as humans do. It works by analysing every frame of the video and comparing the current one to the previous one to detect, what has changed.

I find the whole idea of using an ordinary video camera as input for interaction very intriguing. It creates a lot of possibilities, and interaction wise it is going to be challenging to provide both feedback and especially affordance in this medium. Our idea is to track people as they enter Atrium with a camera placed at the second floor pointing down at them. As output we want a projector to project grahics and markings on to the floor.

We have recorded a video for testing using a smartphone, and we have aldready succeded in tracking moving objects. We have even managed to detect if it is the same. Next we are going to explore the different possibilities, because we do not have concept yet. We are planning to test, what it feels like being tracked and having a circle following you. We will try to develope our concept along the way.

By using motion tracking and projection onto the real world we are entering the field of mixed realities. On Milgram et al.s reality-virtuality continuum we are dealing with some kind of augmented reality (Milgram et al., 1994). It is going to correspond to Lev Manovich's "augmented space", where a virtual layer of information augments real space (Manovich, 2006). But we still have to decide, what should make up this information layer

References:

Manovich, Lev (2006): ”The Poetics of Augmented Space” in Visual Communication, vol. 5, no. 2, SAGE Publications, Californien, USA, pp. 219-240.

Milligram, Paul, Haruo Takemura, Akira Utsumi, & Fumio Kishino (1994): ”Augmented Reality: A class of displays on the reality-virtuality continuum” in SPIE - Telemanipulator and Telepresence Technologies, vol. 2351, International Society for Optics and Photonics, pp. 282-29

[frederikhojlund]

Fieldwork and exploring opportunities for interaction

In my group we started by exploring ITU in the setting of visiting the place. We walked around the building and looked for interesting spots and situations. Also we tried to imagine and remember, what it felt like to visit ITU for the first time: Where do you go? What do you notice and think? What is your first impression?

Deciding on situation We quickly concentrated our around Atrium, because this is the place, where all visitors enter into ITU and their first impression takes place. Atrium is very modern and architecturally impressive, but we saw an opportunity to emphasise the IT aspect in this impression. Therefore we decided to focus on the situation of entering Atrium. We used the situation as a starting point to pick an appropriate form of interaction and technology.

Deciding on technology In the situation of entering Atrium some visitors might be willing to actively interact and engage, but others are just passing through immersed in something else. Therefore our concept needed to invite for interaction, but at the same time offer the option not to do so. We brainstormed for technologies and looked for inspiration online. In the end we decided to explore motion tracking as our technology for interaction input, because it suits well in the situation of entering Atrium. Visitors are tracked and interact with the system even though they don’t intend to. This way the system automatically attracts visitors attention and introduce them to the artifact or installation. But it can be up to the individual user, whether he will interact passively or actively. Furthermore motion tracking can take advantage of the actions (people walking) already taking place in Atrium. This way the system don’t require special effort from the visitor. None of us have worked with motion tracking before, but from our initial research it seems to be doable.

This way we did not approach the brief by thinking out a concept from the very beginning. Instead we decided on constrains about a concrete situation to design for and a specific technology to use. Setting up these constrains narrows the design space and makes the design situation more concrete and less difficult to navigate. It provides a more bounded situation for beginning a conversation with its materials (Schön, 1992).

References:

Schön, Donald A. (1992): “Designing as reflective conversation with the materials of a design situation”, Research in Engineering Design.

[frederikhojlund]

Final project and brief

Now we are entering the final project in this course. Here we are allowed more freedom to explore different technologies and forms of interaction. This post includes my initial thoughts on the final brief, which is phrased as follows:

Design an interactive artifact, that lets ITU visitors experience, that they are visiting an IT University.

This brief is quite open and broad, which leaves it up to us, to decide which technology, interaction and concept is best for solving the task. The brief consists of three parts:

The artifact and interaction has to take place at ITU.

The target user is visitors at ITU, which means people, who both do visit on a daily basis and do not visit daily.

The experience of the interaction should demonstrate or represent the area of information technology in an university setting.

This way we have a setting for interaction and a target user. But we have to define a more concrete situation and context ourself. The experience of IT can be approached in several ways. One way is to demonstrate IT-technology through interactions, that are based very much on novel and unusual technology. Other ways can be to let the user explore, what IT is about in a university setting, or inviting the user to reflect on IT-related subjects.

[frederikhojlund]

This is the final result of our solution to the second brief. Here the different light patterns are built into the plant.

[frederikhojlund]

Expressing personality using blinking LEDs

Inspired by the study of light patterns by Harrison et al. (Harrison et al., 2012) we try to find the right combinations of blinking patterns, brightness and color to express different reactions to user action. The ideal is, that it should be clear to the user, how the lamp “feels” about the way, he interact with it.

When the user gets nearby the lamp Using a proximity sensor we detect if the user gets nearby. Following the personality of a dog, the lamp should get excited, when this happens, and it should search for attention. We want to give the expression, that the wakes up and begins to “wag its tail”. We do this by slowly fading on the light and then pulsating it rapidly.

If the user strokes the lamp When the lamp gets attention from the user stroking its leaves, the light pulsates even faster between purple and white. This is detected by a tilt sensor.

If the user ignores the lamp When the user has been nearby the lamp for some time without stroking it, we want to express disappointment and resignation. This is done by suddenly fading the color to bright red and after a short period fading it slowly out.

References:

Harrison, Chris, Horstman, John, Hsieh Gary & Scott E. Hudson (2012): ”Unlocking the Expressivity of Point Lights” in CHI’2012, ACM, Texas, USA.

[frederikhojlund]

Complex Expression with Blinking LEDs

The video above shows LED-light, that is a lot more expressive than the one in our solution to the first brief.

When working on the first brief we experienced, that the output, reaction and expression of the lamp’s light (LEDs) was crucial for the user to understand the personality of the artifact and for obtaining an appropriate mental model about the interaction. Therefore we need to work on the expression of the LEDs during the second brief. We need to be able to totally control the LEDs’ blinking pattern, colors and fading.

A common way of controlling blinking in Arduino is to use the delay() function, but there is a huge problem. Delay() does not just delay the blinking of a LED, it simply delays the whole execution of the program - it stops all processes temporarily. This means, that we cannot check for sensor inputs while blinking, which results in unintended delays. Feedback like responding to a user pressing a button might be delayed so much, that it might not be possible to receive a response at all. We have managed to find a solution inspired by Mogens Jacobsen’s Javascript example from the One Pixel Project. We have also found a library for fading LEDs. All this makes it possible to control complex expressions of LEDs in Arduino. But it has taken a lot of effort.

[frederikhojlund]

Second Brief: Physical Object and Context

The second brief within the physical project is as follows:

You must integrate your light into a physical object and create a physical interactive artefact that expresses (and encourages) one of these three values through the interaction: - Love - Hate - Desire

This brief introduces the shape of the artifact into the interactional attributes. Interaction with the artifact is no longer “just” about the relation between different actions and functions (Wensveen et al., 2004), but also about the physical shape, color and feeling of the artifact. This makes it possible to incorporate i.a. affordances, constrains (feedforward) and a metaphorical context into the design of the interactive artifact.

Our solution to the first brief actually matches this second brief. We could simply build our sleeping LED-concept into a doll or teddybear, and it would express the value of “love”. But of course we want to challenge ourself and further explore the interaction possibilities in this new brief. We are still aiming for the theme of “love”, but instead of a teddybear we want to make an interactive lamp. Our initial idea is a lamp, that responds to the user caring for it and showing it love.

Choosing the right object

We needed an object or shape, that invited the user to care for it. First we explored different shapes for a traditional lamp, but we soon realised, that an untraditional shape might be more appropriate. We ended up with the idea of a potted plant, where light shines out through its leaves. A house plant is organic and alive. It also depends on someone taking care of it by giving it water and light. You are used to taking care of a lamp - but of course you are not used to a plant functioning as a lamp.

Finding the right personality analogy

We wanted the lamp/plant to express a profound personality, that asks for care. At first we thought a fireplace would be a natural analogy, because it needs someone to pile wood on the fire. But this personality was too simple - a fireplace has no real personality. Instead we ended up with the personality of a dog. This way the lamp would be loyal to the user if he took care for and loved it. This could be a relationship, that changes and builds up over time. The lamp could become attached to one or several persons.

References:

Wensveen, S. A. G., Djajadiningrat, J. P. & Overbeeke, C. J. (2004): “Interaction Frogger: A Design Framework to Couple Action and Function through Feedback and Feedforward” in DIS2004, ACM, Massachusetts, USA

[frederikhojlund]

Communicating personality and detecting human helpfulness

The first brief proved to be quite a challenge in two ways: First of all it is hard to communicate a personality using one LED only. Of course the personality don’t reside in the LED’s blinks and color only, but also in the interactive relation to the user - in the way it reacts to the user’s actions. But still it is hard to afford or invite the user to be helpfull without having any shape or context for the artifact. Secondly we need to detect the user being helpful or showing care for the LED, which is very much a social gesture. Using the binary logic of a computer doesn’t fit easily with such abstract gestures. This is our primary code for detection:

bool sleeping = false;

bool pickedUp = false; int upSwing = 0; int downSwing = 0; int maxSwing = 5;

void checkPickup() {  if (!pickedUp && x > 250 && x < 290 && !sleeping) {    pickedUp = true;    Serial.println("Picked Up!");  } else if (sleeping && x > 315 && x < 345) {    pickedUp = false;    Serial.println("Put down!");    upSwing = 0;    downSwing = 0;

   delay(5000);    sleeping = false;  } }

void updateSwingers() {  if (pickedUp && upSwing < maxSwing) {    if (y > 350 && y < 370 && upSwing <= downSwing) {      upSwing++;      delay(1000);      Serial.print("Upswing: ");      Serial.println(upSwing);    } else if (y > 300 && y < 315 && downSwing <= upSwing) {      downSwing++;      Serial.print("Downswing: ");      Serial.println(downSwing);    }  } else if (upSwing >= maxSwing) {    sleeping = true;    Serial.println("Sleeping!");    mood = "calm";  } else if (!sleeping) {      Serial.println("ARRRR!!");      mood = "angry";   } }

We check whether the LED (and breadboard) is picked up in a certain way and afterwards check for how many times, the user have lulled (tilted) the LED up and down. Then the LED goes to sleep - blinks slowly. After some time it wakes up yet again. This simple and strict logic demonstrates the complexity of dealing with human and social behaviour. We are only able to detect very mechanical and consistent actions. Therefore the helpful interaction becomes very unnatural and mechanical.

[frederikhojlund]

First brief: Designing personality and value-laden action

We have been introduced to the brief within the physical interaction project:

Try to give your light a personality in its dynamic form.

Try to design an interaction with your light, that invites the “user” to act according to a human value.

We are constrained to LED light as output. So similar to the one pixel project we have brightness, color and blinking pattern as ways of reacting to the user’s actions.

For this brief my partner, Anton, and I want to explore a physical way of interacting with an artifact. This means no buttons or ordinary switches. We want the interaction to be gentle and be about caring for the artifact. So according to Shalom H. Schwartz’ human values, which we were introduced to today, we are aiming for benevolence and especially helpfulness and responsibility. We have used Hans Eysenck’s personality traits as a starting point for picking a personality for our blinking light. To correspond to the value of helpfulness the blinking light should express a need or demand for help. So we are aiming for a choleric, restless and changeable personality trait. We are inspired by the sleeping pulsating pattern on Mac computers, and our idea is a light that sleeps (uses the same sequence), but suddenly wakes up and starts blinking rapidly. The user must lull the light back to sleep like a baby.

To realise this we are working with an accelerometer to detect if the user lulls the light.