Sniff febrero 15, 2010Posted by christian saucedo in Large scale projections.
Tags: Large scale projections, Large scale projections - Interactive
Tipo. Proyección a gran escala – interactiva
Autor de la pieza. Karolina Sobecka & Jim George
Edificio. 804 East Market Street
Ciudad. Louisville, Kentucky
As you walk down the street you are approached by a dog. He is on his guard trying to discern your intentions. He will follow you and interpret your gestures as friendly or aggressive. He will try to engage you in a relationship and get you to pay attention to him.
Sniff is an interactive projection in a storefront window. As the viewer walks by the projection, her movements and gestures are tracked by a computer vision system. A CG dog dynamically responds to these gestures and changes his behavior based on the state of engagement with the viewer.
Sniff is created with Unity3d Game Engine which renders the dog in real time and allows to dynamically change his behavior based on the video tracking data. The sidewalk is illuminated with infrared lights. An infrared-sensitive camera is used to monitor the sidewalk in front of the display windows. We track the position of each viewer and implement simple gesture recognition, so that fast, big actions are interpreted as threatening and slow actions directed at the dog (for example hand extended in his direction) are interpreted as friendly. The dog keeps track of the attitude of the viewer and forms a relationship with them over time based on the history of interaction.
Sniff, simulating the visceral satisfaction of reality’s responsiveness and dynamism, is also an exploration of engagement of two different planes of understanding, and of relationships created by body’s presence in an environment. The experience is very familiar yet strange, leading us to re-examine notions we take for granted. Dog’s behavior externalizes the process of assessment, evaluation and testing we perform every time anything new enters in the scope of our experience. Sniff has us unwittingly enter into an exchange simply by following the basic instinct of stopping and looking at something that is paying attention to us. A tension is produced by a mixture of fears and expectations, curiosity and interest.
Sniff is supported by a Finishing Funds grant from the Experimental Television Center. The Experimental Television Center’s Finishing Funds program is supported by the Electronic Media and Film Program at the New York State Council on the Arts.
HOW IT WORKS
SNIFF is composed of two main components, a video tracking system and a game engine for real time graphics. The video tracking system is built in openFrameworks; for the game engine we chose is Unity3d.
The low-poly dog was modeled and animated in Blender3d . The model with attached animated actions was imported into Unity3d, without smoothing to retain its polygonal appearance.
The real-time flow of information in SNIFF looks a bit like this:
People on the sidewalk are monitored by an IR camera in openFrameworks. In oF each individual person is isolated and assigned a unique id for the duration of their interaction. Each persons’ position and gesture information is continually sent to Unity3d via OSC networking protocol. In Unity, an artificial intelligence system representing the dog forms relationships with the individuals. He chooses which person to pay attention to, is able to move towards them or back away, responds to their gestures and initiates gestures of his own. Based on the interaction he gets excited or bored, friendly or aggressive, which is reflected in his behavior.
To detect people we are using the standard OpenCV blob extraction packaged with openFrameworks. Beyond the standard blob tracker, we have a ‘PersonTracker’ which assigns persistent blob ids between frames, and dampens the movement of the centroids to reduce jittery head tracking on the dog. We also run custom metrics to filter out false-positive detections, such as the glare from headlights that otherwise would register as people. Through blob detection we are given a good indications of the locations of people, but we are also interested in how they are behaving. During each frame of video we run optical flow to detect motion. Within each person’s bounding rectangle we sum all the motion to determine if that individual is making sudden movements. With this method, we found that walking back and forth triggers a lot of optical motion yet shouldn’t be considered as gesturing. To disfavor this false-positive we dampen the total motion for each person by the movement of their centroid. This favors movements such as waving arms and clapping hands while standing in front of the dog. But if someone is to quickly rush towards the dog it is considered an aggressive gesture and he will react. But don’t think you can just be mean to Sniff, because we also run analysis on the shape of each silhouette to determine if the person is holding out their hand calmly. This is considered a friendly gesture, and Sniff may start to trust them. Based on watching people interact with Sniff we intend to implement recognition of more complex behaviors that are taken from observation of natural tendencies people have when they encounter the dog.
The abstracted information for each individual is sent to Unity where the artificial intelligence and animation engine is contained. Communication between the two applications is handled by OSC messages containing person ids, location and gesture information. The Unity3d system works with an object oriented architecture, containing instances of a dog, people and relationships between them. SNIFF also contains a mood, which is determined by the combination of friendliness and enthusiasm metrics which change over time as people interact with him. If a lot of people are around gesturing at him he’ll go berserk. If he’s been left alone for a while or you aren’t interacting with him, he may just lay down or pay attention to someone else. He has the capacity to form opinions about people over time and will behave according to how he has been treated.
friendly gesture headlight filtered out