Skip navigation

Category Archives: Assignment 2

My documentation


Paul Touches Food.

Our project which began life as a pipe-organ with animated monsters slowly transformed into an exploration of people’s choices when presented with food. In many ways it was incomplete and unfulfilling. This should likely be attributed to our overall lack of vision with the project, shifting many times before coming up with our presentation, much of which was untested. The visual display was useless, the input worked but was too long, and didn’t pull in users, and the “art” was absent.

However even with these failures I think there are a few important take-aways

  • We were concerned about wear and tear on our input kiosk, but instead (almost) no one touched it. Interaction, start-up, needs to be easier.
  • Our testing needs to be more iterative, we can’t finish everything at once, and we didn’t have good waypoints in our plan
  • We need to pick an idea, have a vision, and stick to it. This just didn’t happen
  • Art + ML + the Children’s Museum + three weeks, is possibly just too many constraints.

We think there is more too, but as we plan our final assignment for the class, hopefully this experience will provide us a far more solid foundation.

Our documentation.


This installation will create the experience of interacting with a flock of friendly butterflies. As the visitor approaches the installation, they see a woodsy scene with butterflies flying around. The visitor’s shadow appears cast on the projection. The butterflies will react to the shadow, and try to approach.

These butterflies are “alive” and they like people, they grow healthier with human contact, but can be harmed by sudden movement. The butterflies will learn overtime to interact with the visitors to have the most human contact. Overtime, butterflies may fade away.

We’re interested in what kind of behavior the butterflies will learn, what patterns if any may emerge, and if the behavior will change overtime depending on the behavior of the visitors.


–        Word rain


–        existing set-up of black and white camera and projectorbutterfly-fly21

–        computer ( we will bring this)


–        The installation can detect the presence of a visitor

–        Each fly learns how best to behave near a shadow

Machine Learning:

–        each butterfly will learn through the course of its life to optimize the amount of human touches that it can get.


–        butterflies that individually learn to stay close to humans.

Questions to explore:

–        Is there a reason to pick random flight?

–        Include speed in actions

–        Include more states: distance to shadow (allow more long-range decisions)

–        How quickly will butterflies learn?

–        Allow system to keep working when no-one is around (so not all butterflies die out)

–        Showing that a butterfly has “grown” or is healthier

–        Will there be significant change in their learning from interacting with new visitors?

–        Will new butterflies be created that maintain the learning in the system? Will new butterflies be created naïve to show the learning behavior again?

Ideas on hold:

–        Healthy butterflies would be able to mate and produce children that share a combination of their learned behaviors that made their parents fit

–        More active interference with the butterflies’ health using specific gestures like a cupped hand to “feed” and a fist to “net” the butterflies.

This applet responds to the left and right click of the mouse. Small circles are created in response, one kind is more likely to stay within the big grey circle , another is more likely to leave the circle.  The big grey circle increases with the creation of “out” small circles, and decreases with the creation of “in” small circles.

A set of gloves was made as an interface. When the wearer puts their hands together like in a praying position it creates a left click, and if they put their hands in a praying position but closing their fingers over the other hand it creates a right click.

//this sketch is made of a model that holds particles whose only characteristic is a position and speed
//there are two views one that displays the particles as squares and one that displays the particles as circles

//make a dynamic sized array (arraylist) to hold all particles
ParticleCloudModel partCloud;
//make a view cloud
ParticleCloudView partCloudV;
//declare circle
CircleModel circler;
//color for bg
color bgc;
//color for particles
color pc;

void setup(){
size (600,600);
//start circle
circler= new CircleModel(width/2, height/2, 50);


//start the particle cloud
partCloud = new ParticleCloudModel();

//make a view
pc = color (252, 152,0);
SquarePartView spview = new SquarePartView (pc, 10, 10);

//initialize the particle cloud views with the view set up
partCloudV = new ParticleCloudView(spview);

bgc = color(225);

void draw(){
//draw current circle
//update the particle cloud, this is going to circle thru the whole array, delete any particles that left the screen, and update each particle that is still there
//this applies the set up view to all particles in the particle cloud


class CircleModel{
float r;
float cx;
float cy;
CircleModel (float tempcx, float tempcy, float tempr){
void drawCircle(){
ellipse (cx, cy, r*2, r*2);
//checking whether something is outside circle
//returns true if particle is inside or outside radius and should continue moving
//returns false if particle is about to hit boundary and should change course

boolean goingToHit(float tempA, float tempB, float tempDa, float tempDb) {
//receive x, y and dX and dY
float a= tempA;
float b= tempB;
float dA = tempDa;
float dB = tempDb;
//remains false unless particle is about to hit boundary
boolean state = false;

//is about to hit boundary: present position is inside R, next one is out
if ( sqrt(sq(a-cx)+ sq(b-cy))<= r && sqrt(sq(a+dA-cx)+ sq(b+dB-cy))>=r ){
state = true;}

//return final state
return state;


//particle model that has only x, y coordinates, x and y speed and the method to update the x, y
class ParticleModel{
//data for the particle   //we’ll call velocity “d” for change
float x, y, dX, dY;
//this parameter is whether the particle leaves or not
float kind;
//constructor for ParticleModel: makes a particle
ParticleModel (float tempX, float tempY, float tempdX, float tempdY, float tempK){
kind = tempK;

//this method will update (when called) the x and y values by the ds
//method will check whether particle is about to hit the boundary – the circle’s radius
//particles will move in same direction is not
//particles will bounce against boundary at different likelihoods depending on their kind
void update(){
//changing direction: if (mousePressed && mouseButton==LEFT){dX=-dX;dY=-dY;}

if (circler.goingToHit(x, y, dX, dY)){
//if this condition is true then the particle is about to hit the boundary,

float s = random(0,4);
if (kind==1 && s<3) {

//if this is a particle type 1, it will stay in 3 out 4 times
dX = -dX;
dY = -dY;
if (kind==2 && s>3) {
//if this is a particle type 2, it will get out 3 out 4 times
dX = -dX;
dY = -dY;
//move particle


//this method displays objects that belong to the Particle class as squares
class SquarePartView{
//data particular to this view
color c;
int w, h;
//constructor – creates a square
SquarePartView (color tempC, int tempW, int tempH){
c = tempC;
//method: draw the square)
void drawParticle(ParticleModel p){
//the location of x and y come from the particle
//the width and height are characteristics of this class
fill(c, 30);
ellipse(p.x, p.y, w+20, h+20);
ellipse(p.x, p.y, w, h);



//this is an array list that holds all particles, contains methods to create and delete them and to update all particles
class ParticleCloudModel{
ArrayList  particleList;
int counter;

ParticleCloudModel () {
particleList = new ArrayList();

//this method creates a particle
void makePart(float k){

float tempdX = random (-3,3);
float tempdY=random(-3,3);
//this creates a new particle with random dx and dy
ParticleModel newPar = new ParticleModel(width/2, height/2, tempdX, tempdY, k);
//add new particle to array

//Update function cycles thru list and removes all particles no longer on screen, then runs the update on all remaining ones
void update(){
//this function creates new particles depending on the mouse button pressed
if (mousePressed && mouseButton == LEFT) {
//make particles of type 1 – stay in
//decrease r but never less than 10
if (circler.r>=20){
if (mousePressed && mouseButton == RIGHT){
//make particles of type 2 — go out
//increase r but never more than 250
if (circler.r<=250){
circler.r= circler.r+1;}
else {circler.r=250;}
if (!mousePressed){
//make particles a fifth of the time
float r=random(1,60);
if (r<2) {makePart(1);}
if (r>59) {makePart(2);}


//check through whole array and update each particle
for (int i = particleList.size()-1; i >= 0; i–) {
// An ArrayList doesn’t know what it is storing so we have
// to cast the object coming out
ParticleModel p = (ParticleModel) particleList.get(i);
//if a particle has fallen off the screen, delete it
if (p.x>width || p.y>height) {
particleList. remove(i);}

}//end of update()
}//end of class

//this is the viewer to the particlecloudmodel
class ParticleCloudView {
SquarePartView v;
ParticleCloudView(SquarePartView tempV){

//method is applying the view loaded to each particle in particlecloudmodel
void drawPartCloud(ParticleCloudModel a){

//cycle thru array list to get each particle
for (int i = a.particleList.size()-1; i >= 0; i–) {
// An ArrayList doesn’t know what it is storing so we have
// to cast the object coming out
ParticleModel p = (ParticleModel) a.particleList.get(i);
// call that method for each particle


// interaction
void mousePressed(){
if (mouseButton==LEFT){
//initialize the particle cloud view with the view set up
partCloudV.v = new SquarePartView (color(255), 10, 10);
if (mouseButton==RIGHT){
//initialize the particle cloud view with the view set up
partCloudV.v = new SquarePartView (pc, 10, 10);


Well let’s be honest, we are certainly figuring this out as we go.

We have stripped out the generative music (and thus, the pipe organ) from the project. Probably mostly because we thought this would help us distinguish us from the other generative music group, and also because we weren’t committed to that learning anything very interesting. That being said we are still working on defining what part of our project learns – and also, if it is art.

We are keeping the monsters, and the buttons. People will be able to push the buttons to make the monsters do something. Also we will be tracking people as they move, and translating that into monster movement. Very reactive. The plan is to save all of these movement patterns and create generated monsters that interact with the real people-based monsters in the animation.

Things to implement

  • People (blob) tracking
  • Masking the blobs maybe into some sort of particle system that the monster can be drawn as.
  • Building a box to house projector, laptop
  • Constructing button interface
  • Animating the scene with monster (maybe 8-bit)
  • Storing the movement patterns so that they can be recreated
  • Bonus: customizable monster creation?

Final(-ish) description:

This installation appears as a projection of an outside space, with butterflies flying around. As the visitor comes near to the projection, their shadow appears in the screen (as in the word rain installation). The butterflies respond to the shadow, approaching people and flying around them. Over time, butterflies will reproduce with each other. The color of the butterflies is a characteristic that will be passed between generations to help visitors understand what is happening. Butterflies will also change over time to reflect an aspect of the environment, probably the color of the visitor’s clothing.

Machine learning: The system tries to optimize the ability of the population of butterflies’ to touch the shadow. To do this, the characteristics of the butterflies are shuffled from one generation to the next using the genetic algorithm based on each butterfly’s fitness.


– what factors in the enviroment will affect the evolution of flies?

– how are flies adapting?


– programmed detecting visitors and drawing outline

– color palette and concept art

– programming fly class

– programming flies approach to outline (“collision”)

Initial concepts of Stage sets and props