Class 12

Class 12 - More motion tracking applications

Motion tracking examples

Finding flow

Creative examples without motion tracking

Other applications

Flow

For some applications, it may be difficult or not necessary to identify the shape of an object. All we need is to find out the changes in motion within the camera frame. In this section, we try to use a simplified version to identify the optical flow.

In this simple version, we divide the screen into grids. We use two constant variables

NROWS (number of rows)
NCOLS (number of columns)

And we have 2 PImage variables

img1
img2

We use img1 to store the current image from the camera (JMyron) and img2 the previous image. In each frame, we copy the image from img1 to img2 first and then refresh img1 with the latest camera image.

We use a temporary data structure Point class to hold the information for each point.

class Point {
// It is a temporary data structure to hold a point information.
  int x, y;
  
  Point(int _x, int _y) {
    x = _x;
    y = _y;
  }
}

All the data elements for the program are:

import JMyron.*;

JMyron m;
PImage img1, img2;
// img1 and img2 are the location to temporarily store the previous 
// and current frame.
final int NROWS = 18;
final int NCOLS = 24;
final float LIMIT = 40.0;
int w, h;
// w and h are the size of the grid.

The setup() function is pretty straight forward.

void setup() {
   size(320,240);
   m = new JMyron();
   m.start(width,height);
   println(m.getForcedWidth());
   println(m.getForcedHeight());
   m.findGlobs(0);
   img1 = createImage(width,height,ARGB);
   img2 = createImage(width,height,ARGB);
   noFill();
   stroke(255,200,0);
   smooth();
   frameRate(15);
   w = width/NCOLS;
   h = height/NROWS;
}

We delay the complexity of the draw() function by just putting a findFlow() function there.

void draw() {
   background(0);
// copy the image from img1 to img2 - the previous frame.
   img2.copy(img1,0,0,img1.width,img1.height,
    0,0,img2.width,img2.height);
   img2.updatePixels();
   m.update();
// update the current frame to img1.
   m.imageCopy(img1.pixels);
   img1.updatePixels();
   image(img1,0,0);
   findFlow();
}

The findFlow() function is a double for loop to scan through the pixels in img2 first. Note that xOff and yOff are half the size of w and h for the definition of the neighborhood size. We also start the loop at index 1, rather than 0, in order to skip those points on the edges.

void findFlow() {
  int xOff = w/2;
  int yOff = h/2;
  for (int r=1;r<NROWS;r++) {
    for (int c=1;c<NCOLS;c++) {
      Point p1 = new Point(c*w, r*h);
      Point p2 = findPoint(p1.x, p1.y, xOff, yOff);
      drawLine(p1,p2);
    }
  }
}

There are 2 remaining functions, findPoint() and drawLine(). The simple one is drawLine(). It draws a straight line connecting the 2 points, from p2 to p1. In order to show the flow direction, we draw 2 more tiny lines to form an arrow. The two tiny lines are drawn at 30 degree (PI/6) connecting to the end point, _p.

void drawLine(Point _p, Point _q) {
// draw the arrow line from point _q to point _p.
   if (_p.x!=_q.x || _p.y!=_q.y) {
     line(_p.x,_p.y,_q.x,_q.y);
     float ang = atan2(_q.y-_p.y,_q.x-_p.x);
     float ln = w/3.0;
     float tx = _p.x + ln*cos(ang-PI/6);
     float ty = _p.y + ln*sin(ang-PI/6);
     line(_p.x,_p.y,tx,ty);
     tx = _p.x + ln*cos(ang+PI/6);
     ty = _p.y + ln*sin(ang+PI/6);
     line(_p.x,_p.y,tx,ty);
   } 
   else {
     line(_p.x,_p.y,_q.x,_q.y);
   }
}

The complicated one is the findPoint() function, which needs to search through the neighborhood in img2 of a given point in img1.

Point findPoint(int _x, int _y, int _xo, int _yo) {

// Given a pixel (_x, _y) in img1, we search the neighborhood of that
// pixel in img2 and try to find a matching colour.
// 
// The neighborhood size is defined by (w x h) and the boundaries are
// x0 - left
// x1 - right
// y0 - top
// y1 - right

   int x0 = _x - _xo;
   int x1 = _x + _xo;
   int y0 = _y - _yo;
   int y1 = _y + _yo;

// Initialize the minimum difference to a high value.
// Loop through the pixels in img2 within the boundary.
// Find the pixel with minimum difference from the original one 
// in img1.

   float minDiff = 999999999;
   Point p = new Point(_x,_y);
   color c1 = img1.pixels[_y*img1.width+_x];
   color c2 = img2.pixels[_y*img2.width+_x];
   if (!matchCol(c1,c2)) {
      for (int r=y0;r<y1;r++) {
        for (int c=x0;c<x1;c++) {
          c2 = img2.pixels[r*img2.width+c];
          float diff = dist(red(c1),green(c1),blue(c1),
          red(c2),green(c2),blue(c2));
          if (diff<minDiff) {
            minDiff = diff;
            p.x = c;
            p.y = r;
          }
        }
      }
   }
   return p;
}

The matchCol() function is the same one we have been using in previous class.

boolean matchCol(color c1, color c2) {
// Compare two colour values and see if they are similar.
   float d = dist(red(c1),green(c1),blue(c1),
     red(c2),green(c2),blue(c2));
   return (d<LIMIT);
}

Now combine everything to create the first program visualizing the flow.

Flow in region 1

The next step is not to use the whole screen as the control area. We define a smaller region with a class Region and calculate the flow within that region to determine the actions.

Given all those arrows inside the region, all we need to do is to find out the overall (average) movement and store it in 2 variables,

vx - movement in x direction
vy - movement in y direction

such that the region can move according to these values.

You can play around with the source and modify for your own creative needs

Region class

class Region {
               
//  left, top, right, bottom define the region rectangle.
//  xOff, yOff are half the size of the boundary for the neighbours.
//  vx, vy define the x and y components of the velocity.
//  rW, rH are the width and height of the region.

  int left, top, right, bottom;
  int xOff, yOff;
  float vx, vy;
  int rW, rH;

  Region(int _l, int _t, int _r, int _b) {
    left = _l;
    top = _t;
    right = _r;
    bottom = _b;
    xOff = w/2;
    yOff = h/2;
    vx = 0.0;
    vy = 0.0;
    rW = right - left;
    rH = bottom - top;
  }

  void update() { 
//  Calculate the flow information within the region.
    vx = 0.0;
    vy = 0.0;
    for (int r=top;r<bottom;r+=h) {
      for (int c=left;c<right;c+=w) {
        Point p1 = new Point(0,0);
        p1.x = constrain(c,0,img1.width-1);
        p1.y = constrain(r,0,img1.height-1);
        Point p2 = findPoint(p1);
   
// p1 is the pixel in img1 - current image
// p2 is the corresponding point in img2 - previous image
// vx, vy - velocity is computed by the sum of difference
// between the current position and previous position.

        vx += (p1.x-p2.x);
        vy += (p1.y-p2.y);
        drawFlow(p1,p2);
      }
    }
    moveRegion();
    render();
  }

  Point findPoint(Point _p) {
    Point p = new Point(_p.x,_p.y);
    color col1 = img1.pixels[_p.y*img1.width+_p.x];
    color col2 = img2.pixels[_p.y*img2.width+_p.x];
    if (!matchCol(col1,col2)) {
      float minDiff = 9999999999.0;
      int l = constrain(_p.x-xOff,0,img2.width-1);
      int t = constrain(_p.y-yOff,0,img2.height-1);
      int r = constrain(_p.x+xOff,0,img2.width-1);
      int b = constrain(_p.y+yOff,0,img2.height-1);
      for (int j=t;j<b;j++) {
        for (int i=l;i<r;i++) {
          col2 = img2.pixels[j*img2.width+i];
          float d = dist(red(col1),green(col1),blue(col1),
            red(col2),green(col2),blue(col2));
          if (d<minDiff) {
            minDiff = d;
            p.x = i;
            p.y = j;
          }
        }
      }
    }
    return p;
  }

  void drawFlow(Point p1, Point p2) {
    stroke(0);
    if (p1.x!=p2.x || p1.y!=p2.y) {
      line(p1.x,p1.y,p2.x,p2.y);
      float ang = atan2(p2.y-p1.y,p2.x-p1.x);
      float ln = w/3.0;
      float tx = p1.x + ln*cos(ang-PI/6);
      float ty = p1.y + ln*sin(ang-PI/6);
      line(p1.x,p1.y,tx,ty);
      tx = p1.x + ln*cos(ang+PI/6);
      ty = p1.y + ln*sin(ang+PI/6);
      line(p1.x,p1.y,tx,ty);
    } else {
      line(p1.x,p1.y,p2.x,p2.y);
    }
  }

  void moveRegion() {
    vx /= 6;
    vy /= 6;

    left += vx;
    right += vx;
    top += vy;
    bottom += vy;

    if (left<0) {
      left = 0;
      right = rW;
    }
    if (right>width) {
      right = width;
      left = right - rW;
    }
    if (top<0) {
      top = 0;
      bottom = rH;
    }
    if (bottom>height) {
      bottom = height;
      top = height - rH;
    }
  }

  void render() {
    rectMode(CENTER);
    stroke(255,0,0);
    fill(255,255,0,100);
    pushMatrix();
    translate((left+right)/2,(top+bottom)/2);
    rect(0,0,rW,rH);
    popMatrix();
  }
}

Point class

class Point {
  int x, y;
  
  Point(int _x, int _y) {
    x = _x;
    y = _y;
  }
}

Main program

import JMyron.*;


final int NROWS = 18;
final int NCOLS = 24;
final float LIMIT = 40.0;
JMyron m;
Region reg1;
PImage img1, img2;
int w, h;

void setup() {
  size(320,240);
  m = new JMyron();
  m.start(width,height);
  m.findGlobs(0);
  img1 = createImage(width,height,ARGB);
  img2 = createImage(width,height,ARGB);
  noFill();
  stroke(255,200,0);
  smooth();
  frameRate(15);
  w = width/NCOLS;
  h = height/NROWS;
   
//  reg1 = new Region(?,?,?,?);
  m.update();
  m.imageCopy(img1.pixels);
  m.imageCopy(img2.pixels);
  img1.updatePixels();
  img2.updatePixels();
}

void draw() {
  background(0);
  img2.copy(img1,0,0,width,height,
    0,0,width,height);
  img2.updatePixels();
  m.update();
  m.imageCopy(img1.pixels);
  img1.updatePixels();
  image(img1,0,0);
//  reg1.???;
}

boolean matchCol(color c1, color c2) {
  float d = dist(red(c1),green(c1),blue(c1),
    red(c2),green(c2),blue(c2));
  return (d<LIMIT);
}
 


void stop() {

  m.stop();

  super.stop();

}

Another variation: change the flow information into force to control the acceleration, rather than velocity of the Region.

Flow with rotation

The overall motion information from the arrows can give you another piece of hint. Try to imagine that the arrows can also show the direction of rotation of the object (region) on the screen. That is, the object can rotate around its centre of gravity. The arrows are thus indicators of how it rotates.

cx, cy is the centre of the region. When a pixel moves from p2 to p1, we try to approximate the angular movement it induces. We are going to use the sum:

d2*d1*sin(a2-a1)

for the purpose.

Region with translation and rotation

The following combines the linear and angular movements to form an intuitive interface. It may need some time to adapt to the motion.

Sources for discussion:

Applications with Flow

Besides using flow as an interaction device, you can consider creative application by making use of the flow information. Here is a simple example.

Given the flow information for each point in the grid, if we take the length of each arrow and use it as the z shift of an vertex at that point. See what happens.

Going back to the code, the length of each arrow is:

void findFlow() {
  int xOff = w/2;
  int yOff = h/2;
  for (int r=1;r<NROWS;r++) {
    for (int c=1;c<NCOLS;c++) {
      Point p1 = new Point(c*w, r*h);
      Point p2 = findPoint(p1.x, p1.y, xOff, yOff);
      drawLine(p1,p2);
    }
  }
}

The length of each arrow is the distance between p1 and p2, i.e.,

dist(p1.x,p1.y,p2.x,p2.y)

Rather than displaying the webcam video with an image() command, we can use texture map to skin the image onto a grid.

First of all, we modify the Point class to include z dimension.

class Point {

  int x, y, z;
  
  Point(int _x, int _y) {
    x = _x;
    y = _y;
    z = 0;
  }
}

A 2D array grid is defined to maintain the vertex information.

Point [][] grid = new Point[NROWS+1][NCOLS+1];

To initialize the grid information,

void initGrid() {
   for (int r=0;r<=NROWS;r++) {
      for (int c=0;c<=NCOLS;c++) {
         grid[r][c] = new Point(c*w,r*h);
      }
   }
}

To map the image texture onto the grid,

void mapGrid(PImage _i) {
  noStroke();
  for (int r=0;r<NROWS;r++) {
    beginShape(QUAD_STRIP);
    texture(_i);
    for (int c=0;c<=NCOLS;c++) {
      vertex(grid[r][c].x,grid[r][c].y,grid[r][c].z,
      c*w,r*h);
      vertex(grid[r+1][c].x,grid[r+1][c].y,grid[r+1][c].z,
      c*w,(r+1)*h);
    }
    endShape();
  }
}

We can then modify the findFlow() function to change the z value for the points in the grid.

void findFlow() {
  int xOff = w/2;
  int yOff = h/2;
  for (int r=1;r<NROWS;r++) {
    for (int c=1;c<NCOLS;c++) {
      Point p1 = new Point(c*w, r*h);
      Point p2 = findPoint(p1.x, p1.y, xOff, yOff);
      drawLine(p1,p2);
      grid[r][c].z = int(dist(p1.x,p1.y,p2.x,p2.y));
    }
  }
}

You can also use the flow information to change the x and y values of the points in the grid.

Creative examples

A day at the gallery, Romy Achituv

4th Dimension, Zbig Rybczynski

Khronos Projector

Demo video (wmv)

The Last Clock

Demo video (mov)

TX transform

Library

Face detection

Download and unpack the file pFaceDetect.zip into your Processing libraries folder.

Add the following file into the data folder of your sketch.
haarcascade_frontalface_default.xml

If you do not have OpenCV installed, copy the following files in the Processing root folder.

Here is a simple demonstration program.

import pFaceDetect.*;  
import JMyron.*; 
    
PFaceDetect face;  
JMyron m;  
PImage img;   
   
void setup() {    
   size(320,240);    
   m = new JMyron();    
   m.start(width,height);    
   m.findGlobs(0);    
   face = new PFaceDetect(this,width,height,    
     "haarcascade_frontalface_default.xml");    
   frameRate(15);    
   img = createImage(width,height,ARGB);    
   rectMode(CORNER);    
   noFill();    
   stroke(255,0,0);    
   smooth();  
}    
 
void draw() {    
   background(0);    
   m.update();    
   arraycopy(m.cameraImage(),img.pixels);    
   img.updatePixels();    
   face.findFaces(img);    
   image(img,0,0);    
   drawFace();  
}    
 
void drawFace() {    
   int [][] res = face.getFaces();    
   if (res.length>0) {      
      for (int i=0;i<res.length;i++) {        
         int x = res[i][0];        
         int y = res[i][1];        
         int w = res[i][2];        
         int h = res[i][3];        
         rect(x,y,w,h);      
      }    
   }  
}    
 
void stop() {    
   m.stop();    
   super.stop();  
}

Reference