Dynamic Appearance Modeling for Human Tracking

1. What they did

The authors built a computer system that follows walking people through video footage. As a person walks, their appearance in the picture keeps changing, because arms and legs swing, the viewing angle changes and the light changes. Earlier computer trackers often lost a person who passed behind an object or another person, or slowly drifted onto the background.

Their system learns two things from the first few frames of a video, while the person is in full view and the camera is still:

• How the person’s appearance changes. Each picture of the person is reduced to three numbers with a method called Locally Linear Embedding (LLE). For a walking person, these numbers trace a repeating loop, matching the walking cycle. The way the numbers change over time is then modelled with Caratheodory-Fejer (CF) interpolation, a robust method that fits a model to the data rather than assuming one in advance. A Radial Basis Function (RBF) network turns predicted numbers back into a predicted picture of the person.
• How the person moves across the picture. The same robust method is used to model the person’s path, so the system can predict where they will be in the next frames.

The predicted picture at the predicted place is then used as a template to search for the person in each new frame. When the person is hidden, the system keeps predicting where they are and what they look like until it finds them again. When two people cross, it decides who is in front by checking which predicted picture best matches the colours where they overlap.

They tested the system on outdoor videos filmed with stationary and moving colour cameras.

2. What they found

• Better predictions: pictures of a walking person generated by their learned model stayed realistic for longer than pictures generated by assuming simple constant acceleration.
• Hidden behind a tree: a standard tracker that assumed constant speed lost the person after they walked behind a tree. Their system predicted the person’s appearance and position throughout the long occlusion and picked them up again afterwards. Because the template is predicted rather than updated from what is seen, it was not spoiled by the tree, so the tracker did not drift.
• People crossing: when three people walked across each other, the system correctly placed one person behind and the other two in front, and kept track of who was who.
• Moving camera: once the person’s appearance had been learned, the system kept following them even when the camera moved.

The results are shown as example video frames; the paper gives no overall accuracy figures.

3. What they concluded

• Modelling a person’s changing appearance and movement in three steps (reducing each picture to a few numbers, identifying how those numbers change, and turning them back into pictures) lets a computer predict where a person will be and what they will look like.
• These predictions, learned from only a few initial frames, work as templates for tracking and identifying moving people even when they are hidden for a while or the scene is cluttered.
• The authors were working on using the error limits from CF interpolation to tell when the model no longer fits (for example, when a person switches from walking to running) and a new model is needed.

6. Other technologies, skills or innovations

• Robust identification of dynamics by Caratheodory-Fejer (CF) interpolation, which gives worst-case limits on the prediction error.
• Locally Linear Embedding (LLE) to reduce each picture to three numbers while keeping the order of frames in time.
• A Radial Basis Function (RBF) network to rebuild predicted pictures of the person.
• A Condensation filter that compares the colours of the tracked area with the predicted template, pixel by pixel.
• Layer selection: deciding which of two overlapping people is in front by colour similarity.