# Dynamic Appearance Modeling for Human Tracking

> How can a computer keep following a person who walks behind a tree? Engineers built a video system that learns, from just a few frames, how a person’s appearance changes with each stride and where they are heading. It predicts the person while they are out of sight and finds them again afterwards, even when people cross paths or the camera moves.

Summary by Kersey Lawrence, Original Wisdom. Updated 9 November 2026. Page: https://www.originalwisdom.com/tracking-literature/2006-lim-morariu-dynamic-appearance-modeling-human/

## Citation and access

- **Authors:** Hwasup Lim, Vlad I. Morariu, Octavia I. Camps and Mario Sznaier
- **Year published:** 2006
- **Journal or source:** Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), Volume 1
- **Full citation:** Lim, H., Morariu, V. I., Camps, O. I. and Sznaier, M. 2006. Dynamic appearance modeling for human tracking. In: Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), Volume 1: 751–757. IEEE.
- **DOI:** 10.1109/CVPR.2006.104
- **Access status:** Publisher link only
- **Where to find it:** https://doi.org/10.1109/CVPR.2006.104

## Tags

- **Methods and technology:** Machine learning
- **Data type:** Modelling or theoretical
- **Observer qualification:** Not needed
- **Discipline:** Human tracking
- **Year published:** 2006

## Detailed keywords

- **Species:** Studied: humans (people walking, filmed on video). No animals.
- **Location and habitat:** Not reported. The test videos are described only as “outdoor scenes”, one with a tree that hides the person for a while. The authors work at The Pennsylvania State University (University Park, Pennsylvania) and the University of Maryland (College Park, Maryland), United States, but the paper does not say where the videos were filmed.
- **Topics:** computer vision; human tracking in video; following a person’s changing appearance while walking; predicting where a person will be and what they will look like in later frames; occlusion (a person hidden behind an object or another person); telling people apart when they cross paths; tracking with a moving camera; model drift; scene clutter; walking cycle.
- **Statistics or models:** Locally Linear Embedding (LLE), a nonlinear dimensionality reduction method; Caratheodory-Fejer (CF) interpolation for robust identification of the dynamics; autoregressive moving average with exogenous input (ARMAX) model and linear time invariant (LTI) state space models; Radial Basis Function (RBF) network to rebuild images; Condensation filter (a particle filter) for tracking; background subtraction, meanshift algorithm and YCbCr 4:2:0 colour compression for preprocessing; Fourier transform to find the basic frequency of the walking cycle. Compared with constant acceleration and constant velocity dynamics.
- **Data type:** Modelling or theoretical: a computer method tested on several outdoor video sequences from stationary and moving colour cameras. Who filmed the videos, and when, is not reported. Results are shown as example frames; no accuracy figures are given.

## Synopsis

### 1. What they did

The authors built a computer system that follows walking people through video footage. As a person walks, their appearance in the picture keeps changing, because arms and legs swing, the viewing angle changes and the light changes. Earlier computer trackers often lost a person who passed behind an object or another person, or slowly drifted onto the background.

Their system learns two things from the first few frames of a video, while the person is in full view and the camera is still:

- How the person’s appearance changes. Each picture of the person is reduced to three numbers with a method called Locally Linear Embedding (LLE). For a walking person, these numbers trace a repeating loop, matching the walking cycle. The way the numbers change over time is then modelled with Caratheodory-Fejer (CF) interpolation, a robust method that fits a model to the data rather than assuming one in advance. A Radial Basis Function (RBF) network turns predicted numbers back into a predicted picture of the person.

- How the person moves across the picture. The same robust method is used to model the person’s path, so the system can predict where they will be in the next frames.

The predicted picture at the predicted place is then used as a template to search for the person in each new frame. When the person is hidden, the system keeps predicting where they are and what they look like until it finds them again. When two people cross, it decides who is in front by checking which predicted picture best matches the colours where they overlap.

They tested the system on outdoor videos filmed with stationary and moving colour cameras.

### 2. What they found

- Better predictions: pictures of a walking person generated by their learned model stayed realistic for longer than pictures generated by assuming simple constant acceleration.

- Hidden behind a tree: a standard tracker that assumed constant speed lost the person after they walked behind a tree. Their system predicted the person’s appearance and position throughout the long occlusion and picked them up again afterwards. Because the template is predicted rather than updated from what is seen, it was not spoiled by the tree, so the tracker did not drift.

- People crossing: when three people walked across each other, the system correctly placed one person behind and the other two in front, and kept track of who was who.

- Moving camera: once the person’s appearance had been learned, the system kept following them even when the camera moved.

The results are shown as example video frames; the paper gives no overall accuracy figures.

### 3. What they concluded

- Modelling a person’s changing appearance and movement in three steps (reducing each picture to a few numbers, identifying how those numbers change, and turning them back into pictures) lets a computer predict where a person will be and what they will look like.

- These predictions, learned from only a few initial frames, work as templates for tracking and identifying moving people even when they are hidden for a while or the scene is cluttered.

- The authors were working on using the error limits from CF interpolation to tell when the model no longer fits (for example, when a person switches from walking to running) and a new model is needed.

### 6. Other technologies, skills or innovations

- Robust identification of dynamics by Caratheodory-Fejer (CF) interpolation, which gives worst-case limits on the prediction error.

- Locally Linear Embedding (LLE) to reduce each picture to three numbers while keeping the order of frames in time.

- A Radial Basis Function (RBF) network to rebuild predicted pictures of the person.

- A Condensation filter that compares the colours of the tracked area with the predicted template, pixel by pixel.

- Layer selection: deciding which of two overlapping people is in front by colour similarity.

### 8. Implications for tracking (Original Wisdom's observations)

_These are Original Wisdom's observations, not the authors'._

- The paper is about “tracking” in the computer vision sense: following a person through video by their appearance, not by tracks and signs on the ground.

- The system’s approach has a familiar shape for trackers: learn how a particular person moves and looks, predict where they will be when out of sight, and pick them up again when they reappear. A tracker who loses the trail does something similar when predicting where to look next.

## Cite this summary

Lawrence, K. 2026. Summary of: Dynamic Appearance Modeling for Human Tracking (Lim and Morariu et al., 2006). Original Wisdom, Tracking Literature. https://www.originalwisdom.com/tracking-literature/2006-lim-morariu-dynamic-appearance-modeling-human/