Selecting the right features for the representation of human motion dynamics is the first step towards developing a recognition paradigm. We cannot just use intensity or color trajectories as they are not representative of the true motion. In the past, researchers have used motion capture data, however accurately extracting limbs and joints from a video without using stick-up markers is difficult. Moreover, there is always the question whether the representation should be unique to the whole person i.e. global, or whether a collection of local representations such as a parts-based approach is better. Global motion models allow for the construction of a spatially compact and temporally consistent formulation. However the drawback is that these models break down when there is a large amount of occlusion of the subject, in which case parts-based or local approaches are applicable. Local models however do not take into account of the dynamics of the scene. Since our goal is to explicitly model the dynamics of human motion, we propose a global model.

Assuming that there is only one person moving in the scene and the background is stationary, we first compute the optical flow for each frame of the entire video. We quantize the optical flow as shown in figure 2. Each optical flow vector is binned according to its principal angle with the horizontal. The contribution to each bin is wiehted by the magnitude of the optical flow vector. This gives us a scale and fronto-parallel direction invariant feature, a Histogram of Oriented Optical Flow (HOOF) in each frame. The time series of such HOOF represents the motion in the scene.

We model the temporal evolution of different lip features using the Linear Dynamical Systems Framework. More specifically, a sequence of feature trajectories is assumed to be a realization from a second-order stationary stochastic process and hence it can be modeled with a state space model: To find the parameters for the Linear Dynamical System M = (x₀, A, B, C, R) we consider the use of two popular Linear System Identification methods, namely:
1. N4SID
2. PCA-based suboptimal ID

Once the system parameters have been identified using any of the above mentioned methods, we need to define a measure of affinity between two given Linear Dynamical Systems. Over the past few years, a number of metrics have been proposed on the space of linear dynamical systems. A number of distances are based on the notion of subspace angles between two systems. These subspace angles are precisely the angles between the infinite observability matrices of the two systems. Since the infinite observability matrices cannot be found in a practical situation, we can estimate the subspace angles indirectly through the solution of a generalized eigen-value problem that translates to finding the solution of a Lyapunov equation. Once the subspace angles, {θ_i}²ⁿ_i=1, have been found, a number of distances can be defined as:

Finsler distance
Martin distance
Frobenius distance

Another method for defining a measure of affinity between two LDS is through a kernel. The Martin Kernel, defined as:

Martin kernel

comes directly from the associated Martin Distance and is immediately computable from the subspace angles. Chan and Vasconcelos proposed another metric based on the Kullback-Leibler Divergence between the probability distributions of the outputs of the dynamical systems:

Vishwanathan et. al. introduced the family of Binet Cauchy kernels for the analysis of dynamical systems. One of these kernels, called the trace kernel between two ARMA model can be evaluated as:

Once an affinity metric has been defined between two linear dynamical systems, we use k-Nearest Neighbors (k-NN) Classification in the case of distances or k-Furthest Neighbors (k-FN) Classification for kernels. Also to perform more sophisticated classification, we use kernel SVMs with the previously defined kernels.

We perform a large number of experiments, based on both the scenarios, namely 'Name' and 'Digit' using both the Landmarks features and the Distances features. To evaluate the performance of the different lip features, identification methods, distances and kernels for LDSs, and classification methods, we performed classification of the data sets G4, G8, and G12 (G12 is a group of 12 people who's lip motions were investigated. G4 is a smaller group of the first 4 people of these 12 people and G8 consists of the first 8 people) using leave-one-out classification. The following figure shows the group make up:

Groups - First row of the figure shows G4, the first 2 rows make up G8 and all the figures together represent G12.

The following table shows the classification errors of all groups in the name and digit scenarios for landmark trajectories, L, using the PCA-based identification method, and using both 1-NN classification with 5 different distances and SVM classification with 3 different kernels.