S. Vedula, S. Baker, P. Rander, R. T. Collins, T. Kanade. Three-Dimensional Scene Flow. Transactions on Pattern Analysis and Machine Intelligence, 2005.
A fundamental assumption in optical flow is brightness constancy. Let $\boldsymbol{x} = (x, y, z)$ be a point on a moving surface and $\boldsymbol{u} = (u, v)$ its projection onto the image plane:
$u = \frac{P_1 (x, y, z, 1)^T}{P_3 (x, y, z, 1)^T}$(1)
$v = \frac{P_2 (x, y, z, 1)^T}{P_3 (x, y, z, 1)^T}$(2)
where $P = (P_1, P_2, P_3)^T \in \mathbb{R}^{3 \times 4}$ projection matrix. Optical flow is the change of $\boldsymbol{u}$ with respect to $t$: $\frac{\partial \boldsymbol{u}}{\partial t}$. Similarly, scene flow is the change of $\boldsymbol{x}$ with respect to $t$: $\frac{\partial \boldsymbol{x}}{\partial t}$.
The brightness constancy condition can be derived by considering the net radiance of light at point $(x, y, z)^T$ at time $t$ denoted by $\boldsymbol{r}(x, y, z; t)$. Assuming $\boldsymbol{x}$ to lie on a Lambertian surface with normal $\boldsymbol{n}(\boldsymbol{x};t)$, the registered intensity is proportional to the albedo $\rho(\boldsymbol{x};t)$:
where $K$ depends on the camera (and $\boldsymbol{u}$ and $\boldsymbol{x}$ are related as in Equations (1) and (2)). Assuming that the albedo does not change, i.e. $\frac{\partial \rho}{\partial t} = 0$, the basis of optical flow algorithms is the time derivative of Equation (3):
where $\nabla_\boldsymbol{u}$ denotes the spatial gradient. Vedula et al. mention two cases where $\frac{\partial}{\partial t}\left(\boldsymbol{n}(\boldsymbol{x}; t)^T \boldsymbol{r}(\boldsymbol{x};t)\right) = 0$ can be assumed:
the illumination is constant;
or the surface normal does not change suddenly
Then, the well-known brightness constancy assumption is
where $\nabla_u I$ is the spatial gradient of $I$.
The relationship between the optical flow $\frac{\partial \boldsymbol{u}}{\partial t}$ and the scene flow $\frac{\partial \boldsymbol{x}}{\partial t}$ is given by the Jacobian matrix $\frac{\partial \boldsymbol{u}}{\partial \boldsymbol{x}}$:
As the right-hand side only depends on the scene (i.e. albedo, scene structure and illumination), $\frac{\partial \boldsymbol{x}}{\partial t}$ need to be constant in order for Equation (4) to hold. Therefore, for any number of points on the same surface, the corresponding copies of Equation (4) will be linearly depending. This mans that it is impossible to compute scene flow independently for points on the same surface.
What is your opinion on this article? Let me know your thoughts on Twitter @davidstutz92 or LinkedIn in/davidstutz92.
A fundamental assumption in optical flow is brightness constancy. Let $\boldsymbol{x} = (x, y, z)$ be a point on a moving surface and $\boldsymbol{u} = (u, v)$ its projection onto the image plane:
$u = \frac{P_1 (x, y, z, 1)^T}{P_3 (x, y, z, 1)^T}$(1)
$v = \frac{P_2 (x, y, z, 1)^T}{P_3 (x, y, z, 1)^T}$(2)
where $P = (P_1, P_2, P_3)^T \in \mathbb{R}^{3 \times 4}$ projection matrix. Optical flow is the change of $\boldsymbol{u}$ with respect to $t$: $\frac{\partial \boldsymbol{u}}{\partial t}$. Similarly, scene flow is the change of $\boldsymbol{x}$ with respect to $t$: $\frac{\partial \boldsymbol{x}}{\partial t}$.
The brightness constancy condition can be derived by considering the net radiance of light at point $(x, y, z)^T$ at time $t$ denoted by $\boldsymbol{r}(x, y, z; t)$. Assuming $\boldsymbol{x}$ to lie on a Lambertian surface with normal $\boldsymbol{n}(\boldsymbol{x};t)$, the registered intensity is proportional to the albedo $\rho(\boldsymbol{x};t)$:
$I(\boldsymbol{u}; t) = -K \rho(\boldsymbol{x}; t) \left(\boldsymbol{n}(\boldsymbol{x};t)^T \boldsymbol{r}(\boldsymbol{x};t)\right)$(3)
where $K$ depends on the camera (and $\boldsymbol{u}$ and $\boldsymbol{x}$ are related as in Equations (1) and (2)). Assuming that the albedo does not change, i.e. $\frac{\partial \rho}{\partial t} = 0$, the basis of optical flow algorithms is the time derivative of Equation (3):
$\frac{\partial I}{\partial t} = \nabla_\boldsymbol{u} I^T \frac{\partial \boldsymbol{u}}{\partial t} + \frac{\partial I}{\partial t} = - K \rho(\boldsymbol{x}) \frac{\partial}{\partial t}\left( \boldsymbol{n}(\boldsymbol{x}; t)^T \boldsymbol{r}(\boldsymbol{x};t)\right)$.
where $\nabla_\boldsymbol{u}$ denotes the spatial gradient. Vedula et al. mention two cases where $\frac{\partial}{\partial t}\left(\boldsymbol{n}(\boldsymbol{x}; t)^T \boldsymbol{r}(\boldsymbol{x};t)\right) = 0$ can be assumed:
Then, the well-known brightness constancy assumption is
$\nabla_u I \frac{\partial \boldsymbol{u}}{\partial t} + \frac{\partial I}{\partial t} = 0$
where $\nabla_u I$ is the spatial gradient of $I$.
The relationship between the optical flow $\frac{\partial \boldsymbol{u}}{\partial t}$ and the scene flow $\frac{\partial \boldsymbol{x}}{\partial t}$ is given by the Jacobian matrix $\frac{\partial \boldsymbol{u}}{\partial \boldsymbol{x}}$:
$\frac{\partial \boldsymbol{u}}{\partial t} = \frac{\partial \boldsymbol{u}}{\partial \boldsymbol{x}} \frac{\partial \boldsymbol{x}}{\partial t}$.
In this sense, the brightness constancy condition can be rewritten as
$\nabla_u I \left(\frac{\partial \boldsymbol{u}}{\partial \boldsymbol{x}} \frac{\partial \boldsymbol{x}}{\partial t}\right) + \frac{\partial I}{\partial t} = 0.$(4)
Differentiating Equation (3) with respect to $\boldsymbol{x}$ yields:
$\nabla_\boldsymbol{u} \frac{\partial \boldsymbol{u}}{\partial \boldsymbol{x}} = - K \cdot \nabla_\boldsymbol{x} \left(\rho(\boldsymbol{n}(\boldsymbol{x};t)^T \boldsymbol{r}(\boldsymbol{x};t)\right)$.
As the right-hand side only depends on the scene (i.e. albedo, scene structure and illumination), $\frac{\partial \boldsymbol{x}}{\partial t}$ need to be constant in order for Equation (4) to hold. Therefore, for any number of points on the same surface, the corresponding copies of Equation (4) will be linearly depending. This mans that it is impossible to compute scene flow independently for points on the same surface.