Availability of technology

From xkcd

“A technology that is ’20 years away’ will be 20 years away indefinitely.”

Features matching and geometric consistency.

Here I want to talk about matching in image registration. We are doing registration in 3D or 2D, and using feature points for that. Next stage after extraction of feature points from the image is finding corresponding points in two(or more) images. Usually it’s done with descriptors, like SIFT, SURF, DAISY etc. Sometimes randomized trees are used for it. Whatever methods is used it usually has around .5% of false positives. False positives create outliers in registration algorithm. That is not a big problem in planar trackers or model/marker trackers. It could be a problem for Structure From Motion though. If CPU power is not limited the problem is not very serious. Heavy-duty algorithms like full-sequence bundle adjustment and RANSAC cope with outliers pretty well. However even for high-end mobile phones such algorithms are problematic. Some tricks can help – Georg Klein put full-sequence bundle adjustment into separate thread on PTAM tracker to run asynchronously, but I’m trying to do local, 2-4 frames bundle adjustment here. The problem of false positives is especially difficult for images of patterned environment, where some image parts are similar or repeated.
Here mismatched correspondence marked with blue line (points 15-28).

As you can see it’s not easy for any descriptor to tell the difference between points 13(correct) and 15(wrong) on the left image – their neighborhood is practically the same:


Such situations could easily happen not only indoor, but also in cityscape, industrial, and others regular environments.
One solution for such cases is to increase descriptor radius, to process a bigger patch around the point, but that would create problems of its own, for example too much false negatives.
Other approach is to use geometric consistency of the image points positions.
There are at least two ways to do it.
One is to consider displacements of corresponding points between frames. Here is example from paper by Kanazawa et al “Robast Image Matching Preserving Global Geometric Consistency”

This method first gathering local displacement statistic around each points, filter out outliers and and apply smoothing filter. Here are original matches, matches after applying consistency check and matches after applying smoothing filter.

However this method works best for dense, regular sets of feature points. For small, sparse set of points it does not improving situation much.
Here is a second approach. Build graph out of feature points for each frame.

Local topological structure of the two graphs is different because of false positives. It’s easy to find graph vertices/edges which cause inconsistency – edges marked blue.They can be found for example by signs of crossproducts between edges. After offending vertices found they are removed:

There are different ways to build graph out of feature points. Simplest is nearest neighbors, but may be Delaney triangulation or DSP can do better.

Google Googles, QR codes, localization

There are two recent developments related to Augmented Reality and Google – Google Goggles and Google integrating QR codes into Google Maps. While I was talking on twitter with @noazark the question arise about Google Google not doing real-time localization of the user, thus not being a “real” AR.
Here come QR codes. QR code are extremely easy to recognize in the camera image, and their square shape allow for fast calculation of camera position relatively to QR. In fact each QR code include three fiduciary markers:

And well known marker-tracking technique easily applied to them. Marker tracking could be augmented (pan intended:) by planar tracking of the corners of the pattern itself. That allow for attaching virtual 3d objects/animations to QR codes, but there is more in it. As QR code contain more than 4k of data, exact GPS coordinate, pattern orientation and its’ size could be encoded in the pattern. That way mobile phone seeing the code can easily calculate it’s exact 3d coordinate and orientation, not only relatively to QR, but absolute.
More of it – QR code can have coordinates of nearby QR codes, creating kind of localization grid, which can point user to any location covered by that grid with arrow on the screen of the phone.
Now to markerless tracking – QR code can be used to jump-start markerless tracker and assist it with error-correction(drift compensation), especially mentioned grid of the codes. That is especially relevant to markereless trackers which use planar structures and straight edges.
Now there is one problem here – white QR code is easy to segment out of dark background. But on white background it not so easy to recognize, and embedded fiduciary markers will not be seen form afar. Here is suggestion – make thick black frame around the QR, and make it part of the extended standard. This square shape would be easy to recognize, even if it’s only couple of dozen of pixel in diameter. With incremental tracking phone will be able to track it(after initial close up) even if moved quite far from the QR. If this square frame is part of the standard, always having the same relative size, it could be used for distance estimation.
Now combine it with Google Goggles real time and you have functional AR with 3d registration.

Compressive Sensing and Computer Vision

Thanks to Igor Carron for pointing out this video lecture
Compressive Sensing for Computer Vision: Hype vs Hope
It start with comprehensible explanation of what compressive sensing is about (BTW wiki article on compressive sensing is wholly inadequate).
Basically it’s about imagining the lower-dimensional signal(image) as projection by rectangular matrix from mostly zero high-dimensional vector. It happens that this sparse high-dimensional vector can be restored if the matrix is almoste orthonormal (Restricted Isometry Property). Discrete Fourier Transform and random matrices have that property.
This sparse vector could be considered as classification space for original signal. So application of Compressive Sensing to Computer Vision is mostly about classification or recognition. As methods used by CS are convex and linear programming those are not run-time methods, and would not help much in real-time tracking. There is CS-inspired advise at the end of the lecture, about trying to replace norm optimization with norm. That could be actually helpful in some cases. If approximated as iteratively reweighted it’s essentially the same as robustification of least square method.