Mirror Image

Mostly AR and Stuff

Augmented reality, enforced locality, geometric hashing

I had discussion with Lester Madden at linkedin MAR group. The thing we discussed was the concept of the locality in the AR. That is, each AR object should be attached to specific location and accessible only from that location.
I’ll try explain it more in depth here.
Augmented graffiti, augmented reality mail/drop boxes and billboards, user-built reality overlays – all of those should be attached to specific location. This locality could be enforced – only local data would be available (filtered into) in the specific location. This locality of data prevent user from sinking in the augmented noise, generated all other the world, and reduce possibility of spam.
For example you can have neighborhood billboard, leave note for the friends in the park and so on. All those AR objects data could be accessed only locally for both read and write – to read billboard and to post a message on it you would have to go to it.
The user should get the data/content only if he is physically present at the specific location. The same way poster/producer of the data or AR object should physically visit each location where it placed.
If locality is enforced, to place note for your friend in the park you have to visit park, and there is no way around it.
Locality could be enforced with location-based encryption. I think this encryption could be made with use of geometric hashing. User scan environment and make 3d registration with his mobile or wearable device. Encryption key is generated by mobile device from the scanned 3d model of the environment.
If user want to get data attached to the location, he access the server, retrieve local data and decrypt them with that key.
In the opposite direction, if user want to attach some object or data to location, mobile device encrypt data with part of the hash key and send other part of the key to server. Before storing data the server do uniqueness check. Nearby data already stored on the server are checked, and the new data allowed in only if there is some distance from new key to keys of all the other stored data. After that new data encrypted with the second part of the key by server and stored.
Each object encrypted by two keys, one of which is server side. Server have no access to content of the data, but have access to the part of the location hash key. That way no two objects or data attached to exactly the same location. Clattering of AR objects could be reduced. More importantly if poster have to physically visit location where he want to place AR object, he should have at least some relation to that location, and he is not some spammer from the other end of the world.
If spammer forge location key without actually visiting the place, that will most probably be non-existing location, and no one will be hit by his data.
That all is of cause is a rough outline of how could enforced locality works. Building robust algorithm for extracting geometric hash could be non-trivial.

1, May, 2009 Posted by | Augmented Reality | , , , , | 8 Comments

Why 3d markerless tracking is difficult for mobile augmented reality

I often hear sentiments from users that they don’t like markers, and they are wondering, why there are so relatively few markerless AR around. First I want to say that there is no excuse for using markers in the static scene with immobile camera, or if desktop computer is used. Brute force methods for tracking like bundle adjustment and fundamental matrix are well developed and used for years and years in the computer vision and photogrammetry. However those methods in their original form could hardly produce acceptable frame rate on the mobile devices. From the other hand marker trackers on mobile devices could be made fast, stable and robust.
So why markers are easy and markerless are not ?
The problem is the structure , or “shape” of the points cloud generated by feature detector of the markerless tracker. The problem with structure is that depth coordinate of the points is not easily calculated. That is even more difficult because camera frame taken from mobile device have narrow baseline – frames taken form position close one to another, so “stereo” depth perception is quite rough. It is called structure from motion problem.
In the case of the marker tracker all feature points of the markers are on the same plane, and that allow to calculate position of the camera (up to constant scale factor) from the single frame. Essentially, if all the points produced by detector are on the same plane, like for example from the pictures lying on the table, the problem of structure from motion goes away. Planar cloud of point is essentially the same as the set of markers – for example any four points could be considered as marker and the same algorithm could apply. Structure from motion problem is why there is no easy step from “planar only” tracker to real 3d markerless tracker.
However not everything is so bad for mobile markerless tracker. If tracking environment is indoor, or cityscape there is a lot of rectangles, parallel lines and other planar structures around. Those could be used as initial approximation for one the of structure from motion algorithm, or/and as substitutes for markers.
Another approach of cause is to find some variation of structure from motion method which is fast and works for mobile. Some variation of bundle adjustment algorithm looks most promising to me.
PS PTAM tracker, which is ported to iPhone, use yet another approach – instead of using bundle adjustment for each frame, bundle adjustment is running in the separate thread asynchronously, and more simple method used for frame to frame tracking.
PPS And the last thing, from 2011:

30, March, 2009 Posted by | Coding AR | , , , , , , , , | 4 Comments

Mobile OS for Augmented Reality

Which platform suit better for mobile AR ? Each has it pluses and minuses. I’m trying to make overall estimation, not only form prototype development pov.

1. iPhone
+ beautiful phone
+! no platform fragmentation
+ application store
+ growing market share
+ 3d accelerator, GPS, accelerometer
+ active developer community
-!! No official camera API for now, direct access to camera require undocumented API
– slow camera on the existing model (better in the next model ?)
– CPU underclocked to 412Mhz on existing model (better in the next model ?)

2. Android
+ Open sourced
+ good CPU for existing model (528Mhz for G1)
+ 3d accelerator, GPS, accelerometer for existing model
+ active developer community
+ application store
+ completely open model for developers available.
-! officially java only (10-100 more slow than native code for numerical tasks), installation of native code app require hack on the consumer model.
– low market penetration for now(will be better?)

3. Symbian
+! Big market share
+ some models have good CPU (up to 600Mhz)
+ some models have fast camera
+ some models have 3d accelerator, GPS, accelerometer and even electronic compass
+ application store coming soon for Nokia models
+ will be open source soon
+ situation with Symbian Signed may improve in the future.
-! platform fragmentation, different OS versions are only partially compatible.
– Symbian signed prevent access to GPS/accelerometer for early versions(S60 FP3) self-signed application
-! For signed app – each binary version should be paid and signed separately, require expensive Publisher ID
– No self-signed application allowed to app store.
– high learning curve
– Market share is shrinking now, eaten by iPhone

4. WinMobile
Not many specific pluses or minuses.
– Small market share

5. Other flavors of Linux – situation is not clear yet.

27, March, 2009 Posted by | Uncategorized | , , , , , | 4 Comments

Another prospective AR device – Samsung i8910 Omnia HD

Also Symbian OS. 600Mhz CPU, 3d accelerator, accelerometer, proximity sensor, GPS, touchscreen. Some kind of hardware image processor seems too, but with closed API, so that is not really useful.
Here are full spec.

23, March, 2009 Posted by | Augmented Reality, Mobile | , , , , | Comments Off on Another prospective AR device – Samsung i8910 Omnia HD

The Register article mention AR Tower Defense

The Register article on Augmented Reality make a mention of AR Tower Defense.

PS. In relation to this article, if smartphone need better display for AR – what mobile AR device need first

19, March, 2009 Posted by | Augmented Reality, Games, Nokia N95 | , , , , , | Comments Off on The Register article mention AR Tower Defense

Tracking planes in the city

In relation to tracking cityscape I did some planar segmentation test. Segmented FAST generated corners with simple 5-points projective invariant.
In some cases 5-point give some rough approximation:
planar segments
In some cases outliers are quite bad – some point have very close projective invariant but still are in diffferent planes.
bad seggment
So simple method not quite work…

19, March, 2009 Posted by | Coding AR, computer vision | , , , , , , , , , | 4 Comments

Tracking cityscape

One of the big problem in image registration/structure from motion/3d tracking is using global information of the image. Feature/blob extraction, like SIFT, SURF or FAST etc using only local information around the point. Region detector like MSER using area information, but MSER is not good at tracking textures, and not quite stable at complex scenes. Edge detection provide some non-local information, but require processing edges. That could be computationally heavy, but looks promising anyway. There are a lot of methods which use global information – all kind of texture segmentation, epitome, snakes/appearance models, but those are computationally heavy and not suitable for mobiles. The question is how to incorporate global information from the image into tracker, and make it with minimal amount of operations. One way is to optimise tracker for specific environment – for example use the property of cityscape, a lot of planar structures and straight lines. Such multiplanar tracker wouldn’t work in the forest or park, but could be a working compromise.

12, March, 2009 Posted by | Coding AR | , , , , , , , , , , , , | Comments Off on Tracking cityscape

Nintendo DSi and Augmented Reality revisited

Returning to my old post about DSi and AR, it’s known now that main DSi CPU is 132 Mhz. That is marginally acceptable for markers based AR games. What kind of performance can be achieved for DSi ?
Here is my old Nokia 6600 demo which run at 107Mhz CPU phone. With DSi 133Mhz CPU+second 33Mhz CPU I’d estimate the same game can run on DSi at 8-10 fps. That’s is not a stellar, but nevertheless playable frame rate. Could be faster with some aggressive optimization or simplifications.

20, February, 2009 Posted by | Augmented Reality, Mobile, mobile games, Nintendo DSi | , , , , | 1 Comment

Future of Augmented Reality social networks :)

From Abstruse Goose

18, February, 2009 Posted by | Augmented Reality | , , , , | 2 Comments

More on the battery life – solar powered phone

I had already written in this blog, the main factor slowing mobile AR development is the battery life. Faster CPU require more energy, and eating device battery very fast.
One way to work around the problem is to use different, task specific architecture for CPU units/coprocessors, thus to get more processing power with the same power consumption.
Another – get more energy. That’s what Samsung did with “”Blue Earth” phone. The phone has full solar panel on its back.
blue erath phone

PS. Some tricks to increase battery life.

14, February, 2009 Posted by | Augmented Reality, Mobile, Nokia N95 | , , | Comments Off on More on the battery life – solar powered phone