FAST with SURF descriptor
Feature detected with multistage FAST and fitted with SURF descriptors
Less strict threshold give a lot more correspondences, but also some false positives
44 Comments
Sorry, the comment form is closed at this time.
Feature detected with multistage FAST and fitted with SURF descriptors
Less strict threshold give a lot more correspondences, but also some false positives
Sorry, the comment form is closed at this time.
Hi, I’m working on an AR project for my master thesis. I’m really interested in this application of FAST keypoint detection coupled with SURF descriptors. Could you please point me to some relevant resources for implementing this o existent source code?
Thanks in advance,
Keep on the good work!
SURF source code present in OpenCV 2.0, you can take descriptors code from there. FAST code you can find on the Edward Rosten’s, FAST developer, site:
http://mi.eng.cam.ac.uk/~er258/work/fast.html
Be careful if you want try ER-FAST. ER-FAST C code is huge and can easily crash visual studio.
Thanks for you answer. I’ll give a look to both.
So you just apply SURF “description” code to keypoints detected by FAST? Did you try FAST with SIFT based descriptors too?
I did slight modification to FAST – used it on scale space – downsampled image and applied FAST to patches produced by fist stage. But you probably don’t want to do it – it’s purely for performance optimization for mobile device.
SURF – it’s not “description”, it’s “descriptor” – string of numbers which are 1-2nd order Haar wavelet moments.
SURF descriptor I used practically without changes. You may want to fiddle with parameters though – they influence results a lot. SIFT I never tried – it’s more slow then SURF.
It doesn’t seem obvious on how to go from a FAST CvPoint keypoint returned by cvCornerFast to a CvSURFPoint which contains a lot more required information such as the laplacian, size, etc. Or do you simply pass in the CvSURFPoint with only the “pt” member set and cvExtractSURF will fill in the rest of the info?
BTW: Thanks for publishing this work — I hope to share my results in the near future as well.
-Micah
I’m not using OpenCV here (actually I’m using it for testing only, because intended target is mobile). I just get point detected by FAST and calculate SURF descriptor for it. That way Hessian is not used at all.
Hi,
Is FAST-SURF combination rotation in-variant?
FAST is rotationally invariant. SURF descriptor could be oriented, which is rotationally invariant (theoretically, in practice not so much) and upright, which is not.
hi
Im using FAST code from Edward Rosten’s website to detect keypoints from the image. The output is a group of detected points. Since im using FAST, i don’t think i need to use the Hessian detector. Now im not sure how to feed it to
cvExtractSURF(CvArr *image, CvArr *mask, CvSeq **keypoints, CvSeq **dexcriptor, CvMemStorage *storage, CvSURFParams params, int useProvidedKeyPts)
Do you know how to input the group of points from FAST to CvSeq** ?
i used the following to create a CvSeq
CvSeq *kp1 = cvCreateSeq(0, sizeof(cvSeq), sizeof(CvSURFPoint), storage)
Did you calculate SURF decriptor for each point using cvExtractSURF?
thanks in advance
See comment 6.
I’m stuck at the same point – how do you produce SURF/SIFT descriptors from FAST corner points?
Can anyone help ?
(I couldn’t find the answer to my question at comment 6)
First you detect candidates with FAST. Use some scoring on them – either FAST scoring, or Hessian/Laplacian approximation.
Use some non-max suppression using score. Or don’t – depend on your strategy. After that apply relevant descriptor to the point.
ftp://ftp.vision.ee.ethz.ch/publications/articles/eth_biwi_00517.pdf
http://en.wikipedia.org/wiki/Scale-invariant_feature_transform
Hello
Im not sure how to obtain scale information using FAST? Could you explain how you did it?
it is important because scale information is used by SURF. Otherwise the tracking performance is restricted.
thanks
FAST is not related to scale space directly. First you have to build scale-space pyramid (possibly with integral images), after that use FAST for each level.
http://en.wikipedia.org/wiki/Scale_space
http://en.wikipedia.org/wiki/Integral_image
Could you explain more details about overall procedure and how you built the scale-space? Can i just use 1/2 downsampling and gaussian blurring? Then i have pixel locations in different orders. for example, there would be different point location for 500×500 image and 250×250 image.
Like you said, would it be fine just copying FAST point data to create SURF descriptors? nothing else like dir, hessian, laplacian, size.
Scale space pyramid is the same as used in SURF. It described in detail in SURF paper, and should be present in SURF open source implementation. 1/2 downsampling is a special case of this scale space, but not quite good IMO, better use overlapping 4 pixel cells downscaled by factor 2, the cost is the same as downsampling by 2 because of integral images. Point location of cause would be different – scaled by factor 2.
I’m not sure I understand last question. You may want to use non-max suppression on FAST, and you can use original FAST non-max suppression, which doesn’t use hessian or laplacian, or you can make anoter, which use hessian or laplacian scoring.
Hi,
could you provide some informations about:
1-timings using FAST+SURF(calculating SURF descriptors
only on the interest points returned by FAST,
for example, on a 320×240 image; Can you do it
real-time on a, say, 300 mhz mobile device?)
2-robustness to viewpoint change
I am searching a fast way to do AR using a texture. It has to be fast cause i need it to run on mobile devices…
Thx
1 Didn’t do timing, but I think yes. Daniel Wagner did FAST-SIFT, and SIFT is more slow.
2 robustness of pure FAST-SURF, without additional consistency checks quite poor. It usually produce around 5% false positives.
I do mobile markerless AR too. Do you have any demo to see?
Hi,
for now i am only able to do marker-based AR. Visit
http://sites.google.com/site/fvlibrary/
Hi
what do you think about Maximally stable extremal regions??
Have you tried it? Do you think that ths approach is suitable for real-time applications??
Luca
Yes,
http://mirror2image.wordpress.com/2009/03/03/region-tracking/
They are fast, but by itself are not very useful for real time 3d registration IMO – they are too big, if extract corner from them they wouldn’t be stable, and they are only partially robust – they have “good” part with high gradient, but also could have unstable part with low gradient. They could work for object tracking, but not good for 3d registration.
hey
could you explain about what kind of consistency check you applied to fast-surf to make it more stable? (in post 18)
thanks in advance
I am wondering whether FAST is much faster than approximate hessian of SURF detector or not. In Wagner’s paper, he used SIFT and FERN, I don’t know well, but could it be fast enough to use SURF detector(hessian)?
Here’s my real-life demo of object recognition based on SURF running on iPhone 3G:
http://www.oggmedia.com/moviephone/
@henry
http://mirror2image.wordpress.com/2009/12/11/features-matching-and-geometric-consistency/
@Derek Most probably FAST faster in any C form (it’s statistically optimized for minimum amount of operations), but may be even more important FAST could be very efficiently assembly-optimized.
Also FAST more or less rotationally invariant form start, not requiring detecting any dominant direction.
About Wagner method – it can not acquire data run-tine, data should be precalculated – either FERN built or SIFT PCA-ed off-line. It’s for already processed environment.
I’m sorry for bothering you. but i’m still stuck in the building scale-space and using cvExtractSURF(…, size, ..). Maybe you didn’t use opencv for this, so you may not know it. Anyone who did this using opencv? when i build scale-space, i didn’t use integral image because this didn’t make FAST corners, so i just use image pyramid. In this time, i just downsample the image using many different factors. and i made Gaussian blurs to make layers. but after this how can i apply this to parameter (size: Feature size) of cvExtractSURF? Could anyone help me and explain the procedure in detail step by step?
Hi,
I have built the scale-space on raw images (not integral images) and applied SURF descriptor to all fast corners detected from the pyramid of images, but performance of the match is very poor. Does anyone have any suggestions?
also could anyone explain about what matching algorithm they used? Im currently just using the simple matching code from the openCV SURF example…
2kagashi, I have the exactly same problem as you for quite a long time. i don’t understand how people do this. if you don’t mind why don’t we have some discussion and try to solve this together. can you give me your email address?
sure its kagashi at gmail dot com
Hello,
How to detect corner using SURF algorithm?
thank you.
Hi mirror2image/Sergey,
You said in comment #14,
“FAST is not related to scale space directly. First you have to build scale-space pyramid (possibly with integral images), after that use FAST for each level.”
What do you mean by using FAST for each level? Did you apply FAST on *integral images*?!
Hi!
I’m currently working on an application that uses opencv on an android phone as my master thesis to create a AR application as well. The thing is though, that for me the extractions of SURF Features takes about a second to run. I have some things I’ll try to fix tomorrow (the size of the image and others) but, I was curious, what results have u gotten and on what kind of mobile devices? I would be more then happy to discuss other things as well,
Feel free to contact me on musse at phew.se
Kindest regards,
Martin
OpenCV implementation of SURF is hopeless. You have to write your own descriptor for real-time mobile. Some implementations can run real-time on hi-end phones, but they usually highly nontrivial, worth master thesis (or more) by itself.
Hi,
Using fast combined with surf descriptors seems a very good solution in AR, there could be some problem that I would like to discuss about:
1 – You may have to rewrite the matching code (I’m thinking about the opensurf implementation) becouse I’m not sure that checking if the nearest and the second nearest are enough distant one from the other could work.
2 – Building the image pyramid can lead to loss of performance: have you think to create mutliple descriptors for every points detected by fast increasing the area considered around it?
Hi, is it possible to use the images provided in this blog post in my background research for my master’s thesis? I wish to show an example of FAST and SURF prior to showing my own work. No comparison will be made between your work and mine.
Thanks,
Alex
Sure, but that is very old staff, everyone(me including) moved light years ahead from this. This approach proved to be too simplistic for real-life apps. I can’t talk about latest staff though – work for client etc.
Do you know of any site that provides information on these “latest stuff” ?
There is no single site or boook per se, there is a lot of papers 2006-2010. I can outline for you some general directions though: use randomness, non-deterministic sampling, machine learning, use global information (like geometric consistency, time-dependent information etc), adjust matching/descriptor technique to your specific task(use info about screen resolution and predominant orientation if possible etc), use heterogeneous features (generated by different method), if you have some fluency in math look up some modern heavy-hitter mathematical methods like algebraic topology and manifolds for image processing, sparse recovery etc. Just be bold and don’t limit yourself to most well know methods
is it possible to improve the performance of face recognition systems by using SURF descriptor along with compressed images?
I’m not sure I understand what do you mean, but the answer most probably is “no”
Hi, in #36 you mentioned that all that is “very old stuff”. You mean local detectors / descriptors in general or especially for mobile devices (concerning performance issues)? The hints you gave in #38 seem reasonable to me but on the other hand (talking about mobile usage) introduce even more performance problems, don’t they? Concerning my thesis on object recognition techniques for mobile devices I’ve also concentrated on local features so far. You really think thats old fashioned? Any comments appreciated
Old fashioned I mean only unmodified straightforward SIFT/SURF.
Wow, amazing how fast you are
Thanks for your reply. So you think using (modified) local descriptors is still a good way to go on mobile devices (e.g. using available algorithms for simple recognition tasks and modified / combined techniques, see above, for more general object recognition with huge database)?
I think it also depends on whether you are working on static images or doing AR-like stuff (e.g. object tracking) where real-time performance is far more important