Mirror Image

Mostly AR and Stuff

FAST with SURF descriptor

Feature detected with multistage FAST and fitted with SURF descriptors
FAST SURF
Less strict threshold give a lot more correspondences, but also some false positives
FAST SURF

Advertisements

25, January, 2009 - Posted by | Coding AR | , , , , , ,

44 Comments

  1. Hi, I’m working on an AR project for my master thesis. I’m really interested in this application of FAST keypoint detection coupled with SURF descriptors. Could you please point me to some relevant resources for implementing this o existent source code?

    Thanks in advance,
    Keep on the good work!

    Comment by bitslim | 15, October, 2009

  2. SURF source code present in OpenCV 2.0, you can take descriptors code from there. FAST code you can find on the Edward Rosten’s, FAST developer, site:
    http://mi.eng.cam.ac.uk/~er258/work/fast.html
    Be careful if you want try ER-FAST. ER-FAST C code is huge and can easily crash visual studio.

    Comment by mirror2image | 15, October, 2009

  3. Thanks for you answer. I’ll give a look to both.
    So you just apply SURF “description” code to keypoints detected by FAST? Did you try FAST with SIFT based descriptors too?

    Comment by bitslim | 15, October, 2009

  4. I did slight modification to FAST – used it on scale space – downsampled image and applied FAST to patches produced by fist stage. But you probably don’t want to do it – it’s purely for performance optimization for mobile device.
    SURF – it’s not “description”, it’s “descriptor” – string of numbers which are 1-2nd order Haar wavelet moments.
    SURF descriptor I used practically without changes. You may want to fiddle with parameters though – they influence results a lot. SIFT I never tried – it’s more slow then SURF.

    Comment by mirror2image | 15, October, 2009

  5. It doesn’t seem obvious on how to go from a FAST CvPoint keypoint returned by cvCornerFast to a CvSURFPoint which contains a lot more required information such as the laplacian, size, etc. Or do you simply pass in the CvSURFPoint with only the “pt” member set and cvExtractSURF will fill in the rest of the info?

    BTW: Thanks for publishing this work — I hope to share my results in the near future as well.

    -Micah

    Comment by Micah | 29, October, 2009

  6. I’m not using OpenCV here (actually I’m using it for testing only, because intended target is mobile). I just get point detected by FAST and calculate SURF descriptor for it. That way Hessian is not used at all.

    Comment by mirror2image | 29, October, 2009

  7. Hi,
    Is FAST-SURF combination rotation in-variant?

    Comment by Kalai | 29, October, 2009

  8. FAST is rotationally invariant. SURF descriptor could be oriented, which is rotationally invariant (theoretically, in practice not so much) and upright, which is not.

    Comment by mirror2image | 29, October, 2009

  9. hi
    Im using FAST code from Edward Rosten’s website to detect keypoints from the image. The output is a group of detected points. Since im using FAST, i don’t think i need to use the Hessian detector. Now im not sure how to feed it to

    cvExtractSURF(CvArr *image, CvArr *mask, CvSeq **keypoints, CvSeq **dexcriptor, CvMemStorage *storage, CvSURFParams params, int useProvidedKeyPts)

    Do you know how to input the group of points from FAST to CvSeq** ?

    i used the following to create a CvSeq
    CvSeq *kp1 = cvCreateSeq(0, sizeof(cvSeq), sizeof(CvSURFPoint), storage)

    Did you calculate SURF decriptor for each point using cvExtractSURF?

    thanks in advance

    Comment by henry | 31, October, 2009

  10. See comment 6.

    Comment by mirror2image | 31, October, 2009

  11. I’m stuck at the same point – how do you produce SURF/SIFT descriptors from FAST corner points?

    Can anyone help ?

    (I couldn’t find the answer to my question at comment 6)

    Comment by sharo baro | 6, November, 2009

  12. First you detect candidates with FAST. Use some scoring on them – either FAST scoring, or Hessian/Laplacian approximation.
    Use some non-max suppression using score. Or don’t – depend on your strategy. After that apply relevant descriptor to the point.

    ftp://ftp.vision.ee.ethz.ch/publications/articles/eth_biwi_00517.pdf

    http://en.wikipedia.org/wiki/Scale-invariant_feature_transform

    Comment by mirror2image | 6, November, 2009

  13. Hello

    Im not sure how to obtain scale information using FAST? Could you explain how you did it?
    it is important because scale information is used by SURF. Otherwise the tracking performance is restricted.

    thanks

    Comment by henry | 13, November, 2009

  14. FAST is not related to scale space directly. First you have to build scale-space pyramid (possibly with integral images), after that use FAST for each level.
    http://en.wikipedia.org/wiki/Scale_space
    http://en.wikipedia.org/wiki/Integral_image

    Comment by mirror2image | 13, November, 2009

  15. Could you explain more details about overall procedure and how you built the scale-space? Can i just use 1/2 downsampling and gaussian blurring? Then i have pixel locations in different orders. for example, there would be different point location for 500×500 image and 250×250 image.
    Like you said, would it be fine just copying FAST point data to create SURF descriptors? nothing else like dir, hessian, laplacian, size.

    Comment by Derek | 16, November, 2009

  16. Scale space pyramid is the same as used in SURF. It described in detail in SURF paper, and should be present in SURF open source implementation. 1/2 downsampling is a special case of this scale space, but not quite good IMO, better use overlapping 4 pixel cells downscaled by factor 2, the cost is the same as downsampling by 2 because of integral images. Point location of cause would be different – scaled by factor 2.
    I’m not sure I understand last question. You may want to use non-max suppression on FAST, and you can use original FAST non-max suppression, which doesn’t use hessian or laplacian, or you can make anoter, which use hessian or laplacian scoring.

    Comment by mirror2image | 17, November, 2009

  17. Hi,

    could you provide some informations about:

    1-timings using FAST+SURF(calculating SURF descriptors
    only on the interest points returned by FAST,
    for example, on a 320×240 image; Can you do it
    real-time on a, say, 300 mhz mobile device?)
    2-robustness to viewpoint change

    I am searching a fast way to do AR using a texture. It has to be fast cause i need it to run on mobile devices…
    Thx

    Comment by Luca | 10, January, 2010

  18. 1 Didn’t do timing, but I think yes. Daniel Wagner did FAST-SIFT, and SIFT is more slow.
    2 robustness of pure FAST-SURF, without additional consistency checks quite poor. It usually produce around 5% false positives.
    I do mobile markerless AR too. Do you have any demo to see?

    Comment by mirror2image | 10, January, 2010

  19. Hi,

    for now i am only able to do marker-based AR. Visit
    http://sites.google.com/site/fvlibrary/

    Comment by Luca | 10, January, 2010

  20. Hi

    what do you think about Maximally stable extremal regions??
    Have you tried it? Do you think that ths approach is suitable for real-time applications??

    Luca

    Comment by Luca | 11, January, 2010

  21. Yes,
    https://mirror2image.wordpress.com/2009/03/03/region-tracking/
    They are fast, but by itself are not very useful for real time 3d registration IMO – they are too big, if extract corner from them they wouldn’t be stable, and they are only partially robust – they have “good” part with high gradient, but also could have unstable part with low gradient. They could work for object tracking, but not good for 3d registration.

    Comment by mirror2image | 11, January, 2010

  22. hey

    could you explain about what kind of consistency check you applied to fast-surf to make it more stable? (in post 18)

    thanks in advance

    Comment by henry | 12, January, 2010

  23. I am wondering whether FAST is much faster than approximate hessian of SURF detector or not. In Wagner’s paper, he used SIFT and FERN, I don’t know well, but could it be fast enough to use SURF detector(hessian)?

    Comment by Derek | 12, January, 2010

  24. Here’s my real-life demo of object recognition based on SURF running on iPhone 3G:

    http://www.oggmedia.com/moviephone/

    Comment by sharo baro | 12, January, 2010

  25. @henry
    https://mirror2image.wordpress.com/2009/12/11/features-matching-and-geometric-consistency/

    @Derek Most probably FAST faster in any C form (it’s statistically optimized for minimum amount of operations), but may be even more important FAST could be very efficiently assembly-optimized.
    Also FAST more or less rotationally invariant form start, not requiring detecting any dominant direction.
    About Wagner method – it can not acquire data run-tine, data should be precalculated – either FERN built or SIFT PCA-ed off-line. It’s for already processed environment.

    Comment by mirror2image | 12, January, 2010

  26. I’m sorry for bothering you. but i’m still stuck in the building scale-space and using cvExtractSURF(…, size, ..). Maybe you didn’t use opencv for this, so you may not know it. Anyone who did this using opencv? when i build scale-space, i didn’t use integral image because this didn’t make FAST corners, so i just use image pyramid. In this time, i just downsample the image using many different factors. and i made Gaussian blurs to make layers. but after this how can i apply this to parameter (size: Feature size) of cvExtractSURF? Could anyone help me and explain the procedure in detail step by step?

    Comment by Derek | 18, January, 2010

  27. Hi,

    I have built the scale-space on raw images (not integral images) and applied SURF descriptor to all fast corners detected from the pyramid of images, but performance of the match is very poor. Does anyone have any suggestions?

    also could anyone explain about what matching algorithm they used? Im currently just using the simple matching code from the openCV SURF example…

    Comment by kagashi | 1, February, 2010

  28. 2kagashi, I have the exactly same problem as you for quite a long time. i don’t understand how people do this. if you don’t mind why don’t we have some discussion and try to solve this together. can you give me your email address?

    Comment by Derek | 1, February, 2010

  29. sure its kagashi at gmail dot com

    Comment by kagashi | 2, February, 2010

  30. Hello,

    How to detect corner using SURF algorithm?

    thank you.

    Comment by ReaLsHot | 3, June, 2010

  31. Hi mirror2image/Sergey,

    You said in comment #14,

    “FAST is not related to scale space directly. First you have to build scale-space pyramid (possibly with integral images), after that use FAST for each level.”

    What do you mean by using FAST for each level? Did you apply FAST on *integral images*?!

    Comment by The suffocated | 2, July, 2010

  32. Hi!

    I’m currently working on an application that uses opencv on an android phone as my master thesis to create a AR application as well. The thing is though, that for me the extractions of SURF Features takes about a second to run. I have some things I’ll try to fix tomorrow (the size of the image and others) but, I was curious, what results have u gotten and on what kind of mobile devices? I would be more then happy to discuss other things as well,

    Feel free to contact me on musse at phew.se

    Kindest regards,
    Martin

    Comment by Martin | 22, September, 2010

  33. OpenCV implementation of SURF is hopeless. You have to write your own descriptor for real-time mobile. Some implementations can run real-time on hi-end phones, but they usually highly nontrivial, worth master thesis (or more) by itself.

    Comment by mirror2image | 22, September, 2010

  34. Hi,
    Using fast combined with surf descriptors seems a very good solution in AR, there could be some problem that I would like to discuss about:

    1 – You may have to rewrite the matching code (I’m thinking about the opensurf implementation) becouse I’m not sure that checking if the nearest and the second nearest are enough distant one from the other could work.

    2 – Building the image pyramid can lead to loss of performance: have you think to create mutliple descriptors for every points detected by fast increasing the area considered around it?

    Comment by Nicola | 29, October, 2010

  35. Hi, is it possible to use the images provided in this blog post in my background research for my master’s thesis? I wish to show an example of FAST and SURF prior to showing my own work. No comparison will be made between your work and mine.

    Thanks,
    Alex

    Comment by Alex Michael | 11, January, 2011

  36. Sure, but that is very old staff, everyone(me including) moved light years ahead from this. This approach proved to be too simplistic for real-life apps. I can’t talk about latest staff though – work for client etc.

    Comment by mirror2image | 11, January, 2011

  37. Do you know of any site that provides information on these “latest stuff” ?

    Comment by Alex Michael | 11, January, 2011

  38. There is no single site or boook per se, there is a lot of papers 2006-2010. I can outline for you some general directions though: use randomness, non-deterministic sampling, machine learning, use global information (like geometric consistency, time-dependent information etc), adjust matching/descriptor technique to your specific task(use info about screen resolution and predominant orientation if possible etc), use heterogeneous features (generated by different method), if you have some fluency in math look up some modern heavy-hitter mathematical methods like algebraic topology and manifolds for image processing, sparse recovery etc. Just be bold and don’t limit yourself to most well know methods

    Comment by mirror2image | 12, January, 2011

  39. is it possible to improve the performance of face recognition systems by using SURF descriptor along with compressed images?

    Comment by Menila James | 29, January, 2011

  40. I’m not sure I understand what do you mean, but the answer most probably is “no”

    Comment by mirror2image | 29, January, 2011

  41. Hi, in #36 you mentioned that all that is “very old stuff”. You mean local detectors / descriptors in general or especially for mobile devices (concerning performance issues)? The hints you gave in #38 seem reasonable to me but on the other hand (talking about mobile usage) introduce even more performance problems, don’t they? Concerning my thesis on object recognition techniques for mobile devices I’ve also concentrated on local features so far. You really think thats old fashioned? Any comments appreciated

    Comment by BH | 17, February, 2011

  42. Old fashioned I mean only unmodified straightforward SIFT/SURF.

    Comment by mirror2image | 17, February, 2011

  43. Wow, amazing how fast you are :)
    Thanks for your reply. So you think using (modified) local descriptors is still a good way to go on mobile devices (e.g. using available algorithms for simple recognition tasks and modified / combined techniques, see above, for more general object recognition with huge database)?

    Comment by BH | 17, February, 2011

  44. I think it also depends on whether you are working on static images or doing AR-like stuff (e.g. object tracking) where real-time performance is far more important

    Comment by BH | 17, February, 2011


Sorry, the comment form is closed at this time.

%d bloggers like this: