To filter or not to filter

Time-domain smoothing filter is somehow controversial question in image registration. The video is inherently smooth, necessity of time-domain smoothing is usually sign of instability of image registration algorithm. Taking average for several frames is especially notorious. Ideally good, stable algorithm don’t need any smoothing in time-domain. However what to do if there is small jittering in otherwise stable and fast real-time algorithm. Just couple of pixel in amplitude, it’s not noticeable on the big virtual objects, but on small, especially stretched or linear objects it’s quite noticeable. I have found that at least in my case, simple, several frames, smoothing filter remove jittering and is not causing any visible alignment artifacts.