Optical Flow on Object

Hi I was wondering how to start adding optical flow into 3D objects. There’s a face filter I would like to create using the patch from the SUGAR store :slight_smile:

Thank you

1 Like

You can try many interesting stuff using optical flow that needed a recursive or gradual changes based on frame difference. The nature of optical flow is to detect/compare changes between frames over time and then it gives us the motion vector in pixel color like flow map or normal map. I recon by using 3d object from scene render pass, it would give us a cleaner result because unlike the camera input that can introduce noise due to dependency of how good the camera and the lighting. You can try to build some fluid simulation, combined with pixel sorting technique to create data mosh effect, or even getting more experimental to apply it as vertex displacement for other 3d object.

1 Like

Exactly, you can use a scene render pass to get a texture of just the objects you want to apply the optical flow shader to. You’ll use that as the input instead of the camera texture

If I haven’t misunderstood what you mean, you probably want to know how to make a 3D model animation based on optical flow technology and then move it by implementing this technology in Spark AR. I think this video will help you

And in my opinion,

this effect is also made with this method, of course, I may be wrong and this effect was made by a shader, and this number of particles, if they were designed in a shader, should probably cause a drop in fps, but when running this effect, this It doesn’t happen, so I guess maybe it’s a 3D model

no that is completely different technique, but the concept of reading pixel color as a vector/direction over time is the same because it is the basic of computer graphics. VAT doesn’t compare any frames, it simply read a texture map to displace/transform vertices. in a super simplified version, just imagine each pixel represent a transform of each vertex per frame. imagine the texture as a spreadsheet that contains an array of position, scale and rotation, the column represent the next position scale and rotation in the next frame. by reading/scanning/scrolling the uv, then we can make it animate, by using filtering it can then interpolate the changes smoother rather than steppy/sudden jump.

Hey Everyone.

Holiday spirit had a whole of me so I’ve been offline. So I’ll be going through all the feedback and see if I can send you guys what I’ve made till then.

Hope your holiday season has been great - there i go again.