Hey Lab members
Recently bought the Flow kit and I am struggling to achieve optical flow for hand and body segmentation.
Not sure if I can get a guide to get this right, but any help would be great.
Please see media for what I wanna do
Can you show some of what you’ve tried so far? My first thought is to use the segmented body texture in the optical flow patch. I don’t remember offhand if OF does anything with the alpha channel, but you could try putting it on a black or white matte.
Here is what I have so far.
Used a smoke tutorial I’ve seen on medium (How to create a smoke effect with render passes in Spark AR Studio | Smoke_tutorial) thinking by playing and/or adding the Flowkit that I’ll achieve what I am looking for. Having a problem picking up the Camera Texture for the Background (Bg) and User rectangles. Basically trying to make BG & User Black & White with the optical flow or movements by the user in colour. I am soooo lost now cause it’s been a lot of trial and error.
1 Like
It looks like optical flow doesn’t really know what to do with the alpha channel, so you will need to massage it a little bit by pre-multiplying the alpha into the RGB channels.
In general, adding some blur patches helps control the smoothness of the effect. I added one in the optical flow step AND the normal flow step.
Here’s the logic for the overlay that I came up with (the top section).
And here’s the whole graph in case you want to pan around.
I think that should be a good starting point, but I’m sure you’ll need to spend some time tweaking variables to get the exact look you want. I hope this helps!
2 Likes
Wow this is getting right where I’d like it to be. Thank you Josh for the assist.
Going to be playing around with the values to get what I like. Really appreciate it. You helped me understand shaders and renderpass better. I used the Blur patch as I didnt have a gaussian blur, will try figure it out if it can make the effect better.
Now I have been playing around by adding a LUT patch to make the background BW. but it keeps making the whole scene BW.
Not sure if there is a specific patch that can do that or creating a canva would be a good idea. Your input is Godsend.
1 Like
We are overtuning the edge softness on the segmentation processor. If you do that, it will bleed out to the whole texture and make it partially transparent. Of course, this could work depending on the effect you are going for, but I wanted to point it out. I should probably change that in the template. It should really cap the edge softness at 1, but it doesn’t.
Seems like applying the grayscale background should be done right before the distortion step. You can swap out the luminance/swizzle with a lut if you want to.
I updated the flow kit project to use the new version of optical flow (with alpha/segmentation support). You can download the update from this page Flow Kit • Sugar – Qualitative asset store for Social AR creators
2 Likes
Hi Josh
Got the updated kit last night and will play around and let you know how it goes.
You are amazing
1 Like