How can I use segmentation with retouching and deformation?

Spark AR Questions
, ,
#1

When I turn add segmentation, the retouching disappears! Is it possible to use both at the same time? What about deformation and retouching?

#2

yeah, i believe atm they are kindda mutually exclussive.
with my dev account I can do all that via different shaders, in a standard account however, its probably harder. deformation will return a sort of normalmappy render, which you have then to multiply the camera image in order to move and stretch pixels accordingly, which to be honest I did not made/own the math, while the beauty pass is pretty much a gaussian blur, for which you can also make a mask pass rendering a painted facemesh to filter out the effect on eyes and mouth areas…

you just use the beauty result as camera info for the deformation pass

luckily i have not yet needed to remake the deformation math in patches, but it should be kind of doable to translate… or just wait as its probably a high priority dev objective for the spark team

2 Likes
#3

Thanks for the response Martin! I have tried this since posting and surprisingly didn’t have any issues. The only thing to note is that you will need multiple face meshes since the meshes can only use one material each. Other than that, it went flawlessly. I actually ended up using 5 face meshes to apply various effects - makeup, deformation, retouching, teeth whitening, eye sharpening - I went a little overboard :slight_smile:

I did try making a custom retouch with renderpass, since you can’t use both of those simultaneously. I’ll have to come back to it, but my initial approach of just painting a mask didn’t work flawlessly. It would cover up glasses, so I think there might need to be a bit of chromakey masking, in addition.

1 Like
#4

oh I think I am using also a custom shader to do the normalmappy thing as well… (a custom facewarpmaterial, and then I apply a custom warper material)
in v95.1 when I try using renderpass + deformation, still shoots the “choose which” message
not my design… so I cant share the shaders

btw, I never added teeth and eye treatment… now I think I really should review that.