Detect face when mouth is covered

Hello there,
Im a design student who’s very new to Spark AR and had a question regarding it.
I wanted to know if it is possible to create a filter that only appears when you are wearing a face mask :mask: . I have seen these types of filters on Snapchat. But I don’t know how to achieve something like that on Spark Ar. I thought of using target tracking on my mask but since the design will be on a curved surface it might not be effective and work properly.
It would be really great if you could suggest a way to achieve it.
Thank you :grinning:

1 Like

I’ve seen the question floating on the sparkAR community and believe Noland Chaliha has taken a crack at it however I haven’t yet.
My hack of a way to approach this would be to use the facetracker. With it you sample colors on several points on the face where the mask wouldn’t cover, and compare that to the color of a point you take at the mouth level, where the mask would be (so theoretically if both colors are radically different, there’s a mask on the face)

2 Likes

Yeah, this is a difficult one without the capability of making our own ML models like on snapchat. You could do some color comparisons, but it’s worth noting that those results can only be used in shaders since they originate from a shader signal. You won’t be able to trigger animations, visibility, or anything like that - only affect color.

1 Like

Yes what Boris describes is (probably) the only approach. Noland did it with ML so it works wonderfully but it’s only on Snapchat. If you are comfortable with math you can use shader signals to drive animations - it gets messy fast - I did it once for an effect that never saw the light of day

1 Like