Tracking a Face from Camera Texture in a Plane (Question)

Hi… Sorry if I’m asking a lot of questions lately. I really interested and excited in Spark AR. I’m hungry to learn so many new things about it. I have a question: How to make a face mesh attached to the face in a plane not the direct face from the camera? My plane uses camera texture, and i want my face mesh to keep tracking the face in there even when I change the scale, position and rotation of the plane. Please helpp

I’m not totally sure if I understand the question (some pics might help), but I think you can take the position and rotation values from the face tracker and manually plug them into the face mesh inside the plane. The face tracker wouldn’t have the mesh as a child.

When I scale up the plane for zoom effect, it creates a distance between the face on the textured plane to the face mesh. but that happen only when the person is looking at any other direction. when the person look straight to the camera, it match. would you like to check out the project file? so you can see if there is any wrong routing to accomplish it? this is the project file: https://drive.google.com/file/d/1t4XZC26fGevmSRMgx5ol9fRVpIq_XMRs/view?usp=sharing

it zooms in and out (scaled up and down) by tapping the screen. I do that because if i don’t zoom it, then the edges will be visible when camera “moves”. basically what i want to is to replicate the tracking effect from this commercial video: https://www.youtube.com/watch?v=Dd1VIeTMGQs

if that follows an object, i want it to track the face.

why i need face mesh is because i want to have retouching and some textures for the face.

the face tracking from the plane works fine, the problem only with the face mesh.

Why do you need the facemesh for this?
From what I see on the commercial you’d like to zoom in on the face of the user and keep the face as the center of the shot right? You can do this by using the facetracker’s position alonside exponential smoothing to displace your zoomed in rectangle I think.

The issue with the facemesh currently is that when you zoom the cameraTexture, you’re not changing the base from which sparkAR is calculating it’s facetracker, so there’s gonna be a disconnect between the face in that zoomed cameraTexture and the actual facetracker calculating its position based on the unzoomed/default cameraTexture. Had this issue months ago but don’t recall ever solving it at the time unfortunately…

I’ve said that earlier here:


yeah… i thought so too. but i thought there is a workaround. Like maybe take some values using script and dump it to patch editor. like I saw script that says focal distance and things like that, but that stuffs can’t be found in patch editor. I think I just missed a little bit, since the scale, the rotation is already on point. it’s just that the position when in “zoom mode” is a bit off when the face is looking way too left or right, also when tilted. have you look at the project filed that I sent? It’s not a bunch of mess i promise, i organize them, well separated and aligned so there are no messy cables from one nodes to another. It’s literally just new blank project + alpha mask for the face and patch editor.

So I took a look at the project file. Honestly I stumbled upon this use case over a year ago and at the time didn’t solve it. I intended on using it a bit differently though so maybe there’s a chance in your case…
The issue is that the facemesh is located inside of the facetracker, which has it’s own coordinate system. I think this is why you can’t get the results you’re looking for by applying the same coordinate changes to the canvas and the facemesh. I don’t really see how to go around this by default tbh, though i’ll try to take a more in-depth look when I have the time!

@josh_beckwith any ideas on this?

Just spoke to Josh who basically brainstormed a solution within 10 seconds (my pride…). Use render pass to scale the canvas + facetracker. Basically use render pass to flatten your cameraTexture + facemesh setup and apply your distortions to that instead

2 Likes

@Boris, @josh_beckwith IT WORKSSS!!! OMG hell yeah!! Thank you Boris! Thank you Josh!!

But… somehow i still can’t figure out how to do the gimbal/stabilizer effect in this one since i can only transform it using 2d transform in texture transform and that only offer 1 rotation. I could do that using plane since it’s 3D object where it has 3 axis to compensate/negate the movement, but the face mesh is still so stubborn. I mean, i could live without that, since more axis compensation means the boundaries get smaller and smaller that will force me to zoom even further. And no one want super zoomed in to their boogers right? lol

Anyway, I found script “worldtofocal” by Keerotic here: https://github.com/keerotic/arp
But since i’m total noob with scripting, it just doesn’t work somehow. is that script no longer available for spark ar v98? :frowning: If it is, then will it be helpful to accomplish my objective here?

can we ask Josh 10 seconds more to brainstorm? :stuck_out_tongue_winking_eye:

There were huge API changes in terms of scripting with v90, so likely this is a script that dates from before that era, so needs a “promise” update (they updated the Javascript Spark AR runs on to use promises unfortunately for pple like me…)

I’m not entirely sure I get what you mean by gimbal/stabilizer effect, what are you looking to achieve still?

Ahhh I see… Gotta learn scripting now then. Woohooo!!

So it’s like to keep the head straight even when the subject tilt their head. So it would look like the person is still straight, but the the surrounding turns gay. sorry, I mean, the surrounding is the one that rotate around the head. So like the head is the pivot where it will still 90 degree straight. I think i won’t do 90 degree but i definitely find the use of making something as an anchor point. My Idea is to create an AR photographer effect, so the phone stay still, the user will try to find an angle themself then the effect will try to find new interesting angle to shoot. I think that idea is kinda unique and do-able. combined with freeze, so then it will capture moments, and will reveal like 4 photos in the end. (of course it will be a video, user can just screen capture it later if they want to lol.

mmmh, I had a script that did something similar I think, used it to make a texture that is centered on the user’s face at the time. It doesn’t tackle rotation, though there might be a quick way to add that in

Found it again, basically it makes the nose the center of the camera at all times (unfortunately it’s not updated to v90+ either, but still seems to function) : (credits to @kamend for this)

StickyNose.arprojpkg (12.1 KB)

1 Like

I think you’d just need to counter the z rotation, right? Otherwise you’d get some weirdness where the facemesh wouldn’t align with the camera image.

Cool. I’ll take a look at that. thanks Boris!

Yeah… Before using the render pass, that’s the main problem. they just don’t glue together. I tried simple math and logic pipeline but still not even close to look decent. it’s still noticeably misalign.

using render pass works great, but i just don’t know how to counter the two other rotation axis since in 2d transform patch it only has 1 side to side (pitch) rotation. i dont have the counter to negate the yaw and the roll

I’m pretty sure the video stabilization technique only corrects rotation on one axis though.

okay then, i think i really need to learn more about the math and the logic to make it works. thanks for your help josh. :smiley:

Oh, I see the problem. The texture rotation looks skewed because it’s a non-square texture. You’ll have to do some counter-scaling based on rotation. I started on it but it’s not quite done. I hope this helps a bit. It’s correct at 45 degrees, at least :slight_smile:

1 Like

Oh, actually there’s no need for the trig :brain:

StickyNoseRotation.arprojpkg (20.6 KB)

1 Like

omg. is that using script that Boris said earlier? can we do further adjustment in the patch editor along with the script? sorry i’m so lost about script and kinda intimidated by it. But I’ll give it a try. once again, thanks guys. can i call you guys my friend/teacher/mentor? damn… i love his community. I’ll work harder to hopefully catch up with you guys. sorry for still being such a pain in the ass. bless you all. sincerely

2 Likes