3D Face Rigging

Hi everyone, hope you’re doing fine.

I have an idea for an upcoming project but I’d like to know if what I want is possible - and if so how.

[ Simplifying: ]
I would have a 3D asset similar to the default Spark AR Face Mesh, I would nest it under the face tracker, my question concerns the rigging of the mesh and the different face landmarks; can I attach different parts of the mesh to the landmarks just like the default Face Mesh does? As an example, the user opens their mouth and the mesh would distort according to the position of the lips.

I don’t mind using either patches or scripts.

Note this isn’t going to be a face mesh - I’d just use the face mesh David made - it’s a 3D model that emulates a character so I can’t just use David’s

Any help is appreciated, thanks :heart:

I think this is a good question to answer for @Boris <3

1 Like

haha yeah it is quite possible indeed ^^
Surprisingly developed this ages ago but only ever used it on one filter (“Hipster Santa” on our @popul_ar.experience account, you’ll notice if you use it that the 3d beard follows the face’s deformation).

I don’t know how comfortable you are with 3d programs but you really just need to import the default facemesh and project the UV unwrapping with the red anchor points, provided in the official face assets pack. From there you recreate each bone and position them on the red points in 3d.
Once you have the bones, rig your 3d object to them. From there you win given your coding skills, you just need to write the script that binds each of your bones to the corresponding bone from the scripting object reference doc. I suggest renaming your bones with the proper names in the 3d scene for ease of understanding.

Oooor, you can wait until next Thursday and it’ll be part of the content up for grabs on our platform : SUGAR :stuck_out_tongue:

3 Likes

Haha thanks, I’m not really confortable but this project is really 3D heavy anyways so I’m learning a lot about 3d objects and programs :grin:.
I’ll try to see if I can pull it off and if I can’t I’ll have yet another reason to be exited about SUGAR

1 Like

Awesome, best of luck! Don’t hesitate to ask me for help if you get hard stuck, your contributions script-wise on other posts have not gone unnoticed, love the mindset :thumbsup:

2 Likes

Thank you so much! That’s pretty much the best you can say to someone like me :heart:. Okey I’ll how it goes.

Btw I haven’t forgotten about some updates to my script and a PR to some code by lasse I mentioned the other day, I just have to turn in a few things from school first :sweat_smile:

2 Likes

I’ve recently built a maxed out version of what Boris is talking about but with about 60 bones added to the 30 something that spark gives you access to and calls ‘lanmarks’. Turns out you can place your own landmarks using face.point(x,y). I can send you the 3d file and the spark script. that puts the skeleton in action. The rigging on your end shouldn’t take more that a few clicks

2 Likes

@Boris I remember your headache when you were trying to rig that beard properly aha

1 Like

Thank you so much! I didn’t know about that function, I don’t think it’s documented. Yes I’d be very grateful if you do, I’ll use it as a reference of how to set things up. My email: pietravallotomas@icloud.com
I am finishing my current project and this one is probably the next big one I’ll attempt, thank you to all of you!

https://we.tl/t-ZgKc27jHSY Here’s the stuff. I think there’s still a few assets from my project you can get rid of. Just import your mesh and scale/place it correctly on the facemesh. Then remove the facemesh and parent your mesh to the armature using a natural weight distribution :slight_smile:

3 Likes

Thank you so much!! This will make it sooooo much easier. When I’m done I’ll take a deep dive into 3d to really understand all of this though from your explanations I think I got most of it. Just out of curiosity, face.point(x,y), are the x&y the position on the UV of the default Spark face mesh?

Thanks again to all of you! And again omg this face.point is amazing, thank you Victor for sharing it :heart:

You’re welcome :)))) Yes that’s absolutely it. And those “landmarks” points accessible through patch and script are just some custom uv points handpicked by the spark team. Have fun :slight_smile:

2 Likes

yeah hell on earth, but only because of rigging in Maya. Spark wouldn’t accept the skeleton for some reason, kept importing it as null groups… Works better now I feel then back then at least