Zoom in to face with segmentation

Hi, so I finally have time to try out LS.
I’m still super new and not familiar with the LS workflow, so please bear with me. :smiley:

To make it a bit clearer about what I’m trying to do, let me explain it in Spark AR workflow.
So, what I wanna do:

  • set 2 rectangle, 1 for the background, and 1 for person segmentation
  • set 1 text where it going to display the username of the user
  • when user tap the screen, these things happen:
  1. The camera freezes (optional), and starts to zoom in 1,5x to the user face or just to the center of the screen.
  2. The text appears from outside of the screen to the bottom center of the screen.
  • when the user tap it again these things happen:
  1. The text fade out, and then go back to its default position outside of the screen.
  2. The camera unfreeze (optional), and starts to zoom out to default.

I know how to do it in Spark AR, but how to do it in LS?

In Spark, we can do it by

  1. freeze the device output using mix delay frame for the freeze frame technique on tap,
  2. get the 2d screen position of the face, and pinlastvalue on tap, and use it as the pivot of the 2D transform. and use the same screen tap to start animation to scale the texture up from 1 to 1.5 to achieve the zoom in 1.5x to face.
  3. use 2D text, put it outside of the screen and on screen tap, it animates from that 2D position to the bottom center of the screen with some math based on screen size/scale minus the text size, minus the bottom inset, minus some offset for fine adjustment.
  4. set all of the above on tap trigger from screen tap, switch, pulse “turn on” output
  5. do the reverse sequence when he tap trigger is set to “turn off” output

but how to do it in LS?

Thanks!

U can scale screen images just like in Spark :wink:
To animate this thing you need a tween script with value and change the float of texture size inside of material parameters

Here s how to control parameters by tween script

1 Like

To freeze the camera, you can use the copyFrame method on the device camera texture.

https://lensstudio.snapchat.com/api/classes/Texture/

You’ll find that you will need to dig through API documentation much more in LS, compared to spark. Every project I’ve done in LS has required me to write at least some script, so my advice is to embrace the fact that you will need to write custom scripts to accomplish what you need to do.

A good place to start digging is the behavior script. You can see how they do things in there, then apply the same logic in your own scripts.

https://lensstudio.snapchat.com/guides/scripting/helper-scripts/behavior/

As far as tweening in reverse, there’s no built-in method for that. I’m not sure why… but the workaround is just to create a second tween script with the start and end values swapped. Otherwise, you can do some tweens in script which is a bit more flexible. Here’s an example:

var introAnimation, outroAnimation
script.transitionIn = function(direction){
    print('transitionIn ' + script.getSceneObject().name)

    var endPosition = headTransform.getWorldPosition()
    var startPosition = new vec3(moveToX * direction, endPosition.y, endPosition.z)
    
    if(outroAnimation) outroAnimation.stop()
    
    introAnimation = new TWEEN.Tween({t: 0})
        .to({t: 1}, transitionDuration)
        .easing(TWEEN.Easing.Quintic.Out)
       	.onUpdate(function(obj) {
            endPosition = headTransform.getWorldPosition()
            var newPosition = vec3.lerp(startPosition, endPosition, obj.t)
            transform.setWorldPosition(newPosition)
        })
        .onComplete(function(){
            script.enableTracking()
        })
        .start()
}

You can check the tweenjs docs for more info GitHub - tweenjs/tween.js: JavaScript/TypeScript animation engine

2 Likes

Thanks Josh!
Do you know how to grab the face landmark so the camera is tracking that landmark? So it’s kinda like the sticky nose project in spark ar? I’m trying to figure out how to make camera tracks some point in 3d and 2d space on LS.

Nevermind, just found the super easy solution for this in here:


Thank god they provide this library. so many amazing stuffs to learn and explore there.

1 Like

haha! Yes! You need always watch assets lib!
There are a ton of great stuff

1 Like

Yeah exactly! There’s a ton of good stuff in there and I’m constantly finding new stuff to learn from. Sorry it took me a while to respond - just got back to the computer after a long time.

Hi again guys. so… I have another problem here. when our face is getting closer to the edge of the screen, I want it to gradually reduce the zoom cuz if not, then the edge of the screen is going to show up. anyone knows how to do it in lens studio?

not sure I get what you want to achieve.