How to: Screen Tap controls object's 3D Position that always facing a "target" on screen

Hi, can someone help me build this 3D-2D projection mechanism?
(I’d love a patch editor method, but if there has to be a script, i’d prefer that script simply to get and then pass the value to the patch editor. Like in the StickyNose project a long while ago)

The mechanism:
1st, tap, hold and slide somewhere on the screen to place the source (position A),
2nd, tap/tap hold and slide again to choose the target (position B).

The idea is to create a trajectory/aiming direction from source position to the target position. I think this could be useful for like maybe a game filter, or even like placing a light source. Imagine like a flashlight in position A, aiming to position B.

The Source
z is locked (somewhere between camera and the screen), so when the user tap and slide the screen, it will only change the x and y position of the source object.

The source object (light/arrow/bullet/ball) should always face the target on position B (on the screen) from the second tap.

The Target
On the screen. just like a rectangle from canvas.

No rotation. It acts just like a point even if in the user interface it’s like a 2d circle or crosshair.

I’ve been messing around with this problem for hours but the coordinate and projection calculation is always F’d up. lol

Also like the placement in the scene panel hierarchy is driving me insane. Should I put the source as a child of the camera? focal distance? device? or even outside of the device? and then also the target is on 2d space but should be in 3d space??? omg. please help. :sob:

This is what I got so far.
I’ve managed to get the screen pan input to position the source in x and y paralel to the screen and the z is locked.
I’ve also managed to make the target following the pan as well.

This is how i translate the 2d position from screen pan to 3D position of the source (in this example, directional light):

Now I have to figure out:

  1. how to differentiate screen pan input. Like when to position the source for the first tap, and then the target. For now, both the source object (directional light) and the target (red circle plane) are moving together.
  2. how to make the source (directional light) always rotates to face/track the target.

Still need help to accomplish this. :smiley:

1 Like

Mission Accomplished!!!
Here is the result:

Project Setup:

The script that I use to make the source is always tracking the target comes from:

Big thanks to GOWAAA for the awesome article and Adam Ferriss for the original script setup.

The Scene Hierarchy:
From what I read in the article, the script, and Adam’s demo project, it seems like lookat API needs a parent and children relationship for it to work. So that’s why the source (spotLight0) and the target plane (Target Guide) are a child of null object. The Script control the rotation of the source’s Null (followNull) to always face the targetNull.
The targetNull doesn’t have to be there, we can even put it as a child of the faceTracker so the source will always aim to the face or face landmarks. Pretty cool stuff.

This is the script:
(I commented the random animation, cuz I don’t need it. :rofl: )

const Scene = require("Scene");
const R = require("Reactive");
const Time = require("Time");


 // Finds an element in the scene
// - Parameters:
//      e: The object to find
//const find = e => S.root.find(e); *NOT NEEDED FOR V85*

// Gets the position of an object as an R.point
// - Parameters:
//      e: The object to get the transform from
const getPosition = e => R.point(e.transform.x, e.transform.y, e.transform.z);

// Sets the rotation based on a transform
// - Parameters:
//      e: The object to rotate
//      p: The transform to use
const setRotation = (e, p) => {
 e.transform.rotationX = p.rotationX;
 e.transform.rotationY = p.rotationY;
 e.transform.rotationZ = p.rotationZ;

// Look at utility function.
// Because of reactive stupidness, we can't actually apply the lookat directly to the looker itself
// We get around this by nesting the looker object inside a null with no values applied to it's transform
// - Parameters:
//      _target: The object in the scene you want to face
//      _lookerParent: The parent object of the object you want to rotate. Should have no transform applied to it
//      _looker: The object that you want to rotate towards the target
const lookAt = (_target, _lookerParent, _looker) => {
 const ptToLookAt = getPosition(_target);
 const lookAtTransform = _lookerParent.transform.lookAt(ptToLookAt);
 setRotation(_looker, lookAtTransform);

const plane = results[0];
const followNull = results[1];
const targetNull = results[2];

// Random animation
// const scl = R.val(0.1);
// targetNull.transform.x = R.sin(;
// targetNull.transform.y = R.cos(;
// targetNull.transform.z = R.sin(;

// Do the look at
lookAt(targetNull, followNull, plane);

This is the “HeadQuarter” of the mechanism:

The Counter is to achieve the 2 step toggle action between setting up the target and the source.
Ssize is just a receiver from screen size of the device patch.
The clamp is to make sure i don’t go over the screen size. So it just stops when it hits the edges.
And then it goes to the Screen Pan to 3D Position patch that i made previously. check out my previous reply to see what’s inside.

This is the logic of the mechanism:

Maybe there is a way to make it even more effective and efficient but this method gets the job done, so… Yeah… I’d love to know if some of you can optimize this setup. Please share it here.

Umm… So… What this does is basically to make user can constantly toggle between setting up the source position and the target position, when they tap and slide/pan/scratch/whateveryouwannacallit one of them will be moving and after they release the tap, the position values are stored using the save vec3 value patch. that’s why the pan 3D position reciever goes to the “if” patches that switch on-off based on the counter in the “HeadQuarter”. Pretty sweet huh? :wink: No? :confused: Okay :tired_face: move on!

To switch between which one to control first, we can just swap all the patches after the “if” patch of the position. The target is directly to its 3d position input, while the source has something more because it’s the one who aim.

For the source, the multiply is set to 8 so the source (in this case a light source) can light up the mask from wider angle. when the multiply is set to 1, the range is perpendicular to the screen boundaries/edges. It then goes to extract the x and y position only using the unpack and pack it again except for the z. Because I want the z to be locked in the camera position, i use the ambient light global position.

To make the position always at the camera position even when you switch to back camera, you can put “absolute” patch between the swizzle z and the pack patch. Or if you want to be able to also set the z position of the source, you can try it with Screen Pinch or UI Slider.

And there you go…
I hope you find it useful and simple enough to follow or replicate for your project.
Let me know if you implement it in your filter, I’d love to check out what you do with it.

:smiley: :beers:


Wow this is great! I’m sure I will be back to this post at some point. I might have some more notes at that point (for now I don’t have time to dig in). In any case, it’s cool to see your process in solving this!