Suggestion of a function to get screen coordinates

[inaridarkfox4231]

Increasing access

Proposal to implement functions like screenX(), screenY(), screenZ() in processing.

I often see suggestions that p5.js also needs a function to calculate screen coordinates.
I thought it would be good to have such a function, even if it's not the method I proposed here.

Most appropriate sub-area of p5.js?

• Accessibility
• Color
• Core/Environment/Rendering
• Data
• DOM
• Events
• Image
• IO
• Math
• Typography
• Utilities
• WebGL
• Build process
• Unit testing
• Internationalization
• Friendly errors
• Other (specify if possible)

Feature request details

I thought of a function like this:

p5.prototype.getNDC = function (v){
  const _gl = this._renderer;
  // Transfer to camera coordinate system using uMVMatrix
  const camCoord = _gl.uMVMatrix.multiplyPoint(v);
  // Calculate ndc using uPMatrix
  const ndc = _gl.uPMatrix.multiplyAndNormalizePoint(camCoord);
  // Drop into canvas coordinates.
  // The depth value is converted so that near is 0 and far is 1.
  const _x = (0.5 + 0.5 * ndc.x) * this.width;
  const _y = (0.5 - 0.5 * ndc.y) * this.height;
  const _z = (0.5 + 0.5 * ndc.z);
  // Output in vector form.
  return createVector(_x, _y, _z);
}

This is a function that calculates screen coordinates and depth values ​​from 3D coordinates for the currently set camera.
Screen coordinates are values ​​based on the coordinate system of the 2D canvas, and depth values ​​are 0 for near and 1 for far.

This kind of technique is actually already used in the p5.js library.

In #6116, when I improved orbitControl, it became necessary to obtain normalized device coordinates, so I implemented functions by suggestion of dave pagurek.
The method used here is based on that method.

"getNDC" is a temporary name and has no particular meaning. I think it needs to be changed with a proper name.

getNDC demo

let gr;

function setup() {
  createCanvas(400, 400, WEBGL);
  gr = createGraphics(400, 400);
  gr.textSize(20);
  gr.textAlign(CENTER,CENTER);
  noStroke();
}

function draw() {
  background(220);
  orbitControl();
  lights();
  fill(255);
  sphere(40);

  const ndc = getNDC(createVector(0,0,0));
  gr.clear();
  gr.text(ndc.z.toFixed(2), ndc.x, ndc.y);

  push();
  noLights();
  camera(0,0,1,0,0,0,0,1,0);
  ortho(-1,1,-1,1,0,1);
  translate(0,0,1);
  texture(gr);
  plane(2);
  pop();
}

p5.prototype.getNDC = function (v){
  const _gl = this._renderer;
  // Transfer to camera coordinate system using uMVMatrix
  const camCoord = _gl.uMVMatrix.multiplyPoint(v);
  // Calculate ndc using uPMatrix
  const ndc = _gl.uPMatrix.multiplyAndNormalizePoint(camCoord);
  // Drop into canvas coordinates.
  // The depth value is converted so that near is 0 and far is 1.
  const _x = (0.5 + 0.5 * ndc.x) * this.width;
  const _y = (0.5 - 0.5 * ndc.y) * this.height;
  const _z = (0.5 + 0.5 * ndc.z);
  // Output in vector form.
  return createVector(_x, _y, _z);
}

2024-05-19.09-02-24.mp4

[davepagurek]

I think this could make sense to add, anecdotally it's probably the question I've been asked the most about p5, although not exclusively to WebGL, so 2D mode could feasibly implement something like this too.

@limzykenneth any thoughts on naming? Flash used to have methods like localToGlobal(coord) and globalToLocal(coord), for inspiration.

(Also, I reopened the issue, was it intended to be closed?)

[ffd8]

FYI - I brought up this issue with @bohnacker years ago (while attempting to port a Processing lib to p5.js that depended on screenX/Y/Z) and they came up with the following solution (I'm not sure if it's been submitted for the upcoming new website (of which all existing ones should be auto-ported incase authors didn't see message to submit) – would of course be great to have them built in!

https://github.com/bohnacker/p5js-screenPosition

[inaridarkfox4231]

After I came up with the idea, I lost motivation, so I closed it.
I'll leave it up to contributors to decide what to do with this discussion. Personally, I think it would be nice to have a function that simply returns screen coordinates and depth values ​​for 3D coordinates.

[Garima3110]

Personally, I think it would be nice to have a function that simply returns screen coordinates and depth values ​​for 3D coordinates.

I too agree to this , it would be nice to have such a function.
Just a suggestion ,
How about naming the function worldToScreen(coord) ?
This name is descriptive and indicates that the function converts 3D world coordinates to 2D screen coordinates.

[inaridarkfox4231]

Having a function like this will help contributors create new methods, like I did in #6116. I also think it is useful because it can be used to place visual information in 3D space.

2024-06-03.21-46-57.mp4

[ffd8]

For reference, two additional issue threads that dove into this topic:
Add screenX, screenY and screenZ
modelX/Y/Z() and screenX/Y/Z() for p5.js (WebGL)

[davepagurek]

Personally, I think it would be nice to have a function that simply returns screen coordinates and depth values ​​for 3D coordinates.

I too agree to this , it would be nice to have such a function.
Just a suggestion ,
How about naming the function worldToScreen(coord) ?
This name is descriptive and indicates that the function converts 3D world coordinates to 2D screen coordinates.

I like worldToScreen, it also leaves open a possibility of screenToWorld in the future too.

[ffd8]

I didn't understand the demo referenced above by @inaridarkfox4231 which had a sphere with lighting changing, but upon digging into your sketches, found a very relevant and well working example: ndc4_depthValue by dark_fox - amazing snippet!

+1 for worldToScreen() (perhaps something like aliases could then be built around it that mimic Processing's screenX()... for added compatibility)

[ffd8]

One small thing to add – Processing's screenX() are also so nice in that they allow 2D canvas x/y vector to screen capture.. so if one uses translate() or rotate(), it still knows where the coordinate is.. it would be ideal if said function could also handle Canvas2D rendering, as at the moment, it's explicitly WEBGL.

[inaridarkfox4231]

In the sketch introduced in the comment above, validation is applied using the depth value so that it is not displayed if it is on the back side.
icosahedron

  // Validate using the depth value of the origin
  const ndc0 = getNDC(createVector(0,0,0));
  for(let i=0; i<12; i++){
    const v = icosaV[i];
    const ndc = getNDC(v);
    if(ndc.z>ndc0.z)continue;
    if(showType===0)guide.text(vertexDataIcosa[i], ndc.x, ndc.y);
    if(showType===1)guide.text("["+i+"]",ndc.x, ndc.y);
  }

(After that, I made various changes for that sketch, and removed the ball because it was unnecessary.)

[davepagurek]

One small thing to add – Processing's screenX() are also so nice in that they allow 2D canvas x/y vector to screen capture.. so if one uses translate() or rotate(), it still knows where the coordinate is.. it would be ideal if said function could also handle Canvas2D rendering, as at the moment, it's explicitly WEBGL.

I left some code for world to local over here in a comment a while ago: https://www.reddit.com/r/p5js/s/6co56yZ78u

The reverse, local to screen, would be something like:

    const matrix = drawingContext.getTransform();
    const localCoord = matrix
      .scale(1 / pixelDensity())
      .transformPoint(
        new DOMPoint(x, y)
      );

[ffd8]

@davepagurek Wow that's awesome! For whatever/obvious reason the pixelDensity() does mess things up, especially since p5 has 2 by default, everything was doubled.. making pixelDensity(1) in the setup solved it. Any idea how/why that is and if it can be made flexible to whatever pixelDensity() is given? I found a solution for pixelDensity(2) using translate(), however it breaks on any other value than 1 or 2...:

const matrix = drawingContext.getTransform();
let pd = pixelDensity();
let scl = 1 / pd;
let transform = createVector(0, 0);
if(pd != 1){
	transform = createVector(-width * scl, -height * scl);
}

return matrix
.scale(scl)
.translate(transform.x, transform.y)
.transformPoint(
	new DOMPoint(v.x, v.y)
);

[davepagurek]

Oops, my fault, I think that's an order of operations problem. So p5 deals with pixel density by applying an initial scale(). This means that drawingContext.getTransform() is equivalent to a multiplication of that density first (call that D), and then multiplication by whatever other transforms you have call that (T): D × T. We just want to get T since that's the coordinate space p5 functions work in. To undo the density, we have to pre-multiply the division by density (D^-1 × D × T = T), but I was accidentally doing it afterwards (D × T × D^-1) which doesn't simplify to the same thing.

Here's a sketch that I think should work:

function setup() {
  createCanvas(200, 200)
  pixelDensity(3)
  
  translate(width/2, height/2)

  const matrix = new DOMMatrix()
    .scale(1 / pixelDensity())
    .multiply(drawingContext.getTransform());
  
  console.log(matrix
    .transformPoint(
        new DOMPoint(0, 0)
    ));
  // Logs 100, 100
}

[ffd8]

@davepagurek This all goes wooosh over my head, but very interesting to follow and worked perfect for all big/tiny densities, thanks for having another look at it.

Here's my implementation for a library I'm in early stages of sketching, where it's great to 'flatten' any transformed points in 2D/3D space and this combined with @inaridarkfox4231 example for WEBGL work amazingly, so many thanks for figuring out these complex issues!

Already flagged to do so, and hope this finds its way into v2.0, since it's something I've wished for many times in the past (having been so used to it from Processing world). I think across the various issues on the topic, there have been multiple folks interested to jump in for a contribution. Here's a basic combination of both – of course a fully fledged out implementation would also need to take into account createGraphics() layers, instanceMode, be ready to accept the vector or any combo of x, y, [z] coords, ideally also with aliases that match Processing's set of functions for additional smooth crossover.

function screenVector(v) {
	if(_renderer.drawingContext instanceof CanvasRenderingContext2D) {
		const matrix = new DOMMatrix()
		.scale(1 / pixelDensity())
		.multiply(drawingContext.getTransform());

		return matrix
		.transformPoint(
			new DOMPoint(v.x, v.y)
			);
	} else {
		const _gl = _renderer;
		const camCoord = _gl.uMVMatrix.multiplyPoint(v);
		const ndc = _gl.uPMatrix.multiplyAndNormalizePoint(camCoord);
		const _x = (0.5 + 0.5 * ndc.x) * width;
		const _y = (0.5 - 0.5 * ndc.y) * height;
		const _z = (0.5 + 0.5 * ndc.z);
		return createVector(_x, _y, _z);
	}
}