AutoUSDZ broken in iOS 17.5 beta

[milesgreen]

Discussed in #4741

@elalish - Following Apple's feedback, I'm elevating this Discussion to an Issue:

With the new iOS 17.5 beta release (21F5048f) opening AR from ModelViewer when using AutoUSDZ in Safari is broken.

When opening a USDZ blob, the user is presented with a prompt that asks:

Do you want to download "model.usdz"?

With options to View or Download

Selecting Download, downloads the file to device storage.

Selecting View opens a new page, displays the USDZ thumbnail image with an AR icon at top right. The user must tap again on thumbnail to finally open the model in AR QuickLook.

Closing the AR QuickLook view returns the user to the USDZ thumbnail preview. From here the user must use the back button to get back to the originating HTML page.

Not a great user experience.

This seemed like a Safari/iOS issue. So I filed a bug here:
https://bugs.webkit.org/show_bug.cgi?id=272321

But now we have a response from Apple:

In iOS 17.5, simulated click events will trigger the download path which is why you see the new behavior.
We will ask three.js if they can open the blob directly from the user action, which would avoid the download path behavior.

But I don't think the three.js USDZ Exporter is simulating a click event. I think that's from within ModelViewer here:

model-viewer/packages/model-viewer/src/features/ar.ts

Line 414 in 99b50ff

anchor.click();

This will break a huge range of 3D/AR experiences across the internet, across many different 3D engines.

Curious what's driving this decision from Apple.

[milesgreen]

I think #4742 is related to this.

[elalish]

In iOS 17.5, simulated click events will trigger the download path which is why you see the new behavior.
We will ask three.js if they can open the blob directly from the user action, which would avoid the download path behavior.

That's fine, but it's still their bug. Activating WebXR on Chrome has exactly this same requirement - it's a good one. However, Chrome is smart enough to track that the simulated click originated from a user action. Safari needs to do the same.

[elalish]

Weird, I was just replying to their bug and in the middle of it they removed my permission to access the bug??

[milesgreen]

Just thinking some more...

Simulated clicks are still working fine in ModelViewer in iOS 17.5 when using a USDZ source file.

It's only an issue when there's a USDZ blob.

So it does seem more of a blob-handling bug rather than intentionally blocking simulated clicks, maybe.

[milesgreen]

I can still see the ticket. And the latest comment is:

After some discussion we reached a solution where the end user behavior for opening the USDZ blob will be identical to opening the USDZ files from https://developer.apple.com/augmented-reality/quick-look/

Hopefully this is a reasonable compromise solution

That's still a bit ambiguous though. So have asked for clarification.

But could be good. 🤞

[hybridherbst]

Also replied there. I didn’t have access rights at first but did after logging in again.

The new behavior is super strange, for example the USDZ renderer in the browser is a different one than in AR (some models don’t even display in the Browser one). Plus it breaks in many different ways on visionOS…, and iFrames don't work at all.

[milesgreen]

Also, as @hybridherbst points out, this totally breaks when opening from an iFrame. The prompt only displays the option to Download. There's no option to View. That's going to break a lot of embedded 3D viewers.

See:
https://modelviewer-extended-params.glitch.me/parent.html

[elalish]

FYI @AdaRoseCannon

[AdaRoseCannon]

Thank you for bringing it to my attention please report it via Feedback Assistant so it gets tracked. Also if you like please send me the code so I can track it.

[milesgreen]

Thank you for bringing it to my attention please report it via Feedback Assistant so it gets tracked. Also if you like please send me the code so I can track it.

Thanks @AdaRoseCannon - now reported via Feedback Assistant with some additional information and test cases (FB13734957)

[AdaRoseCannon]

That's great! Thank you :)

[milesgreen]

I made some more test cases and found some interesting observations. I stripped things back to a bare bones implementation. I removed ModelViewer from the equation and am using three.js USDZ Exporter directly.

See here:
https://arquicklook-usdz-blob.glitch.me/

Step 1. Pick a model. Some are GLB, simple and complex. Some are ThreeJS primitives.

Step 2. View chosen model in ThreeJS.

Step 3. USDZ blob is automatically generated (from three.js) and attached to USDZ anchor link on the page. Time taken to generate blob and the blob size is displayed underneath. Tapping this USDZ link works - all test models open in ARQL.

Step 4. Update USDZ. This generates a new USDZ blob and attaches to the USDZ link. USDZ link still works - all models open in ARQL.

Step 5. Simulate a click event on the USDZ link. Works fine for all models.

Step 6. Generates a new blob, attaches to the USDZ Link, then simulates a click event. Some models work and open to ARQL. Some models fail.

Step 7. Generates a new blob, dynamically creates an anchor tag, attaches the blob, simulates a click (replicating ModelViewer's approach). Some models work and open to ARQL. Some models fail.

Step 8. View it in an iFrame and repeat all the steps above. For Steps 6 and 7, for models that fail, the prompt only shows an option to Download, not View.

Observations:

• Is the simulated click an issue? Simulated click in Step 5 works fine and simulated clicks still work for SOME models in Step 6 and 7. So a simulated click isn't necessarily the issue.

• Is the asynchronous nature of model generation and simulated click an issue? Is it too long a gap between user interaction and simulated click? However, the Astronaut model takes 100ms to generate, and FAILS to open in ARQL. But the Teapot model takes longer - 170ms - and DOES open in ARQL. So time does not set to be the cause.

• Is blob size an issue? However, Astronaut blob is around 1,300 KB and FAILS. But Teapot blob is 11,000 KB and DOES work. So blob size does not seem to be the cause.

• All the models that DO work, have NO textures. All the models that DON’T work, DO have image based texture maps. For example, Chrome model has no textures and works. Rustic Iron model, which uses the exact same geometry, has texture maps, and FAILS to open in ARQL.

Conclusions:

• All models pass steps 3, 4, 5.
• Some GLB models fail steps 6 and 7.
• Some GLB models pass steps 6 and 7.
• All ThreeJS primitives pass all steps.
• All GLB models with no textures pass all steps.
• All GLB models with textures fail steps 6 and 7.

So it seems to be a combined factor of models with textures, generating USDZ blob, and simulating a click.

Maybe...?

[elalish]

Dang, @AdaRoseCannon I hope Apple appreciates all the gratis effort here to help fix their product!

[hybridherbst]

@AdaRoseCannon I had also reported this already as FB13722319 with a number of different cases that all break in slightly different ways, hope that helps

[AdaRoseCannon]

Thank you @hybridherbst

[milesgreen]

Still an issue with iOS 17.5 beta 3

[milesgreen]

Good news! This is now fixed in iOS 17.5 beta 4.

Auto USDZ is once again working in ModelViewer.

All of my tests from https://arquicklook-usdz-blob.glitch.me/ now pass for all model types and all methods of opening the generated USDZ.

Thank you @AdaRoseCannon for helping escalate.

[elalish]

Excellent work all, helping Apple to get this fixed! @AdaRoseCannon does Apple have any kind of public bug tracker where we can alert you to these issues and be notified when they're resolved? Or should model-viewer continue providing that service for you?