VRTK Support

[ghost]

We would love to use this in our project which uses VRTK, a widely used VR library which works with HTC Vive, Oculus VR and Google VR.

It would be great if you could create an Input Module for VRTK.
I made it work in our project with this to test it: https://gist.github.com/JackDn/0891b56d453fcef58c2ed379d54c5e71

[zachkinstner]

Hi @Jackdn, this is great! It looks like you were able to start from the HoverInputVive code, and then swapped in the corresponding API calls for VRTK. Is that correct?

[zachkinstner]

I started a new input-module-vrtk branch. Where should the input module obtain the Left/Right VRTK_ControllerEvent references?

I'm not sure this input module is the right direction, since it places another layer of abstraction between Hover UI and the input device. Looking through the VRTK "getting started" docs, the developer still needs to add/setup the device-specific assets in their scene -- they could simply attach the corresponding Hover UI input module component during this process.

If the developer routes their input through VRTK, they would lose some of Hover UI's device-specific settings/functionality (like default cursor positions/layouts for each device type, the option for cursors to follow the Oculus Touch Avatar's index/thumb, etc.).

@Jackdn, was VRTK's support for Google VR (Daydream) input your primary motivation for adding this input module? Full support for Daydream is on my to-do list for Hover UI Kit, but in the meantime, you might be able to use the Follow input module to lock a Hover UI cursor to the Daydream controller's transform.

[ghost]

Our project targets HTC Vive and Oculus VR.

Obtaining the VRTK_ControllerEvents instances might be a bit hard because they can be anywhere (or not even exist). They usually are a at Actual Left/Right Controller or Script Alias Left/Right Controller assigned in VRTK_SDKManager.

they could simply attach the corresponding Hover UI input module component during this process

That's a good point. However, some of our team members (including me) don't own any VR hardware. We use VRTKs simulator instead to test the game, so we need an input module.

[thestonefox]

If you need any help with this stuff then just hollar and we'll see what we can do to get them working together :)

[zachkinstner]

Hi guys, sorry for my delayed responses. I recently released 'EXA: The Infinite Instrument' (musical VR app), and I'm devoting most of my time/focus to that right now.

@Jackdn, thanks for your note about the VRTK simulator -- that makes more sense to me as a Hover UI Kit use case. For the actual (non-simulated) input devices, configuring Hover UI Kit to receive input data through VRTK might be convenient for devs, but this extra layer of abstraction prevents Hover UI Kit from providing cursor features and default settings that are tailored for each device.

@thestonefox, is VRTK_ControllerEvents the correct object for passing around the latest input data (position, rotation, button/trigger states, etc.)? Are these objects created at run-time, and how should the Hover UI Kit input module find them (notably, for right vs. left controllers)? Thanks!

[thestonefox]

@zachkinstner Yeah VRTK_ControllerEvents is the script that handles listening for the controller events such as button presses etc.

However, this is just a concrete script that calls off to the SDK Bridge to get the relevant values.

The SDK Bridge methods just need either the controller ID or the controller game object, which you can get with calls to the VRTK_DeviceFinder script.

So you could do things like:

VRTK_DeviceFinder.GetLeftController() to get the left controller game object

then pass that through to the SDK Bridge to get a button press.

Or you could assume the person has VRTK_ControllerEvents and do VRTK_DeviceFinder.GetLeftController().GetComponent<VRTK_ControllerEvents>()

That's over simplifying things a bit, but if that's the sort of info you were after then something like that would work.

Let me know if you need more info.

[andybak]

Curious about the status of this. I'm struggling to get HoverUI working at all in VR and if there's going to be some progress on VRTK integration then I might hold off for a bit.

(I suspect my issues are related to my ignorance but I don't want to spend too long digging through if the way that HoverUI and VRTK tie together is likely to change)

[zachkinstner]

Hi @andybak!

I'm struggling to get HoverUI working at all in VR

What issues are you having? Please check out the Getting Started docs, and feel free to ask questions if they come up. Note that this issue about "Image Effects" references has been a common question/issue recently.

if there's going to be some progress on VRTK integration

I'm currently very focused on my "EXA: The Infinite Instrument" VR music project, so I don't think there will be any near-term developments with VRTK integration. You can check out the VRTK Input Module (in the input-module-vrtk branch) code. This was based on earlier work from @Jackdn (see above), but please note that I haven't tested it thoroughly.

In general, I'd like to hear reasons why you need Hover UI Kit to work directly with VRTK. Being able to use Hover UI Kit with VRTK's input simulator was a good reason described before -- do you have others? I ask because there are several benefits for using Hover UI Kit's device-specific input modules (like Vive or Leap Motion modules), which have also been heavily tested -- using those modules would be my recommended approach.

[ohthatsjhall]

Hi @zachkinstner

In general, I'd like to hear reasons why you need Hover UI Kit to work directly with VRTK.

Wondering if you could add to this a little bit? I've used VRTK in many of my projects, and I've played around with HoverUI and I'm a huge fan. I was hoping to use both of these tools in one of my upcoming project.

If I were not using the VRTK simulator, would HoverUI working with VRTK's Vive module? I'm assuming the answer is no, but I wanted to double check.

VRTK offers a ton of out of the box tools that allow people to get up and running very quickly, and from what I can tell, it's user base is growing very rapidly. Although there might be several benefits to the device-specific input modules, I think there is definitely a need to VRTK integration.

I appreciate any guidance you may have and thanks for the hard work you put into HoverUI.

[zachkinstner]

Hi @ohthatsjhall,

Wondering if you could add to this a little bit?

Sure! As far as I know, there's nothing that should prevent you from using both Hover UI Kit and VRTK in the same project. The discussion in this thread is specifically about whether Hover UI Kit can (or should) obtain input device information through the VRTK interfaces.

would HoverUI working with VRTK's Vive module?

Currently, Hover UI Kit communicates with the Vive prefab/SDK directly -- rather than communicating with VRTK to obtain device-agnostic input data.

I want to be clear, I'm not saying an VRTK input module will never happen. In fact, an initial version has already been built, just not very well-tested (by me, at least). See VRTK Input Module code in the input-module-vrtk branch. Eventually, once my work on EXA slows down, I'll have more time to test this and merge it into the main project. In the meantime, feel free to try it out!

Hover UI Kit is similar to VRTK in the sense that both tools create an interface between their core functionality and the various input devices. So, a "VRTK input module" (for Hover UI Kit) essentially adds an extra (unnecessary?) layer of abstraction between Hover UI Kit and the input data. This is likely fine for simple use-cases, but it prevents Hover UI Kit from providing various device-specific features/optimizations.

allow people to get up and running very quickly [...] appreciate any guidance you may have

Let's look at the VRTK+Vive scenario. The VRTK setup instructions require the developer to import the SteamVR SDK and drop its main prefab into the scene. Then, a few clicks for getting it connected to VRTK. Great!

After this process, there are two more steps for Hover UI Kit to receive Vive input data. See the Vive input module docs for more info:

1. (Already done: add the SteamVR prefab to your scene)
2. Attach the HoverInputVive component to the SteamVR prefab object.
3. Define the HOVER_INPUT_VIVE symbol (in the Unity settings).

The last step would need to occur even for a VRTK integration (the symbol would be HOVER_INPUT_VRTK instead). So -- at a cost of one/two steps per supported device, you get Hover UI Kit support that is tailored to the device's specific capabilities. This includes default cursor positions/rotations/sizes, button mappings, and other specific features (like the ability to make the cursor follow the Oculus Touch index finger vs. locked to the controller).

[ohthatsjhall]

@zachkinstner That helps a ton. Thanks Zach!

[0rShemesh]

Got an error:
Assets/Hover/InputModules/VRTK/Scripts/HoverInputVrtk.cs(170,26): error CS1061: Type VRTK.VRTK_ControllerEvents' does not contain a definition for menuPressed' and no extension method menuPressed' of type VRTK.VRTK_ControllerEvents' could be found. Are you missing an assembly reference?

[zachkinstner]

Hi @0rShemesh, the VRTK input module is experimental, and isn't well-tested yet. For now, I'd recommend using Hover UI Kit's device-specific input modules -- my comment above explains why.

Regarding your error: did you import VRTK into your project? I just tested using the latest version of VRTK (from the Unity Asset Store) and was able to compile/run the project without this error.

[0rShemesh]

Thank you very much for answer me. I can't to work with steamvr plugin only with oculus, what should i do for work with vrtk + hover ui kit?

[zachkinstner]

Hi @0rShemesh, maybe there's a misunderstanding here. The example in my linked comment talks about using the Vive input module, but that's not required -- you could use the Oculus Touch input module instead. If you haven't already, please read through the input module documentation, which should be helpful for getting it installed and configured correctly.

It sounds like you're already using both VRTK and Oculus in your project. Hover UI Kit doesn't need to know about (or get controller data from) VRTK. I recommend using the Oculus Touch input module, which allows Hover UI Kit to get information directly from the Oculus Touch devices.

Note: if you already have the Oculus prefabs in your scene, you would probably use the "To install this module manually:" instructions found on the input module page.