mrtk_developmentreleases/2.0.0releases/2.1.0releases/2.2.0
  • 指南
  • API 文档
  • 指南

    Show / Hide Table of Contents
    • MRTK入门
      • 发布说明
      • MRTK包内容
      • 从早期版本更新
      • 从HTK更新
      • 生成和部署MRTK
      • NuGet包
      • MRTK配置对话框
      • 开始使用 MRTK和XR SDK
      • 性能
      • 全息图稳定
      • 在大型项目中使用MRTK
    • 架构
      • 概述
      • 框架和运行时
      • 输入系统
        • 术语
        • 核心系统
        • Controllers, pointers, and focus
      • 系统,扩展服务和 data providers
    • 功能概述
      • 边界系统 Boundary System
        • 边界系统概述
        • 配置边界可视化
      • 相机系统 Camera System
        • 相机系统概述
        • Camera Settings Providers
          • Windows 混合现实相机设置
          • Unity AR相机设置[实验性]
          • 创建camera settings provider
      • 跨平台支持
        • 为iOS和Android配置MRTK
      • 检测平台能力
      • 诊断系统 Diagnostics System
        • 诊断系统概述
        • 配置诊断系统
        • 使用可视化分析器
      • 扩展服务 Extension Services
        • 扩展服务创建向导
        • 场景过渡服务概述
      • 输入系统 Input System
        • 输入概览
        • 输入动作 Input Actions
        • 输入事件 Input Events
        • Input Providers
          • Input Providers 概述
          • 创建一个 input data provider
        • 控制器 Controllers
        • 眼动跟踪 Eyes
          • 概述
          • 入门
          • 通过代码访问数据
          • 验证跟踪校准 Validate Tracking Calibration
        • 凝视 Gaze
        • 手势 Gestures
        • 手部 Hands
        • 如何添加近距离交互 Near Interaction
        • 编辑器内输入模拟 In-Editor Input Simulation
        • 指针 Pointers
        • 语音输入 Voice Input
          • 语音转录 Dictation
          • 语音(命令和控制) Speech
      • 多场景系统 Multi Scene System
        • 多场景系统概述
        • 场景类型 Scene Types
        • 内容场景加载 Content Scene Loading
        • 监测内容加载 Monitoring Content Loading
        • 光照场景操作 Lighting Scene Operations
      • 打包 Packaging
        • MRTK包 MRTK Packages
        • MRTK模块化 MRTK Modularization
      • 配置文件 Profiles
        • 配置文件概述
        • 配置指南 Configuration Guide
      • 渲染 Rendering
        • 材质实例概述 Material Instance Overview
        • Shaders
          • MRTK标准着色器 MRTK Standard Shader
      • 服务 Services
        • 什么是混合现实特性
        • 什么是 MixedRealityServiceRegistry 和IMixedRealityServiceRegistrar
        • 扩展服务 Extension services
      • 空间感知系统 Spatial Awareness System
        • 空间感知概述
        • Spatial Observers
          • 为设备配置 Observers
          • 为Editor配置 Observers
          • 通过代码控制 Observers
          • 创建自定义 Observer
      • 传送系统概述 Teleport System Overview
      • 工具
        • Dependency Window
        • Extension Service Creation Wizard
        • Holographic Remoting
        • Input Animation Recording
          • Input Animation File Format Specification
        • Optimize Window
        • Runtime tools
          • Controller Mapping tool
      • UX 构建模块
        • 工具箱窗口 Toolbox Window
        • 按钮 Button
        • 边界框 Bounding Box
        • 对象操纵器 Object Manipulator
        • Manipulation Handler [废弃]
        • Slate
        • 系统键盘 System Keyboard
        • 可交互的 Interactable
        • Solvers
          • 点击放置 Tap to Place
        • 对象集合 Object Collection
        • 提示工具条 Tooltips
        • 滑动条 Slider
        • 手部菜单 Hand Menu
        • 近处菜单 Near Menu
        • 应用程序栏 App Bar
        • 指尖可视化 Fingertip Visualization
        • 进度指示器 Progress Indicator
        • 对话框 Dialog [实验性]
        • 手势教学 Hand Coach [实验性]
        • 可滚动对象集合 Scrolling Object Collection [实验性]
        • 脉冲着色器 Pulse Shader [实验性]
        • 停靠控件 Dock Control [实验性]
        • HoloLens键盘助手 [实验性]
      • 示例场景
        • 示例中心
        • 手势交互示例场景
        • 眼动跟踪示例场景
    • Contributing
      • Contributing Overview
      • Coding Guidelines
      • Writing and Running Tests
      • Writing Documentation
      • Pull Requests
      • Experimental Features
      • Breaking Changes
      • How to use DocFX
    • Planning
      • Roadmap
    • Notice
    • Authors

    How to Configure Leap Motion (by Ultraleap) Hand Tracking in MRTK

    A Leap Motion Controller is required to use this data provider.

    The Leap Motion Data Provider enables articulated hand tracking for VR and could be useful for rapid prototyping in the editor. The data provider can be configured to use the Leap Motion Controller mounted on a headset or placed on a desk face up.

    LeapMotionIntroGif

    Using Leap Motion (by Ultraleap) hand tracking in MRTK

    1. Prepare MRTK project for Leap Motion

      • This step only applies for the Leap Motion Unity Core Assets version 4.4.0 and if the source of MRTK is cloned from the git repo, NOT from the Unity packages. This step is not required if the Core Assets from Leap Motion Unity Modules version 4.5.0 are used. If the MRTK source is going to be from the Unity packages, start at the next step

      • Navigate to Mixed Reality Toolkit > Utilities > Leap Motion > Configure CSC File for Leap Motion. Updating the csc file filters out the obsolete warnings produced by the Leap Motion Unity Core Assets. The MRTK repo contains a csc file that converts warnings to errors, this conversion halts the Leap Motion MRTK configuration process. The obsolete warnings issue is tracked here.

      LeapMotionUpdateCSCFile

    2. Importing MRTK and the Core Assets from Leap Motion Unity Modules version 4.5.0

      • Import the Microsoft.MixedReality.Toolkit.Foundation package into the Unity project.
      • Install the Leap Motion SDK 4.0.0
      • Download and import the Core Assets from Leap Motion Unity Modules version 4.5.0
        • Leap Motion Unity Core Assets version 4.4.0 and the Core Assets from Leap Motion Unity Modules version 4.5.0 are supported, but Core Assets from Leap Motion Unity Modules version 4.5.0 are preferred.
        • If using Core Assets from Leap Motion Unity Modules version 4.5.0, import the Core package into the Unity project.
      Note

      On import of the Core Assets, test directories are removed and 10 assembly definitions are added to the project. Make sure Visual Studio is closed.

      • If using Unity 2018.4.x

        • After the Core Assets import, navigate to Assets/LeapMotion/, there should be a LeapMotion.asmdef file next to the Core directory. If the asmdef file is not present, go to the Leap Motion Common Errors. If the file is present, go to the next step.
      • If using Unity 2019.3.x, go to the next step

    3. Adding the Leap Motion Data Provider

      • Create a new Unity scene
      • Add MRTK to the scene by navigating to Mixed Reality Toolkit > Add to Scene and Configure
      • Select the MixedRealityToolkit game object in the hierarchy and select Copy and Customize to clone the default mixed reality profile.

      LeapMotionProfileClone

      • Select the Input Configuration Profile

      InputConfigurationProfile

      • Select Clone in the input system profile to enable modification.

      LeapMotionInputProfileClone

      • Open the Input Data Providers section, select Add Data Provider at the top, a new data provider will be added at the end of the list. Open the new data provider and set the Type to Microsoft.MixedReality.Toolkit.LeapMotion.Input > LeapMotionDeviceManager

      LeapAddDataProvider

      • Select Clone to change the default Leap Motion settings.

      LeapDataProviderPreClone

      • The Leap Motion Data Provider contains the LeapControllerOrientation property which is the location of the Leap Motion Controller. LeapControllerOrientation.Headset indicates the controller is mounted on a headset. LeapControllerOrientation.Desk indicates the controller is placed flat on desk. The default value is set to LeapControllerOrientation.Headset. If the orientation is the desk, an extra property LeapControllerOffset will appear. LeapControllerOffset is the anchor for the position of the desk leap hands. The offset is calculated relative to the main camera position and the default value is (0,-0.2, 0.2) to make sure the hands appear in front and in view of the camera.
      • EnterPinchDistance and ExitPinchDistance are the distance thresholds for pinch/air tap gesture detection. The pinch gesture is calculated by measuring the distance between the index finger tip and the thumb tip. To raise an on input down event, the default EnterPinchDistance is set to 0.02. To raise an on input up event (exiting the pinch), the default distance between the index finger tip and the thumb tip is 0.05.
      LeapControllerOrientation: Headset (Default) LeapControllerOrientation: Desk
      LeapHeadsetGif LeapDeskGif
      LeapHeadsetInspector LeapDeskInspector
    4. Testing the Leap Motion Data Provider

      • After Leap Motion Data Provider has been added to the input system profile, press play, move your hand in front of the Leap Motion Controller and you should see the joint representation of the hand.
    5. Building your project

      • Navigate to File > Build Settings
      • Only Standalone builds are supported if using the Leap Motion Data Provider.
      • For instructions on how to use a Windows Mixed Reality headset for Standalone builds, see Build and Deploy.

    Getting the hand joints

    Getting joints using the Leap Motion Data Provider is identical to hand joint retrieval for an MRTK Articulated Hand. For more information, see Hand Tracking.

    With MRTK in a unity scene and the Leap Motion Data Provider added as an Input Data Provider in the Input System profile, create an empty game object and attach the following example script.

    This script is a simple example of how to retrieve the pose of the palm joint in a Leap Motion Hand. A sphere follows the left Leap hand while a cube follows the right Leap hand.

    using Microsoft.MixedReality.Toolkit;
    using Microsoft.MixedReality.Toolkit.Input;
    using Microsoft.MixedReality.Toolkit.Utilities;
    using System.Collections.Generic;
    using UnityEngine;
    
    public class LeapHandJoints : MonoBehaviour, IMixedRealityHandJointHandler
    {
        private GameObject leftHandSphere;
        private GameObject rightHandCube;
    
        private void Start()
        {
            // Register the HandJointHandler as a global listener
            CoreServices.InputSystem.RegisterHandler<IMixedRealityHandJointHandler>(this);
    
            // Create a sphere to follow the left hand palm position
            leftHandSphere = GameObject.CreatePrimitive(PrimitiveType.Sphere);
            leftHandSphere.transform.localScale = Vector3.one * 0.03f;
    
            // Create a cube to follow the right hand palm position
            rightHandCube = GameObject.CreatePrimitive(PrimitiveType.Cube);
            rightHandCube.transform.localScale = Vector3.one * 0.03f;
        }
    
        public void OnHandJointsUpdated(InputEventData<IDictionary<TrackedHandJoint, MixedRealityPose>> eventData)
        {
            if (eventData.Handedness == Handedness.Left)
            {
                Vector3 leftHandPalmPosition = eventData.InputData[TrackedHandJoint.Palm].Position;
                leftHandSphere.transform.position = leftHandPalmPosition;
            }
    
            if (eventData.Handedness == Handedness.Right)
            {
                Vector3 rightHandPalmPosition = eventData.InputData[TrackedHandJoint.Palm].Position;
                rightHandCube.transform.position = rightHandPalmPosition;
            }
        }
    

    Unity editor workflow tip

    Using the Leap Motion Data Provider does not require a VR headset. Changes to an MRTK app can be tested in the editor with the Leap hands without a headset.

    The Leap Motion Hands will show up in the editor, without a VR headset plugged in. If the LeapControllerOrientation is set to Headset, then the Leap Motion controller will need to be held up by one hand with the camera facing forward.

    Note

    If the camera is moved using WASD keys in the editor and the LeapControllerOrientation is Headset, the hands will not follow the camera. The hands will only follow camera movement if a VR headset is plugged in while the LeapControllerOrientation is set Headset. The Leap hands will follow the camera movement in the editor if the LeapControllerOrientation is set to Desk.

    Removing Leap Motion from the Project

    1. Close Unity
    2. Close Visual Studio, if it's open
    3. Open File Explorer and navigate to the root of the MRTK Unity project
      • Delete the UnityProjectName/Library directory
      • Delete the UnityProjectName/Assets/LeapMotion directory
      • Delete the UnityProjectName/Assets/LeapMotion.meta file
    4. Reopen Unity

    In Unity 2018.4, you might notice that errors still remain in the console after deleting the Library and the Leap Motion Core Assets. If errors are logged after reopening, restart Unity again.

    Common Errors

    Leap Motion Obsolete Errors

    If the source of MRTK is from the repo and the Unity version is 2019.3.x, the following error might be in the console after the import of the Leap Motion Unity Core Assets:

    Assets\LeapMotion\Core\Scripts\EditorTools\LeapPreferences.cs(84,6): error CS0618: 'PreferenceItem' is obsolete: '[PreferenceItem] is deprecated. Use [SettingsProvider] instead.
    

    In Unity version 2018.4.x, the following obsolete errors might be logged:

    Assets\LeapMotion\Core\Scripts\Attachments\AttachmentHands.cs(105,7): error CS0618: 'PrefabType' is obsolete: 'PrefabType no longer tells everything about Prefab instance.'
    
    Assets\LeapMotion\Core\Scripts\Attachments\AttachmentHands.cs(105,31): error CS0618: 'PrefabUtility.GetPrefabType(Object)' is obsolete: 'Use GetPrefabAssetType and GetPrefabInstanceStatus to get the full picture about Prefab types.'
    

    These errors appear if Mixed Reality Toolkit > Utilities > Leap Motion > Configure CSC File for Leap Motion was not selected BEFORE the Leap Motion Unity Core Assets import.

    Solution

    • Navigate to Mixed Reality Toolkit > Utilities > Leap Motion > Configure CSC File for Leap Motion, this will update the csc file to filter out Leap Motion warnings that are converted to errors in MRTK repo.
    • Close Unity
    • Reopen Unity

    Leap Motion has not integrated with MRTK

    This error can occur if the Unity version is 2018.4.x, the MRTK source is from the Unity packages and after the import of the Leap Motion Unity Core Assets.

    To test if MRTK recognizes the presence of the Leap Motion Unity Core Assets, open the LeapMotionHandTrackingExample scene located in MRTK/Examples/Demos/HandTracking/ and press play. If the Leap Motion Unity Core Assets are recognized a green message on the informational panel in the scene will appear. If the Leap Motion Unity Core Assets are not recognized a red message will appear.

    If the Leap Motion Unity Core Assets are in your project and you see a red message on the informational panel, the project needs to be configured.

    Solution

    • Navigate to Mixed Reality Toolkit > Utilities > Leap Motion > Configure Leap Motion
      • This will force Leap Motion integration if the configuration process was not started after the Leap Motion Unity Core Assets import.

    Copying assembly Multiplayer HLAPI failed

    On import of the Leap Motion Unity Core Assets this error might be logged:

    Copying assembly from 'Temp/com.unity.multiplayer-hlapi.Runtime.dll' to 'Library/ScriptAssemblies/com.unity.multiplayer-hlapi.Runtime.dll' failed
    

    Solution

    • A short term solution is to restart Unity. See Issue 7761 for more information.

    Leap Motion Example Scene

    The example scene uses the DefaultLeapMotionConfiguration profile and determines if the Unity project has been configured correctly to use the Leap Motion Data Provider.

    The example scene is contained in the Microsoft.MixedReality.Toolkit.Examples package in the MRTK/Examples/Demos/HandTracking/ directory.

    See also

    • Input Providers
    • Hand Tracking
    • Improve this Doc
    In This Article
    • Using Leap Motion (by Ultraleap) hand tracking in MRTK
    • Getting the hand joints
    • Unity editor workflow tip
    • Removing Leap Motion from the Project
    • Common Errors
      • Leap Motion Obsolete Errors
        • Solution
      • Leap Motion has not integrated with MRTK
        • Solution
      • Copying assembly Multiplayer HLAPI failed
        • Solution
    • Leap Motion Example Scene
    • See also
    Back to top Generated by DocFX