mrtk_developmentreleases/2.0.0releases/2.1.0releases/2.2.0
  • 指南
  • API 文档
  • 指南

    Show / Hide Table of Contents
    • MRTK入门
      • 发布说明
      • MRTK包内容
      • 从早期版本更新
      • 从HTK更新
      • 生成和部署MRTK
      • NuGet包
      • MRTK配置对话框
      • 开始使用 MRTK和XR SDK
      • 性能
      • 全息图稳定
      • 在大型项目中使用MRTK
    • 架构
      • 概述
      • 框架和运行时
      • 输入系统
        • 术语
        • 核心系统
        • Controllers, pointers, and focus
      • 系统,扩展服务和 data providers
    • 功能概述
      • 边界系统 Boundary System
        • 边界系统概述
        • 配置边界可视化
      • 相机系统 Camera System
        • 相机系统概述
        • Camera Settings Providers
          • Windows 混合现实相机设置
          • Unity AR相机设置[实验性]
          • 创建camera settings provider
      • 跨平台支持
        • 为iOS和Android配置MRTK
      • 检测平台能力
      • 诊断系统 Diagnostics System
        • 诊断系统概述
        • 配置诊断系统
        • 使用可视化分析器
      • 扩展服务 Extension Services
        • 扩展服务创建向导
        • 场景过渡服务概述
      • 输入系统 Input System
        • 输入概览
        • 输入动作 Input Actions
        • 输入事件 Input Events
        • Input Providers
          • Input Providers 概述
          • 创建一个 input data provider
        • 控制器 Controllers
        • 眼动跟踪 Eyes
          • 概述
          • 入门
          • 通过代码访问数据
          • 验证跟踪校准 Validate Tracking Calibration
        • 凝视 Gaze
        • 手势 Gestures
        • 手部 Hands
        • 如何添加近距离交互 Near Interaction
        • 编辑器内输入模拟 In-Editor Input Simulation
        • 指针 Pointers
        • 语音输入 Voice Input
          • 语音转录 Dictation
          • 语音(命令和控制) Speech
      • 多场景系统 Multi Scene System
        • 多场景系统概述
        • 场景类型 Scene Types
        • 内容场景加载 Content Scene Loading
        • 监测内容加载 Monitoring Content Loading
        • 光照场景操作 Lighting Scene Operations
      • 打包 Packaging
        • MRTK包 MRTK Packages
        • MRTK模块化 MRTK Modularization
      • 配置文件 Profiles
        • 配置文件概述
        • 配置指南 Configuration Guide
      • 渲染 Rendering
        • 材质实例概述 Material Instance Overview
        • Shaders
          • MRTK标准着色器 MRTK Standard Shader
      • 服务 Services
        • 什么是混合现实特性
        • 什么是 MixedRealityServiceRegistry 和IMixedRealityServiceRegistrar
        • 扩展服务 Extension services
      • 空间感知系统 Spatial Awareness System
        • 空间感知概述
        • Spatial Observers
          • 为设备配置 Observers
          • 为Editor配置 Observers
          • 通过代码控制 Observers
          • 创建自定义 Observer
      • 传送系统概述 Teleport System Overview
      • 工具
        • Dependency Window
        • Extension Service Creation Wizard
        • Holographic Remoting
        • Input Animation Recording
          • Input Animation File Format Specification
        • Optimize Window
        • Runtime tools
          • Controller Mapping tool
      • UX 构建模块
        • 工具箱窗口 Toolbox Window
        • 按钮 Button
        • 边界框 Bounding Box
        • 对象操纵器 Object Manipulator
        • Manipulation Handler [废弃]
        • Slate
        • 系统键盘 System Keyboard
        • 可交互的 Interactable
        • Solvers
          • 点击放置 Tap to Place
        • 对象集合 Object Collection
        • 提示工具条 Tooltips
        • 滑动条 Slider
        • 手部菜单 Hand Menu
        • 近处菜单 Near Menu
        • 应用程序栏 App Bar
        • 指尖可视化 Fingertip Visualization
        • 进度指示器 Progress Indicator
        • 对话框 Dialog [实验性]
        • 手势教学 Hand Coach [实验性]
        • 可滚动对象集合 Scrolling Object Collection [实验性]
        • 脉冲着色器 Pulse Shader [实验性]
        • 停靠控件 Dock Control [实验性]
        • HoloLens键盘助手 [实验性]
      • 示例场景
        • 示例中心
        • 手势交互示例场景
        • 眼动跟踪示例场景
    • Contributing
      • Contributing Overview
      • Coding Guidelines
      • Writing and Running Tests
      • Writing Documentation
      • Pull Requests
      • Experimental Features
      • Breaking Changes
      • How to use DocFX
    • Planning
      • Roadmap
    • Notice
    • Authors

    Eye-Supported Navigation in MRTK

    MRTK

    Imagine you are reading information on a slate and when you reach the end of the displayed text, the text automatically scrolls up to reveal more content. Or you can fluently zoom in where you are looking at. The map also automatically adjusts the content to keep the things of interest within your field of view. Another interesting application is the hands-free observation of 3D holograms by automatically bringing the parts of the hologram that you are looking at to the front. These are some of the examples that are described on this page in context of eye-supported navigation.

    The following descriptions assume that you are already familiar with how to set up eye tracking in your MRTK scene and with the basics of accessing eye tracking data in MRTK Unity. The examples discussed in the following are all part of the EyeTrackingDemo-03-Navigation.unity scene.

    Summary: Auto scroll of text, eye-gaze-supported pan and zoom of a virtual map, hands-free gaze-directed 3D rotation.

    Auto Scroll

    Auto scroll enables the user to scroll through texts without lifting a finger. Simply continue reading and the text will automatically scroll up or down depending on where the user is looking. You can start off from the example provided in EyeTrackingDemo-03-Navigation.unity. This example uses a TextMesh component to allow for flexibly loading and formatting new text. To enable auto scroll, simply add the following two scripts to your collider component of the textbox:

    ScrollRectTransf

    To scroll through a TextMesh or more generally speaking a RectTransform component you can use the ScrollRectTransf script. If you want to scroll through a texture instead of a RectTransform, use ScrollTexture instead of ScrollRectTransf. In the following, the parameters of ScrollRectTransf that are available in the Unity Editor are explained in more detail:

    Parameters Description
    LimitPanning If enabled, will stop the scrollable content at its boundary.
    RectTransfToNavigate Reference to the RectTransform to scroll in.
    RefToViewport Reference to the parent RectTransform of the scrollable content to determine the correct offset and boundary.
    AutoGazeScrollIsActive If enabled, the text will automatically scroll if the user looks at an active region (e.g., the top and bottom part of your scroll panel if the vertical scroll speed is not zero).
    ScrollSpeed_x If set to a value unequal to zero, horizontal scroll will be enabled. Negative values mean a change in scroll direction: Left to right vs. right to left.
    ScrollSpeed_y If set to a value unequal to zero, vertical scroll will be enabled. Negative values mean a change in scroll direction: Up to down vs. down to up.
    MinDistFromCenterForAutoScroll Normalized minimal distance in x and y from center of the target's hit box (0, 0) to scroll. Thus, values must range between 0 (always scroll) and 0.5 (no scroll).
    UseSkimProofing If enabled, it prevents sudden scroll movements when quickly looking around. This may make scrolling feel less responsive though. It can be tuned with the SkimProofUpdateSpeed value.
    SkimProofUpdateSpeed The lower the value, the slower the scrolling will speed up after skimming. Recommended value: 5.

    Eye-supported scroll setup in Unity

    EyeTrackingTarget

    Attaching the EyeTrackingTarget component allows for flexibly handle eye-gaze-related events. The scroll sample demonstrates scrolling text that starts when the user looks at the panel and stops when the user is looking away from it. Eye-supported scroll setup in Unity: EyeTrackingTarget

    Gaze-supported Pan and Zoom

    Who hasn't used a virtual map before to search for their home or to explore entirely new places? Eye tracking allows you to directly dive into exactly the parts that you're interested in and once zoomed in, you can smoothly follow the course of a street to explore your neighborhood! This is not only useful for exploring geographical maps, but also to checking out details in photographs, data visualizations or even live-streamed medical imagery. To use this capability in your app is easy! For content rendered to a Texture (e.g., a photo, streamed data), simply add the PanZoomTexture script. For a RectTransform use PanZoomRectTransf. Extending the Auto Scroll capability, we essentially enable to scroll both vertically and horizontally at the same time and magnify content right around the user's current focus point.

    Parameters Description
    LimitPanning If enabled, will stop the scrollable content at its boundary.
    HandZoomEnabledOnStartup Indicates whether hand gestures are automatically enabled to perform a zoom gesture. You may want to disable it at first to avoid accidentally triggering zoom actions.
    RendererOfTextureToBeNavigated Referenced renderer of the texture to be navigated.
    Zoom_Acceleration Zoom acceleration defining the steepness of logistic speed function mapping.
    Zoom_SpeedMax Maximum zoom speed.
    Zoom_MinScale Minimum scale of the texture for zoom in - e.g., 0.5f (half the original size).
    Zoom_MaxScale Maximum scale of the texture for zoom out - e.g., 1f (the original size) or 2.0f (double the original size).
    Zoom_TimeInSecToZoom Timed zoom: Once triggered, a zoom in/out will be performed for the given amount of time in seconds.
    Zoom_Gesture Type of hand gesture to use to zoom in/out.
    --- ---
    Pan_AutoScrollIsActive If enabled, the text will automatically scroll if the user looks at an active region (e.g., the top and bottom part of your scroll panel if the vertical scroll speed is not zero).
    Pan_Speed_x If set to a value unequal to zero, horizontal scroll will be enabled. Negative values mean a change in scroll direction: Left to right vs. right to left.
    Pan_Speed_y If set to a value unequal to zero, vertical scroll will be enabled. Negative values mean a change in scroll direction: Up to down vs. down to up.
    Pan_MinDistFromCenter Normalized minimal distance in x and y from center of the target's hit box (0, 0) to scroll. Thus, values must range between 0 (always scroll) and 0.5 (no scroll).
    UseSkimProofing If enabled, it prevents sudden scroll movements when quickly looking around. This may make scrolling feel less responsive though. It can be tuned with the SkimProofUpdateSpeed value.
    SkimProofUpdateSpeed The lower the value, the slower the scrolling will speed up after skimming. Recommended value: 5.

    Eye-supported pan and zoom setup in Unity

    Attention-based 3D Rotation

    Imagine looking at a 3D object and the parts you want to see more closely magically turn toward you - as if the system would read your mind and know to turn the item toward you! That is the idea for attention-based 3D rotations which enable you to investigate all side of a hologram without lifting a finger. To enable this behavior, simply add the OnLookAtRotateByEyeGaze script to the part of your GameObject with a Collider component. You can tweak several parameters that are listed below to limit how fast and in which directions the hologram will turn.

    As you can imagine, having this behavior active at all times may quickly become pretty distracting in a crowded scene. This is why you may want to start out with this behavior disabled and then enable it quickly using voice commands. Alternatively, we added an example in EyeTrackingDemo-03-Navigation.unity to use TargetMoveToCamera for which you can select a focused target and it flies in front of you - simply say "Come to me".

    Once in the near mode, the auto rotation mode is automatically enabled. In that mode, you can observe it from all sides either simply leaning back and looking at it, walking around it or reaching out to grab and rotate it with your hand. When you dismiss the target (look & pinch or say "Send back"), it will return to its original location and will stop reacting to you from afar.

    Parameters Description
    SpeedX Horizontal rotation speed.
    SpeedY Vertical rotation speed.
    InverseX To inverse the horizontal rotation direction.
    InverseY To inverse the vertical rotation direction.
    RotationThreshInDegrees If the angle between 'Gaze to Target' and 'Camera to Target' is less than this value, do nothing. This is to prevent small jittery rotations..
    MinRotX Minimum horizontal rotation angle. This is to limit the rotation in different directions.
    MaxRotX Maximum horizontal rotation angle. This is to limit the rotation in different directions.
    MinRotY Minimal vertical rotation angle to limit the rotation around the x axis.
    MaxRotY Maximum vertical rotation angle to limit the rotation around the y axis.

    Eye-supported 3D rotation setup in Unity

    In summary, the above scripts should enable you to get started to use eye gaze for various input navigation tasks such as scrolling texts, zooming and panning textures as well as rotating investigating 3D holograms.

    Related Topics

    • Basic MRTK setup to use eye tracking
    • Eye-supported target selection

    Back to "Eye tracking in the MixedRealityToolkit"

    • Improve this Doc
    In This Article
    • Auto Scroll
      • ScrollRectTransf
      • EyeTrackingTarget
    • Gaze-supported Pan and Zoom
    • Attention-based 3D Rotation
      • Related Topics
    Back to top Generated by DocFX