Reach
An interface between attention and action.
Reach explores how people with limited mobility can direct robots through gaze and EEG-based selection. Physical robot demonstrations and a separate retrieval simulation investigate different parts of that interaction.
Physical Go1 demonstration · external camera recording
01 / The interaction
How can an intention become a robot action?
In the physical Go1 setup, a custom eye tracker identifies a candidate in the robot’s camera view. A visual EEG target lets the wearer confirm it. The operator reported that the robot then handled the approach without further steering.
The custom eye tracker and EEG decoder use separate calibration steps.
02 / Research & engineering
What went into it.
Gaze & calibration
An eye-facing camera and pupil detector estimate where the wearer is looking. Calibration maps pupil position to the display.
Telepresence
The robot camera streams into Unity and the VR headset. Hardware-accelerated video and a networked control path connect what the wearer sees to the robot’s actions.
Visual selection
Point-prompted segmentation isolates the candidate. A NextMind visual target provides EEG-based confirmation.
Robot control
Unity and network messages connect the interface to ROS and the Unitree controller. Direct movement commands were also available.
Object retrieval
A separate Isaac Sim scene uses a Rebot arm to pick up, present, return or put down objects. AR viewing and fixed-cursor selection are working.
Depth perception
An OAK-D Lite is mounted on the physical Rebot arm. Stereo and neural depth fusion are being explored to support object localization and grasping.
03 / Progress
Built. Tested. Next.
Gaze-to-approach on Go1
A physical setup connects custom eye tracking, NextMind selection and robot approach. The recording captures robot motion.
Retrieval in Isaac Sim
The simulated arm can pick and present objects through a fixed-cursor interface viewed in AR glasses.
Arm-mounted depth camera
The OAK-D Lite mounting establishes a hardware foundation for depth-informed grasping.
Navigation study
A numbered-flag course and aerial trajectory tracking provide an evaluation design for comparing intended and actual paths. Study results are not reported here.
Connect and evaluate
Integrate gaze and EEG into the arm simulation, and measure the full selection-to-motion loop.
The Go1 demonstration and arm simulation are separate implementations. The recording does not establish selection accuracy or end-to-end latency.
Continue exploring