Gesture-Based Computer Control

Entirely computer vision based control of computer, including cursor, keyboard, and scrolling

Summary

This project was built for two target experiences- as a presentation tool and as an accessibility tool. As a presentation tool, it excels of keeping a presenter from needing to hide themselves behind a computer in order to do anything from it. In front of large audiences, being able to control the mouse by pointing, type with sugn language, and scroll with two fingers completely changes the way tech can be utilized in a presentation. As an accessibility tool, though it is not particularly helpful to the deaf community (typing is faster), it is helpful to be able to type from farther away, and it is very intuitive to point to the screen with your finger instead of touching it or using a mouse. The project began with a motion capture glove to isolate the gesture recognition techniques from stable motion data and then transferred onto a pre-trained computer vision model.

Key Contribution: Designed and implemented a fully computer vision-based computer control including cursor movement, typing, and scrolling through mid-air hand gestures. Prototyped gesture semantics using motion capture for ground-truth stability, then translated the interaction model to a camera-only computer vision pipeline suitable for presentations and accessibility-focused use cases.

Skills

Touchless Interaction Design ▪︎ Gesture-to-Action Mapping ▪︎ Accessibility-Oriented Interface Design ▪︎ Computer Vision Model Deployment ▪︎ Real-Time Interaction Pipelines ▪︎ Rapid Interaction Prototyping ▪︎ Python ▪︎ Numpy ▪︎ Tensorflow ▪︎ OpenCV ▪︎ C++

You may also be interested in:

F.R.I.D.A

A Star Wars-inspired animatronic with face tracking eyes and LLM-enabled communication

Ocean Site One

Lead Engineer for an International Event, Turning a Large Format 360° Video into a Touchscreen-Like Explorable World Using Computer Vision

Founder

Inventor of Core Patent-Pending Technology of a Speaker Manufacturing and IP Company

‘Deep Gerchberg-Saxton’: A Physics Informed Neural Network Architecture

A novel deep learning approach to Volumetric Phase Retrieval problems; applied to immersive displays

Life in India

A documentation of my time and life at a university in Bangalore, India.

Pastel and Mixed Media Impressions

A collection of a recent movement in my art: impressionistic and figurative works with oil pastel, ink, and water soluble pastel.

Top

Nathan Gollay Portfolio