F.R.I.D.A. (Friendly Imperial Droid Assistant)

A Star Wars-inspired animatronic with face tracking eyes and LLM-enabled communication

Summary

FRIDA is a droid from the ice planet Hath that has been reprogrammed to help out in a lab setting. She has full range of motion with her head, eyes, and two arms along with a quick witted personality that has real time GPT powered conversations. Her brain is a Raspberry Pi and her eyes are cameras, allowing her to look you in the eyes as you talk. She also has a mimicry feature through which you can control her arm and head movements with your own, through computer vision and inverse kinematics. 

I led her development in 2023, including her body modifications, movement design, personality design, installation, interaction, electronics, machine learning, computer vision, and programming. 

Skills

Professional Movie Studio Operation ▪︎  Show-Critical Technical Operations ▪︎ Technical Storytelling ▪︎ Creative R&D ▪︎ Motion Capture ▪︎ Licensing ▪︎ Donor/Investor Relations ▪︎ Lab Management ▪︎ Projectionist ▪︎ Dolby Atmos 

Details:

Face Tracking

FRIDA has the ability to look you in the eyes. She does this by using computer vision to locate your face’s location in relation to the current direction of her gaze. Then, she simply moves towards centering her gaze with the location of your eyes. Her motion towards you is smoothed with an exponential moving average.

Eyesight and Movement

In early iterations, her eyesight was a bit of an eyesore. Here, you’ll see a Logitech USB camera with blue lights acting as her vision. You’ll also notice that the smoothing function and response times were not quite optimized. 

Still, you can see her making her best effort to show good manners and look you in the eyes.

Deep Learning Model

FRIDA uses a lightweight deep learning computer vision model model that runs natively on the Raspberry Pi, with computation offloaded to a Google Coral.

It is by and large the most computationally expensive part of her personality.

Beyond purely technical, FRIDA’s eyesight and eye contact are integral to the immersion of interacting with a real life lab assistant. She looks, feels, and acts the part of a Friendly Reprogrammed Imperial Droid Assistant. 

Teleoperation (Mimicry)

Teleoperation, or mimicry, is a term used in robotics for controlling a robotic apparatus with your own movements. Ie, if you move your right arm up, the robot moves its right arm up, and so forth. FRIDA used her keen eyesight to mirror a user’s arm and head movements. If you wave, she waves back. If you look to the left, so does she. 

Pose Detection 

FRIDA makes use of Google’s mediapipe pose detection pretrained model. This secondary computer vision function is also relatively computationally expensive, and works only due to our optimization and offloading onto the Google Coral.

Here you can see my general pose outlined in red, with green points representing the joints. On the software side, I extract the angles between joints, determine the relative rotation matrices starting from the shoulders and apply inverse kinematics to allow FRIDA to mimic those same rotations with her arms and head.

Mirroring and Inverse Kinematics

There were some interesting design questions when it comes to designing the most immersive mimicry. After all, FRIDAs arms are not human arms, nor is every human arm the same approximate dimension. And, the last I checked, human heads cannot spin 360°. 

We decided that the ideal would be adhering to the closest possible visual representation of the human movement. To this end, the mirror of the arms and head are purely rotation based: a 30° rotation of the elbow joint corresponds to 30° in FRIDA. And, we limited her range of motion to whatever the mirrored subject’s flexibility allows them to do. 

To my recollection, we never had anyone double jointed or a skilled Yogi really put her to the test, but we can be pretty sure that FRIDA could out flex any guest. 

FRIDA’s mirror function solidifies her role as assistant. While she is Friendly first, the choice of including mimicry improves her utility. It’s also just really fun seeing an Imperial Droid mimic your movements.

Conversation And Personality (GPT)

FRIDA’s conversation is integral to her personality. She is able to have full conversations powered a OpenAI GPT- meaning you can talk to her as if she were a person, and she responds helpfully in the tone and knowledge base of her personality. It was essential her conversation personality it aligned very well with the character profile. Conversation, it turns out, is also among the most memorable features of her interaction for guests, in addition to being the most helpful in the lab. Thus, in addition to technical design, there is also an aspect of artistic design in shaping her word choices, tone, and cadence. 

Movement and Construction

FRIDA has three main parts of her that move, all of which combine to feel like you’re interacting with a fully autonomous entity. These three are her head and two arms. 

Her body is mostly an XPS foam construction, so she is surprisingly light weight. There are some metal and 3D printed accents that provide some structural integrity, allow for movement, and draw her whole appearance together to feel like she is entierly made from Star Wars universe metal. 

Movement

FRIDA’s articulation, like her other capabilities, was built with the intention of creating a magical experience. She can wave hello, look you in the eyes, and ask you about your day and for a moment you wonder whether you’re not actually on a rebel base talking to a reprogrammed droid. 

Electrical Hardware

FRIDA’s brain went through several iterations. In her most up to date configuration, the main processing is handled on a Raspberry Pi 5, with computation offloaded to a Google Coral, and some hardware interfaces passed off to Arduino’s and ESP32 via Serial, which have better PWM handling. Motion is driven by affordable stepper motors and drivers owing to the lightweight foam construction of the body. She also has a Raspberry Pi camera for sight, microphone for listening, and a built in bluetooth speaker for conversation in addition to various lights and controls. All of these come together to build FRIDA’s personality and interaction. 

Raspberry Pi 5

The Raspberry Pi, for those unfamiliar, is a brilliantly small computer running its own lightweight operating system with direct hardware interface connection inputs and outputs.

FRIDA asked use to get the top of the line, and we obliged with the Raspberry Pi 5 16gb ram with built in cooling and a large high speed micro SD. Also in consideration was the Nvidia Jetson Nano, but due to back orders, these were ruled out. 

The two most important criteria in picking her brain were parallel computation capabilities and modularity.

The Google Coral USB Accelerator has a powerful use case for compact intelligent systems. It being compact itself, it acts to increase the speed of parallel computation operations with its TPU coprocessor. Parallel computation essentially means doing many small operations simultaneously and is foundational to machine learning training and inference. 

In FRIDA, the Google Coral USB Accelerator offloaded the computational load of the computer vision processes, freeing up more CPU usage for other essential processes like movement and speech. 

Google Coral

ESP32’s and Arduinos

ESP32’s and Arduinos are the bread and butter of the microcontroller world. They allow for easy interface with lights and servo’s which are the purposes that they served in FRIDA- a middleman between the Raspberry Pi and the energy heavy devices. 

ESP32’s especially are incredibly robust in their Pulse Width Modulation (PWM) implementation, giving great control over the LEDs and stepper motors. Their bluetooth integration with external programs like  iOS apps is much more user friendly than even the Raspberry Pi itself. 

FRIDA’s main brain, the Raspberry Pi, communicates with these secondary microcontrollers via USB Serial. 

There are many types of Raspberry Pi cameras that connect to the Raspberry Pi 5 via a MIPI CSI-2 connection. In choosing the correct one, there are several important criteria: information throughput, focus control, and field of view (FOV). 

For those unfamiliar with computer vision, you may be wondering why resolution did not make that list- surely 1080p would be preferable to 720p. However, that is not the case. Computer vision applications will typically crop down images to increase efficiency. In fact, by choosing a camera with lower resolution, you may increase efficiency by reducing the number of data operations. 

Focus control is important because it is intimately involved with immersion. What distance from FRIDA will people interact with her? Will they stand close or talk to her from afar? With focus control, we can actually update the distance based on the size of the detected head in the frame, which correlates inversely with the approximate distance from the camera.

FOV is essential because it sets the information and interaction stage. Where do the people that FRIDA interacts with stand? Do they stand far out to the side while they’re working? Or does she only really need to help people directly in front of her? This question determines whether a narrow FOV or a wide FOV is preferable. In the end, I decided that a middle-wide FOV with software based automated control for scanning corners or focussing in narrow on someone was the best for FRIDA’s personality. 

Raspberry Pi Camera

SERVOS and Stepper Motors

Finally, Servos and Stepper Motors are responsible for all of FRIDA’s movement. 

FRIDA’s body is exceptionally lightweight, and so we could prioritize precision and affordability over torque. FRIDA’s head originally used a 270° rotation servo. Eventually, we changed it to a stepper motor for full 360° control, so that should someone walk all the way around her, she could follow them the whole way.

It also made for a great dance to “You Spin Me Right Round (Like a Record)” in which, you guessed it, she would spin her head… a lot. 

The choice of hardware, while technical, is founded in creativity. While interesting in itself, hardware is most valuable in its contribution to the art- to the animatronic. Every design decision on the technology was made to this end, to increase guest immersion and experience. 

Software

FRIDA’s software all runs natively on a Raspberry Pi. Essentially all of her programming is in Python in order to ensure maximum portability and compatibility with her machine learning capabilities.  Because of Python’s GIL lock, multiprocessing was esstnial to maximize compute resources. A driver program served as the frontal lobe, while several processes acted as eyes, mouth, ears etc, and several others acted as the unconscious nervous system moving motors and things like that. Built like hierarchical tree, all information is sorted, sent to, and delegated by the frontal lobe- the main driver process. 

Credits and Shoutouts

Most of what’s documented here is my contribution to FRIDA, but there is much more to acknowledge. 

FRIDA’s body design and first concepts of her personality were created by Andrew Gater. Andrew is extremely talented, worked very hard on FRIDA and without his original work, FRIDA would have never come into being. 

The team that worked with me on this: Casey Hartley, Trisha Guha, and Aidan Ream, were excellent and contributed tons to FRIDA as well.