How do you teach two very different robots to follow one person?

01 · Origins

It started as a robot that could play tag

Before any of the multi-robot work, the goal was simple: build a robot that could play tag with a person. That splits into two problems: put the robot and the person in the same coordinate frame, then decide how to move toward them.

The first platform was a TurtleBot with a 2D LiDAR and a forward-facing monocular camera. The camera detected the person and gave a bearing; the LiDAR gave a precise range, but only out to 4 meters. Past that, I estimated distance from the size of the person's bounding box in the image.

Camera-only range estimate against ground truth as a person steps toward the robot (thesis Fig. 4.3). Because it assumes typical human proportions, it was the fallback to the LiDAR, not a replacement.

Then the team grew. A palm-sized Bitcraze Crazyflie joined, at first with no camera at all. Once the two robots' coordinate frames were aligned, the ground robot could localize a person and send the drone to their position. That idea, one robot's perception tasking another robot, is the seed of everything that follows.

The hardware then grew up: the TurtleBot gave way to a Clearpath Jackal with a 3D LiDAR and an RGB-D camera, and the Crazyflie gained a small grayscale camera. With perception on both robots, the question changed from "can they find a person?" to "how should two very different robots share the job of following one?"

02 · The Problem

Robots that follow people are useful — until the person turns a corner

A robot that can autonomously follow a person is a building block for a lot of real missions: hauling gear for a search-and-rescue team, assisting patients, carrying supplies for soldiers, or providing social assistance.

Most existing systems quietly assume the person will cooperate: walk at robot speed, stay in the open, glance back to make sure their robot buddy is keeping up. In high-stakes settings, nobody has time for that. The person moves how they move — and the robot has to deal with three hard constraints:

03 · The Idea

Pair an odd couple: a thinker and a scout

No single robot solves all three constraints at once. So the thesis pairs two robots whose strengths are exactly each other's weaknesses, plus the person they're trying to keep up with:

The drone becomes the ground robot's eyes around the corner: it races to wherever it can see the person and streams their position, while the ground robot uses that feed to plan the smartest possible route — even a route that temporarily gives up its own view.

04 · Three Strategies

Who decides where each robot goes?

The heart of the thesis is a bake-off between three motion strategies — a hand-engineered forecaster, a learned multi-agent policy, and a deliberately simple baseline.

Strategy 1 · FORWARD-PF (heuristic)

The forecaster: predict, score, assign

FORWARD-PF (Forecasting Optimization of Robotic Way-point Assignment for Robust and Dynamic Person-Following) treats the problem like a chess player thinking one move ahead:

1

Model the person's motion

Fit a third-degree polynomial to the person's recent positions — a smooth curve describing where they've been heading.

2

Project it forward

Extend the curve a few seconds into the future, then scatter Gaussian samples around each predicted point — a cloud of "places they might plausibly be."

3

Score a grid of candidate spots

Lay a grid of candidate goal locations around the person. Score each one twice: for the follower, closeness to the predicted position; for the observer, how many predicted points it can see (line-of-sight), minus travel cost.

4

Send each robot to its best spot

The follower gets the best proximity cell, the observer the best visibility cell — unless the spot each is already standing on scores higher, in which case: don't move. Repeat every cycle.

Every cycle, a grid of candidate goals is scored around the person. Green cells promise proximity or a clear view of the predicted motion cloud; the best cell for each robot becomes its next waypoint.

Under the hood: the scoring math

Tracking the person

Camera detections are projected onto the shared map and filtered with an extended Kalman filter over position and velocity to suppress sensor noise.

Motion model

A cubic polynomial P(t) is fit to the filtered positions over a sliding position-history window, then sampled once per second out to the time horizon z. Each predicted point seeds 10 Gaussian samples (σ = 0.9 m, tuned to the obstacle spacing), giving the motion cloud F_p(z).

Follower score (lower is better)

Distance from candidate cell c to the center of the predicted cloud. In simulation, the goal was to get within a 5 m buffer of the person. follower(c) = ‖ c − avg(Fp(z)) ‖₂

Observer score (higher is better)

How many predicted points are visible from c (raycast against the live occupancy grid), minus how far c strays from the preferred stand-off distance d_s (10 m in simulation). observer(c) = los(c, Fp) − | ds − ‖ c − avg(Fp(z)) ‖₂ |

Keeping the camera on target

While it moves, the drone yaws to face the person's current position. θgoal = atan2(Ty − Uy, Tx − Ux)

Avoiding dithering

Each cycle, a robot's current goal is re-scored against the updated map and person position. It is only re-tasked if a new candidate actually beats it, so the robots don't twitch between near-equal cells.

What the observer is optimizing (thesis Fig. 5.4). The light-blue cone is the drone's current view; the dark-blue circle is its chosen goal and the view it will have from there. Black dots are the person's predicted positions, and the red circle is the ground robot's goal.

FORWARD-PF running in simulation: candidate goals colored from green (best) to red (worst), the blue circle marking the selected goal as the target moves.

Strategy 2 · Multi-Agent RL

The learner: let the robots figure it out

The learned approach uses MADDPG — Multi-Agent Deep Deterministic Policy Gradient — with centralized training and decentralized execution. Three agents train together in a simple 2D particle world (PettingZoo's Multi-Particle Environment): a target that learns to evade, a follower rewarded for closing distance, and an observer rewarded for keeping line-of-sight, all penalized for collisions.

The catch: policies learn in a flat world of frictionless holonomic particles, then must drive a real differential-drive robot through 3D clutter. Bridging that gap — smoothing discrete actions, simplifying observations, rescaling coordinates — was half the battle.

Each agent observes its own position and velocity, the other agents' positions and velocities, and nearby obstacles. Its action is one of five discrete moves: stay, up, down, left, or right. Training is the easy part. Getting a policy that learned on frictionless 2D particles to drive a real differential-drive robot through 3D clutter meant closing three gaps:

Gap 1 · Actions

Discrete moves → smooth driving

The policy picks a discrete move 10 times a second, which a differential-drive robot can't execute directly. For the ground robot, I averaged the last second of actions into one direction vector, so alternating "right" and "up" becomes up-right. That vector is projected 5 m ahead and handed to the local planner. The holonomic drone executes moves directly.

Every agent works in the world frame, so "up" means the same thing whichever way the robot faces. The local planner also catches small obstacles the LiDAR saw inconsistently, which had been causing collisions.

Gap 2 · Observations

Any number of obstacles → a fixed-size input

Training expected the position of every obstacle, so a map with 11 obstacles broke a model trained on 10, and training time grew exponentially with obstacle count. A "10 closest obstacles" list didn't work either, because its order reshuffled as the robot moved.

The fix was to split the 360° around the robot into 10 sectors and report only the nearest obstacle in each, with its distance bucketed into 5 proximity levels. The input is the same size in any environment.

Gap 3 · Scale

A 2 × 2 training box → a 100 m world

The model trained in a 2 × 2 unit space with coordinates normalized to [−1, 1]; the deployment world was roughly 100 × 100 m. Positions were scaled down and offset from the experiment's starting point, then hand-tuned by about 10%.

Scale matters more than it looks. Shrink a cluttered world too far and the model perceives obstacles as closer and its own moves as less consequential. Past a certain size and obstacle density, performance degrades however you tune it.

The adapted observation in the 3D simulator (thesis Fig. 4.12). Spheres mark the nearest obstacle in each of the 10 sectors around the ground robot (left) and the drone (right). The green arrow is the averaged action sent to the planner.

05 · The Gauntlet

Nine paths, three levels of chaos

All three strategies ran in a Unity simulator driven by Phoenix, the Army Research Laboratory's ground autonomy stack, which handles person detection and tracking, online SLAM, and global and local planning. A simulated Clearpath Warthog played the follower and a generic quadrotor the observer, each with a forward camera that detects the person out to 20 m.

In each of three areas, the simulated person walked an easy, a medium, and a hard path. Difficulty is measured by tortuosity, the path's length divided by the straight-line distance from start to end: about 1.0 for easy, 1.3 for medium, and 2.0 for hard. That's 9 paths × 3 methods = 27 configurations, each run repeatedly. The robots start side by side, and the clock starts when the person first enters their view.

The full test course (thesis Fig. 4.8). Blue circles are the three starting locations, white circles mark where each path ends, and the line style shows difficulty: dashed easy, dot-dashed medium, solid hard.

06 · Results

The forecaster beat the learner — and both crushed the baseline

The visibility numbers tell the underlying story. When the drone keeps the person in sight, the ground robot always knows where to go; every MARL failure came on runs where visibility collapsed below 20%.

Fraction of each run the person stayed visible to the robot team, per path. * marks failed runs.

The baseline deserves some credit. On the three easy paths it finished, it had the shortest arrival delay, the shortest route, and the smallest gap to the person, because a purely reactive controller moves the instant it sees the person while the two-robot methods pay for planning time. Its problem is recovery: once it loses sight, it's done.

Seconds between the person stopping and the ground robot arriving, per path (thesis Appendix A). Bold marks the fastest finisher on each path; — marks a method that didn't complete it. Averaged per path, FORWARD-PF arrived 1.95× faster than MARL.

The unexpected bonus: shortcuts

Something emerged that wasn't programmed in. Because the drone keeps reporting the person's position, the slow ground robot doesn't have to retrace the person's winding path. It can cut the corner and plan the shortest route to wherever they are now. Both two-robot methods found this on path 3, FORWARD-PF found it again on path 9, and it showed up in the hardware validation too.

The real thing (thesis Fig. 5.9): person in yellow, drone in blue, ground robot in red. The drone traces the loop to keep the person in view while the ground robot cuts through the middle.

07 · Off the Screen

Real robots, real lab

FORWARD-PF also ran on real hardware: a Clearpath Jackal running the same Phoenix autonomy stack, with a Velodyne VLP-16 LiDAR for SLAM and an Intel RealSense camera for detection, paired with a palm-sized Crazyflie 2.1 drone.

The hardware validation side by side with RViz (thesis Fig. 4.13): person in yellow, Jackal in red, Crazyflie in blue. The green-to-red shading is FORWARD-PF scoring candidate goals on the Jackal's live map.

08 · Takeaways

What I'd tell you over coffee