Overview
Tesla has long promised a fleet of robotaxis that operate without human supervision—vehicles that can pick up passengers, navigate complex urban environments, and park themselves, all while generating revenue for their owners. However, the company has repeatedly missed self-imposed deadlines for a full-scale rollout. Recent reports, including one from CleanTechnica, have highlighted that while the targets were massively missed, there is tangible progress. A reader, Ole Laursen, pointed out that some unsupervised robotaxis are indeed being deployed—though at low volume and with considerable limitations. This guide provides a technical deep dive into how Tesla’s unsupervised robotaxi system actually works, from sensor fusion to deployment logistics, and what hurdles remain.
Prerequisites
Before diving into the step-by-step implementation, it helps to understand the foundational technologies and assumptions Tesla uses:
- Hardware Foundation: Tesla vehicles equipped with Hardware 3 or Hardware 4, including eight cameras, twelve ultrasonic sensors, and a forward-facing radar (on some models). Notably, Tesla has removed radar and LiDAR from more recent cars, relying entirely on camera-based vision.
- Software Stack: Tesla’s Full Self-Driving (FSD) beta software—version 12 or later—which uses end-to-end neural networks trained on millions of miles of real-world driving data.
- Network Connectivity: Constant connection to Tesla’s cloud infrastructure for map updates, edge-case training, and fleet learning.
- Regulatory Approvals: Unsupervised operation requires permits in specific jurisdictions (e.g., California, Texas, Nevada).
- User Account Setup: A Tesla owner must opt into the Robotaxi program via the Tesla app, allowing the vehicle to be dispatched when idle.
Step-by-Step Instructions
1. Perception: Building a 3D World from Cameras
Unlike competitors (Waymo, Cruise) that use LiDAR, Tesla’s perception stack is entirely camera-based. The system processes eight video streams at 36 frames per second through a transformer-based architecture called Occupancy Networks. Here’s a simplified example of how a perception loop might be structured in code (using pseudo-Python):
def perception_loop(camera_frames):
# Each frame is a 1280x960 RGB image
features = extract_features(camera_frames) # CNN backbone (e.g., RegNet)
occ_grid = occupancy_network(features) # 3D voxel grid
objects = detect_dynamic_objects(occ_grid) # Cars, pedestrians, etc.
road_graph = predict_lane_geometry(occ_grid, semantic_labels)
return occ_grid, objects, road_graph
The output is a bird’s-eye-view representation of the environment, including static obstacles, moving agents, and drivable area. This step is critical because any error in perception—like misclassifying a shadow as a pedestrian—can lead to unsafe behavior.
2. Planning: From Route to Motion Commands
Once the environment is understood, the planning module computes a safe, comfortable trajectory. Tesla uses a hybrid approach: a global planner (for route from A to B) and a local planner (for collision avoidance and lane-keeping). The local planner runs on a receding horizon, typically 8 seconds ahead, solving an optimization problem:
def local_planner(state, obstacles, goal):
# state: [x, y, v, heading]
# obstacles: list of (x, y, w, h, v) from perception
# goal: final waypoint
for t in horizon:
cost = w1 * distance_to_goal + w2 * jerk + w3 * collision_risk
# use apollo-like polynomial curve fitting
candidate_path = generate_path(state, end_state)
return select_best(candidate_paths, cost)
Note that Tesla’s actual planner may be entirely neural-network-based (imitation learning from human driving). The key requirement is that the planning module must handle rare edge cases (e.g., construction zones, emergency vehicles) without human intervention.
3. Control: Executing the Plan
The trajectories from the planner are converted into steering, acceleration, and braking commands. Tesla’s control system uses a combination of PID controllers and model predictive control (MPC). A simplified steering controller might look like:
def steer_control(target_curvature, current_yaw, speed):
error = target_curvature - current_yaw
# PID with feedforward
steer_cmd = Kp*error + Kd*derivative(error) + feedforward(speed)
return clamp(steer_cmd, -MAX_STEER, +MAX_STEER)
The controller must also handle emergency braking (AEB) and traction control—functions already present in production Teslas. For unsupervised operation, redundant fallback mechanisms are required (e.g., if the main computer fails, a secondary processor can safely halt the vehicle).
4. Validation and Self-Checking
Before each ride, the car runs a suite of diagnostic checks: camera calibration, sensor integrity, software version compliance, and network latency. If any check fails, the vehicle returns to a safe state and notifies the fleet server. During operation, the system monitors its own uncertainty via a confidence network:
def safety_check(perception_conf, planner_conf, control_conf):
if any(conf < THRESHOLD):
# attempt minimal risk maneuver (pull over, stop)
execute_minimal_risk_maneuver()
Tesla has also introduced a “shadow mode” where the robotaxi software runs in parallel with human driving, but never takes control. Discrepancies are uploaded to Tesla’s server for training.
5. Deployment and Fleet Management
The final step is actual deployment. A Tesla owner lists their car on the network via the app. The fleet management system assigns rides based on vehicle location, state of charge (SOC must be above 100 miles remaining), and rider ratings. The car navigates to pickup, identifies the passenger via Bluetooth phone key, and drives them to the destination. Payment is processed from the rider’s account. Currently, the rollout is limited to a small fleet—likely tens of vehicles—in a few cities. The reader Ole Laursen noted that some cars are indeed operating unsupervized, but volume is far below the 1 million promised by CEO Elon Musk.
Common Mistakes
- Overreliance on Occupancy Networks: Without LiDAR, the system can misjudge distances in poor lighting or adverse weather (rain, fog). Tesla’s solution is to train on synthetic data, but this is still an active area of research.
- Ignoring Edge Cases in Planning: Many robotaxi trial runs have failed due to unexpected road closures or construction. A common mistake is not including a robust rerouting mechanism that works without human input.
- Inadequate Self-Monitoring: Some early prototypes would continue driving even when a camera was occluded. Modern versions now detect blockage and pull over immediately.
- Network Latency Issues: Fleet management servers occasionally suffer delays, causing a robotaxi to arrive late or miss a pickup. Redundant cellular connections (dual SIM) help, but costs increase.
- Regulatory Hurdles: Companies often underestimate the time needed to get permits for unsupervised operation. Tesla’s recent limited rollout suggests they have received conditional approval, but full expansion requires proving safety over millions of miles.
Summary
Tesla’s unsupervised robotaxi achievement—though small in scale—represents a significant technical milestone. The system uses camera-only perception, end-to-end neural planning, and redundant control to navigate without a human behind the wheel. However, the same problems that caused earlier missed targets persist: low volume, incomplete edge-case handling, and regulatory constraints. As of now, only a handful of robotaxis are operating, but the architecture provides a blueprint for scaling to full autonomy. For developers and enthusiasts, understanding each component—from occupancy networks to safety checks—is essential to appreciating what works and what still needs improvement.