Tesla FSD Cannonball Run 002: v13 Better, But Hardware Has Limits
Software eats the world, but hardware drives it.
By Alex Roy, General Partner, NIVC
In December 2024 we announced the first Tesla Full Self-Driving (FSD) Cannonball Run record, inspired by Elon Musk’s claim that Teslas would be capable of driving coast-to-coast without human intervention by the end of 2017. In order to determine how close Tesla is to that goal, we began testing successive generations of FSD on the traditional Cannonball Run route between midtown Manhattan and Redondo Beach, California.
Interior cockpit @ night. FLIR display @ top center, Waze @ bottom left, Highway Radar @ bottom right.
Drive 001: The Tesla FSD Cannonball Run Record
On Drive 001 we tested FSD to set what we call the Production Autonomy Cannonball Run record, creating the benchmark for future testing. The key metrics are Engagement Rate and Disengagements:
12/13 - 12/14/2024
Hardware 4
Software v12.5.6.4
Distance (Miles): 2,833
Engagement Rate: 98.52%
Disengagements (Total): 32
Drive 002: Did Not Finish (DNF)
On Drive 002 we tested FSD v13.2.2, which showed improvements over v12, but was unable to operate effectively and consistently in moderate rain. The drive was cut short for safety 836 miles from the finish line.
12/25 - 12/27/2024
Hardware 4
Software v13.2.2
Distance (Miles): 1997 — DNF
Engagement Rate: 99.51%
Disengagements (Total): 14
Given how well the vehicle performed in the storm after we called off Drive 002, the Engagement Rate on the entire (uncompleted) trip would likely have been below 70%. The DNF negates the 99.51% Engagement Rate, but provides a perfect boundary for understanding the Operational Design Domain (ODD) of FSD 13.2.2 running on Tesla’s latest hardware (HW4). More on ODDs at the end.
Drive 002 Car
Drive 002 used the same vehicle as 001: a 2025 Tesla Model S with HW 4. The only significant addition was for safety: a pre-production FLIR thermal camera, which makes it possible to see objects the Tesla forward-facing cameras may not, and simplifies post-drive disengagement analysis.
A full list of aftermarket hardware and accessories is available here.
Tesla Cannonball Run Drive 002 co-driver Joel Johnson
Drive 002 Team
The Drive 002 co-driver was Joel Johnson, a former network engineer and media executive with extensive racing experience.
Drive 002 Route
Whereas Drive 001 was East-to-West, Drive 002 was West-to-East. Although this may have some minor effect on efficiency, it has no effect on the metrics that measure FSD progress toward unsupervised operations.
Drive 002 Weather
On a traditional Cannonball Run attempt — where time is the only metric of success — weather must be avoided. When testing FSD progress toward unsupervised operation, weather is an opportunity to discover system limitations.
Whereas Drive 001 had clear-to-mixed weather, weather projections for Drive 002 showed a severe storm in the final third of the drive. We chose to proceed, for as Frank Herbert said, "To know a thing well, know its limits. Only when pushed beyond its tolerances will its true nature be seen."
Approaching the storm.
Tesla FSD Cannonball Run Drive 002 Data
The key metrics for a Tesla FSD Cannonball Run record are Engagement Rate and Disengagements, although the second number must be deconstructed to understand system performance over subsequent software releases.
Date: 12/25/2024 - 12/27/2024
Software: FSD v13.2.2
Engagement Rate: 99.51%
Disengagements (Total): 14
Distance (Miles): 1997 — DNF
Disengagements (Involuntary): 4
Disengagements (Voluntary, Safety-Related): 1
Disengagements (Voluntary, Navigation-Related): 1
Disengagements (Voluntary, Charging): 8
Illegal Maneuver: 1
System Resets (Navigation-related): 0
System Resets (Other): 3
Drive 001 vs Drive 002 Comparison
Since Drive 002 was 836 miles shorter than Drive 001, a fair comparison must focus on incidents and disengagements per thousand miles.
How we classify Disengagements (DE’s):
(IVDE) Involuntary
(VDE) Voluntary, for Charging
(VDE) Voluntary, Safety-Related
(VDE) Voluntary, Navigation-Related
Drive 001 vs Drive 002 Comparison: Incidents per 1k miles.
The Good: FSD v13.2.2 Is Better Than FSD v12.6.5.4
FSD v13.2.2 enabled autonomous entry and exit from Tesla Superchargers, dramatically reducing the volume of VDE’s for charging.
FSD v13.2.2 required far fewer VDEs for safety, specifically for lane incursions by trucks.
FSD v13.2.2 feels much smoother and more comfortable than v12.5.6.4. Alas, we didn’t bring a G-meter for granular measurement of ride quality. We will consider this addition to future runs.
Cabin Interior ~200 miles into trip. Note IVDE warning in instrument cluster at left.
The Par: Involuntary Disengagements
FSD v13.2.2 had twice as many IVDEs as its predecessor, but two of these were related to weather not encountered on Drive 001.
Drive 002 still had two unexplained IVDEs, in clear, low-complexity scenarios.
Both unexplained IVDEs required system reboots for FSD functionality to return, which also occurred on Drive 001.
Tesla Nav screen indicating moderate rainstorm, which precipitated multiple IVDEs, and drive cancellation for safety.
The Limiting Factor: Tesla HW4 Cameras
Upon hitting the storm in Indiana, IVDEs became so frequent that FSD was effectively unusable, and the drive was cancelled.
These IVDEs were accompanied by camera obstruction and system error alerts.
One IVDE was triggered by manual activation of the windshield wipers.
Tesla is aware of weather-related camera issues, and recommends periodic precision cleaning of the forward-facing cameras. This may mitigate and/or postpone visibility issues and IVDEs, but will not eliminate them.
Tesla has also been improving camera hardware with almost every new model release, but the performance limitation of current hardware forces clarification of when and where current (and older) Teslas will be capable of unsupervised FSD operations, regardless of future FSD software improvements.
Amateurs Talk Geofences, Engineers Talk Operational Design Domain
Some Tesla fans mock Waymo — the only unsupervised driverless commercial robotaxi service currently operating in the USA — because Waymos operate within strictly defined geofences in three cities, and Tesla FSD operates nationwide, albeit with onboard supervision.
Only amateurs talk geofences. A geofence is a geographic boundary inside which an autonomous vehicle (AV) operates. A geofence is but a single data point in evaluating AV capabilities. To claim a system is “better” merely because it lacks a geofence suggests ignorance of the other performance dimensions necessary for commercializing autonomy.
AV developers talk Operational Design Domain, or ODD. An ODD includes a geofence, and boundaries including (but not limited to) complexity, lighting, time, and weather. An ODD may be elastic, with one or more dimensions rapidly compressing as conditions evolve. Having an ODD makes it possible to create operational policies that maximize vehicle and passenger safety, and the latter’s comfort.
For example, in perfect weather a robotaxi might operate anywhere in a city, but an impending storm could trigger the fleet’s return to base, and a pause in passenger pickups. No one wants to be stranded in an immobilized AV, or dropped off anywhere but their destination.
In a privately owned AV, a passenger would be alerted to the storm minutes before system deactivation, such that the driver could take over and proceed, turn around, or stop without risk or stress.
Waymo has an ODD, of which their geofence is the only component shared with the public. If you want to learn more about Waymo's ODD, go to SF during a bad storm, open the Waymo app, and see if any robotaxis are available.
Tesla FSD has an ODD, but nothing about it is public. Tesla owners feel FSD’s contours from time to time, but only discover its hard boundaries in extreme conditions.
Conclusion
Tesla’s FSD software is making slow but methodical progress toward unsupervised operation. Given the obvious improvements in Tesla FSD performance from v12 to v13, I am confident that software is not the bottleneck in current (HW4) vehicles. It’s the camera hardware. If I am correct, then once FSD is capable of unsupervised operation:
Current (HW4) Teslas will likely have an unsupervised FSD ODD boundaried by moderate rain/snow
Next-Gen (HW5) Teslas will likely have an unsupervised FSD ODD boundaried by heavy rain/snow
HW5 Teslas will be able to operate unsupervised at higher speeds than HW4 (or earlier) vehicles, in the same ODD
Next Steps: Tesla FSD Cannonball Drive 003/004
Our Model S has just completed refitting for Drive 003, with multiple additions/upgrades:
FLIR Prototype Thermal Camera + Display Upgrade
Nexar AI-Enabled Dashcam
Point One Navigation Hi-Resolution GPS
And of course, FSD 13.2.5 has been installed.
Next report in March!
For additional information/questions, please e-mail me.