Tesla Wins Patent for Finding Rare Driving Moments to Train AI

Tesla has been granted a U.S. patent credited to former AI chief Andrej Karpathy that tackles a practical problem for training driving software: picking out useful moments from vehicle sensor data instead of sending everything back for review.

The patent, titled “System and method for obtaining training data,” was issued on November 10 as US 12,792,972 B2. It lists Karpathy as the sole inventor and Tesla as the assignee, bringing work from his time leading the company’s AI efforts into a newly granted patent.

Karpathy left Tesla in July 2022. During his tenure, Tesla was expanding the computing infrastructure behind Autopilot training, making the question of which driving examples to collect an important companion to the hardware doing the learning.

How the data collection works

The first claim describes a neural network processing sensor data about a vehicle’s surroundings. The system checks an intermediate result inside that network for a trigger condition indicating it cannot make a predetermined classification.

When that condition is met, a second neural network runs a “trigger classifier.” Trained away from the vehicle using earlier intermediate results, this classifier scores whether the data matches a target object, scenario or feature. The system compares that score with a threshold to decide whether to upload some of the associated sensor data.

The selected data can then be used to retrain the original neural network or a different one. In plain terms, the approach describes a way to look for particular training examples within the stream of information a car already processes.

That is useful context for understanding fleet learning: collecting more miles and finding examples that address a specific weakness are different challenges.

The record lists 20 claims and traces priority to September 14, 2018. The international application was filed in September 2019, and the U.S. application was published in September 2021, so the underlying idea has been public for years.

Today’s development is the patent grant. It does not establish when Tesla deployed the method or demonstrate an improvement in FSD performance.

Which difficult driving situations would you most want Tesla’s training systems to focus on?