Detecting potholes with AI from a smartphone mounted in a car

Model Lab has released a detection model that finds potholes in images of the road ahead taken by a smartphone mounted in a car. It detects one class, potholes, and outputs each hole's position and a score (confidence, 0 to 1). The weights are published for developers under Apache-2.0.

A hole on a narrow, cracked road and a hole on a shaded narrow road, each outlined in a red box with scores of 0.84 and 0.83 — Photos: RDD2022 (Arya et al., CC BY 4.0)

Why find potholes?

A pothole is a hole where the pavement has broken away. Left alone it grows, and can cause flat tyres or bring down a motorcycle. Road managers find them by eye from patrol cars or through reports from residents.

During patrols there are times when someone has to keep watching the road while driving and note where each hole is. If a smartphone mounted in the car automatically keeps the images that show a hole, the photos and locations can be reviewed later to check nothing was missed.

Who is it for?

Patrol and repair staff at national, local and expressway road managers
Companies contracted for patrols and pavement surveys
Developers building services that collect road footage from in-car cameras or dashcams

How does it find them?

The model is based on RF-DETR Nano. Public weights pre-trained on COCO were further trained on pothole images.

A photo of a shaded road is stretched into a square, changing its aspect ratio, to become 384 × 384 — Photo: RDD2022 (Arya et al., CC BY 4.0)

Each image is stretched to a 384 × 384 pixel square before it goes into the model. Detections with a score of 0.45 or higher count as results. Raising this threshold reduces false positives; lowering it reduces misses.

How well does it find them?

Counted on 29 test images (44 holes) not used in training. Precision is the share of detections that were correct; recall is the share of holes that were found.

AP50, a summary measure of detection accuracy, was 0.737. With only 29 evaluation images, the numbers can swing widely. The threshold was chosen after seeing the test results, so the test figures may be better than real performance. Performance while actually driving has not been measured yet.

Left: a gravelly road where the hole on the left was found at 0.81 and a small hole on the right was missed. Right: a wet road where the hole at lower right was detected at 0.61 and a dark wet patch on the left was falsely detected at 0.60 — Photos: RDD2022 (Arya et al., CC BY 4.0)

Small holes can be missed, and wet road surfaces that look dark can be mistaken for holes.

License and formats

The weights and code are released under Apache-2.0. They can be built into business systems and products, and modified and redistributed. Three formats are available: PyTorch, ONNX and CoreML (iOS 17 or later); with CoreML it runs on an iPhone. The PyTorch weights load directly with the rfdetr library, so they also work as a starting point for training on your own data.

Things to keep in mind

It only detects potholes, not cracks or other damage. Performance at night, in rain or against the light has not been checked. Test it on images from the place you will use it before building it in. Do not make inspection or safety decisions on this model's results alone.

Model details (inputs and outputs, evaluation results): Pothole RF-DETR Nano

Weights: Hugging Face

Apps that use models like this to find road damage or equipment in video can also be built on request: AI App Development

Cover photo: RDD2022 (Arya et al., CC BY 4.0)