Vehicles
Vehicle RF-DETR Small
An object detection model that finds vehicles and sorts them into nine Japanese types, including cars, vans, trucks and buses. Combined with tracking on fixed intersection cameras, it counts vehicles entering and leaving by road, and which road they went from and to.
How vehicles are counted from video
- 1
Find vehicles
Vehicles are detected in each frame, keeping boxes scoring 0.3 or higher. Overlapping boxes of different classes on one vehicle are merged.
- 2
Link across frames
Tracking (ByteTrack) links each box to the same vehicle in the previous frame. If a track ID changes midway, tracks that vanish and reappear close in place and time are joined into one vehicle.
- 3
Count vehicles crossing gates
A gate is placed across the mouth of each road into the intersection. When the bottom center of a vehicle's box crosses a gate inward, it counts as one entry from that road; outward, one exit. The entry and exit gates also show which road it went from and to.
Input and output
- Detects
- Vehicles, in nine classes: car, van, kei truck, small truck, medium/large truck, trailer, bus, motorcycle and special vehicle (construction and work vehicles). The release includes a table for rolling these up into the 4 categories (car, small freight, bus, regular freight) and 2 categories (small, large) used in Japanese traffic surveys. Bicycles and people are not covered.
- Base model
- RF-DETR Small (public weights pretrained on COCO, Apache-2.0), fine-tuned on images taken on Japanese roads.
- Input
- A 512 × 512 pixel color image (RGB).
- Output
- 300 candidates per image, with positions (as 0 to 1 fractions of the image), scores (confidence, 0 to 1) and classes. No duplicate suppression (NMS) is needed.
- Formats
- PyTorch, ONNX
- Threshold
- 0.45. Candidates scoring at or above this count as vehicles. It was set on the validation images for the best balance of misses and false positives.
- Counting
- Detected vehicles are linked across frames by tracking (ByteTrack), and each one that crosses a line on screen or a gate placed on each road at the intersection is counted once. The code is published in the same place.
- License
- Apache-2.0
Evaluation
| Metric | Test (429 images, 1,369 vehicles) |
|---|---|
| AP50 (detection accuracy across all nine classes) | 0.642 |
| Precision (share of detections that were correct) | 0.843 |
| Recall (share of objects found) | 0.752 |
| Recall (share detected as a vehicle, any class) | 0.820 |
| Recall (car) | 0.816 |
| Recall (bus) | 0.813 |
| Recall (medium/large truck) | 0.576 |
| Recall (motorcycle) | 0.561 |
| Recall (van) | 0.508 |
| Entries per road counted correctly, 3 minutes of intersection video | 90% (93 of 103) |
| Exits per road counted correctly, same video | 94% (98 of 104) |
| Origin and destination road correct, same video | 79% (72 of 91) |
Detection figures are at a threshold of 0.45 (except AP50). The test images come from eight regions not used in training and were used for this evaluation only once. The threshold was fixed before looking at them. Test annotations were made by AI, with the rarer classes checked by the developer, not reviewed image by image. Kei trucks, small trucks, trailers and special vehicles each have fewer than 30 in the test set, so their figures are not reported. Counting figures compare against a reference count of vehicles crossing each road's gate in a 3-minute daytime recording of the Shibuya Scramble Crossing live camera, made by AI from the frames without seeing the model's output.
Notes on use
- Small vehicles (under 32 × 32 pixels) are missed more often, as are motorcycles in footage shot from high up.
- Vans are sometimes classed as cars (test recall for vans: 0.508). Kei trucks, trailers and special vehicles had few training images and are missed more often than other classes.
- There are cases where an advertising truck with an illuminated body was not detected at night.
- When vehicles pass each other at a busy intersection, a track ID can jump to another vehicle and throw off the origin and destination. The same happens with vehicles hidden behind large ones.
- Do not base traffic survey or safety decisions on this model's output alone.
Start by describing your situation
Whether it is a MICHIKARTE trial or an AI app, your plans do not need to be final. Describe the photos or video you have and what you are struggling with, and you will get an email reply on what can and cannot be done. The contact form is in Japanese, but you are welcome to write in English.