Skip to content
All projects
CV/AIComputer vision · autonomous driving

Road Scene Segmentation

Pixel-level perception for autonomous driving.

Started
Jun 2026
Stack
PyTorch · SegFormer · Computer Vision · ONNX

01 · Problem

The problem

A self-driving stack needs to know what every pixel in front of it is: drivable road, sidewalk, a car, a pedestrian. The safety-critical classes are the small ones: pedestrians and traffic signs cover about 1% of the pixels and get swamped by road and buildings.

  • Compare classic and modern segmentation architectures fairly, in one pipeline.
  • Report honest numbers, on images the model never saw, at full resolution.
  • Make the predictions explainable and deployable, not just accurate in a notebook.

02 · Approach

How it works

  1. 01Fix the data before the modelAn audit found that masks were being remapped twice, which silently scrambled the labels, so the old headline score was meaningless. Now there is one label conversion, a 6-class scheme that Cityscapes actually annotates, void pixels are ignored, and 24 regression tests keep it that way.
  2. 02One pipeline, four modelsA model factory swaps U-Net from scratch (31M parameters), ResNet50-UNet (41M), DeepLabV3+ with ASPP (40M) and SegFormer-B2 with an ImageNet-pretrained transformer encoder (27M) over the same data, loss and metrics.
  3. 03A loss built for imbalanceCross-entropy for the per-pixel baseline, Dice so every class counts equally, and Focal loss (γ = 2) to down-weight the easy pixels.
  4. 04Evaluation you can trustCityscapes' test labels aren't public, so four cities are held out for validation and the official val set (500 images) is the untouched test set, scored at full 1024×2048 resolution.
  5. 05A serious training loopMixed precision, warmup then cosine decay, early stopping, and resumable checkpoints that record their own architecture and classes. Models were compared on an 8 GB laptop GPU, then the best one was retrained at full 1024×2048 resolution on a 16 GB RTX A4000.
  6. 06Test where it breaksA model that only sees German cities was scored, unchanged, on the India Driving Dataset (IDD), whose polygon labels are drawn into the same six classes. Road transfers; sidewalks, pedestrians and signs do not. Fine-tuning on both datasets together then closed the gap without losing Cityscapes accuracy.
  7. 07Explain and deployPer-class Grad-CAM attention maps, ONNX export that matches PyTorch on 100% of pixels, and a Streamlit app for images, video and live webcam.

03 · Architecture

Architecture

Six classes per pixel, from road image to explainable, exportable model.

ROAD SCENES6-class CityscapesNETWORK1 of 4 architecturesCLASS MAP6 × 512 × 1024COMBINED LOSSCE + Dice + FocalGRAD-CAMper-class attentionONNX EXPORTvalidatedSTREAMLIT APPimage/video/webcam
  1. 01
    Road scenes in

    Cityscapes remapped once to road, vehicle, pedestrian, sidewalk, traffic sign and background (void pixels ignored), or rendered synthetic scenes for quick runs.

  2. 02
    Pick an architecture

    U-Net, ResNet50-UNet, DeepLabV3+ or SegFormer-B2, all behind one factory.

  3. 03
    A class for every pixel

    Each network outputs a 6 × 512 × 1024 map of class logits, upsampled to full resolution for scoring.

  4. 04
    Train against imbalance

    CE + Dice + Focal loss flows back into the network (dashed line).

  5. 05
    Explain it

    Grad-CAM shows which regions drove each class.

  6. 06
    Ship it

    Validated ONNX export and a Streamlit app for images, video and webcam.

04 · Results

Results on Cityscapes and Indian roads

Held-out test sets: 500 Cityscapes images and 981 IDD images (each dataset's official val set), never used for training or model selection, scored at full resolution.

Cityscapes test mIoU, SegFormer-B2 trained on Cityscapes + IDD
0.867
Indian roads (IDD) test mIoU, same model
0.868
IDD test mIoU of the Cityscapes-only model
0.679
Cityscapes test mIoU of the first laptop model (512×1024)
0.833
Per-class IoU on CityscapesFull resolution lifted traffic signs by 7 points and pedestrians by 5 over the laptop model, and adding IDD cost no class anything. Small, thin objects are still the hard part.
  • Road95.3%
  • Background94.3%
  • Vehicle93%
  • Pedestrian83.5%
  • Sidewalk78.6%
  • Traffic sign75.4%
Per-class IoU on Indian roads981 IDD test images, after training on Cityscapes + IDD; in brackets, the Cityscapes-only model. Sidewalk, the biggest gap, rose from 0.28 to 0.80.
  • Road (was 95.3)98.4%
  • Background (was 92.3)97.9%
  • Vehicle (was 80.7)92.6%
  • Pedestrian (was 63.3)81.6%
  • Sidewalk (was 28.0)79.6%
  • Traffic sign (was 47.8)70.8%
Accuracy vs speedLaptop models: forward pass per 512×1024 image, FP16, RTX 5050. SegFormer is better on every class with fewer parameters, at about 1.7× the cost.
  • SegFormer-B2 · 0.833 mIoU35 ms
  • DeepLabV3+ · 0.825 mIoU21 ms

An earlier version of this project reported 0.89 mIoU. An audit traced that number to a label bug: masks were remapped twice, so the model trained and was scored on two scrambled classes. The number was withdrawn, the pipeline fixed and covered by tests, and every model retrained.

These are 6-class scores and are not comparable to the 19-class Cityscapes leaderboard, where merged classes like "vehicle" don't exist.

IDD (Varma et al., WACV 2019) is scored on its 981 official val images, since its test labels aren't public. Its labels are mapped to the same six classes, and ambiguous "drivable fallback" ground is ignored.

05 · Next

Next steps

  • 01Add roads from more countries (e.g. Mapillary Vistas) and check that each new domain costs the old ones nothing.
  • 02Lift traffic signs, still the weakest class (0.75 IoU on Cityscapes, 0.71 on IDD), with class-balanced sampling.
  • 03Quantise to INT8 and deploy with TensorRT on edge hardware such as NVIDIA Jetson.
  • 04Add temporal consistency for video segmentation.
Next case studyAgentic LedgerAutonomous multi-agent financial engine.