Deep learning case study exploring the use of synthetic datasets for training semantic segmentation models.
The project demonstrates an end-to-end computer vision pipeline, from synthetic dataset generation in Unity3D to DeepLab v3 fine-tuning in a PyTorch environment.
The goal was to investigate how procedurally generated training data can be used to train semantic segmentation models when annotated datasets are limited.
Real Scene
Generated Binary Mask
Generated Multiclass Mask
A Unity3D-based synthetic data generator was developed to produce labeled training images for vehicle segmentation.
The generator simulates a controlled 3D environment where an orbital camera system captures images of vehicles from multiple viewpoints while automatically producing segmentation masks.
Key characteristics:
Orbiting camera system capturing images around a 3D object
Automatic adjustment of camera orientation toward the target object
Configurable pitch, roll, and yaw variations
Automatic dataset generation with image capture at fixed angular intervals
Real-time access to camera pose and orientation metadata
This approach allows the creation of large labeled datasets without manual annotation, significantly accelerating dataset generation for segmentation tasks.
Scene with orbiter camera
[CLICK TO SHOW VIDEO]
The generator produces paired data suitable for supervised training:
Original RGB image
Binary segmentation mask (background vs vehicle)
Multiclass segmentation mask (vehicle parts)
This structure enables training models for both binary semantic segmentation and multiclass segmentation tasks.
Generated Dataset available as follow
Original Image (1st Image)
Ground Truth (GT) Mask for Binary Semantic Segmentation (2nd Image)
GT Mask for Multiclass Semantic Segmentation (3rd Image)
Binary Semantic Segmentation - Two Classes (Background and Car)
Multiclass Semantic Segmentation - Several Classes (Background and Car parts)
For the segmentation task, the project uses DeepLab v3, a deep convolutional neural network architecture designed for semantic image segmentation.
The model was fine-tuned in a PyTorch environment using the generated synthetic dataset.
Training focused on adapting the pretrained network to the vehicle segmentation task, using synthetic images and ground-truth masks generated by the Unity-based dataset generator.
DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
In this work we address the task of semantic image segmentation with Deep Learning and make three main contributions that are experimentally shown to have substantial practical merit. First, we highlight convolution with upsampled filters, or ‘atrous convolution’, as a powerful tool in dense prediction tasks. Atrous convolution allows us to explicitly control the resolution at which feature responses are computed within Deep Convolutional Neural Networks. It also allows us to effectively enlarge the field of view of filters to incorporate larger context without increasing the number of parameters or the amount of computation. Second, we propose atrous spatial pyramid pooling (ASPP) to robustly segment objects at multiple scales. ASPP probes an incoming convolutional feature layer with filters at multiple sampling rates and effective fields-of-views, thus capturing objects as well as image context at multiple scales. Third, we improve the localization of object boundaries by combining methods from DCNNs and probabilistic graphical models. The commonly deployed combination of max-pooling and downsampling in DCNNs achieves invariance but has a toll on localization accuracy. We overcome this by combining the responses at the final DCNN layer with a fully connected Conditional Random Field (CRF), which is shown both qualitatively and quantitatively to improve localization performance. Our proposed “DeepLab” system sets the new state-of-art at the PASCAL VOC-2012 semantic image segmentation task, reaching 79.7% mIOU in the test set, and advances the results on three other datasets: PASCAL-Context, PASCAL-Person-Part, and Cityscapes. All of our code is made publicly available online.
The training pipeline includes:
dataset formatting and preprocessing scripts
preparation of segmentation masks
PyTorch-based model training workflows
transfer learning using pretrained DeepLab v3 models
The final result is a fine-tuned DeepLab v3 model capable of performing vehicle segmentation on the generated dataset.
DeepLab v3 Source Model
Whole project realized by Manpreet Singh posted at GitHub.
Car Segmentation Model
Fine tuned DeepLab v3 model. A common PyTorch convention is to save models using either a .pt or .pth file extension.
