R&D · drones and synthetic data
Real2sim
Drone footage of a real place becomes a photorealistic digital twin, then a simulator for flights and a factory for labelled training data.
Private repository · walkthrough on request
- 597kGaussians in the first cloud splat
- $0.75for that run, debugging included
- 0.57 pxreprojection error in the local SfM test
- PX4SITL flying locally in WSL2
The problem
Detection models for drones need thousands of labelled aerial images of the places they will fly over, and real labelling is slow. The idea is to rebuild the real place once and generate the data inside it.
How it works
- 01
Plan the survey
30–60 m above ground, 1–2.5 cm per pixel, a grid plus at least three orbits and facades, 65–80% overlap.
- 02
Frames and geotags
ffmpeg extracts sharp frames; positions come from the drone's SRT telemetry.
- 03
Structure from motion
COLMAP on a cloud GPU recovers camera poses and aligns the model to GPS.
- 04
Gaussian splat
splatfacto-big with MCMC trains the photorealistic scene. The first cloud run: 597k Gaussians from 88 photos on an RTX 4090.
- 05
Into the simulator
A simplified collision mesh plus the splat go to USD and into Isaac Sim, where PX4 flies the drone over the twin.
- 06
Labelled data
Replicator inserts objects and exports labels in COCO, converted to YOLO; training goes synthetic first, then fine-tunes on real frames.
Limits I know about
- Heavy steps need a cloud GPU: a local CPU took 60 minutes for 16 photos.
- Only commercially usable licences are allowed (Apache, BSD, Isaac for R&D), which rules out several popular research codebases.