Article · · 6 min read
A photorealistic digital twin from drone video for $0.75
Planning a drone survey, structure from motion on a rented GPU and a Gaussian splat with 597k Gaussians from 88 photos, as the first step towards synthetic training data for drone detectors.
Detection models for drones need thousands of labelled aerial images of the places they will fly over. Labelling real footage is slow. The idea behind Real2sim is to rebuild a real place once, from drone video, and then generate labelled data inside the copy.
Plan the survey before you fly
Reconstruction quality is decided in the air. The plan I use: 30 to 60 m above ground, 1 to 2.5 cm per pixel, a grid plus at least three orbits and the facades, with 65 to 80% overlap between photos. Too little overlap and the cameras cannot be matched; too much altitude and the detail is gone.
From video to posed cameras
ffmpeg extracts sharp frames from the video, and positions come from the drone's SRT telemetry. COLMAP on a cloud GPU recovers the camera poses and aligns the model to GPS. In a local test the reprojection error was 0.57 px. The heavy steps really do need a GPU: a local CPU took 60 minutes for just 16 photos.
The splat
splatfacto-big with MCMC trains the photorealistic scene. The first cloud run produced 597k Gaussians from 88 drone photos on an RTX 4090, and the whole run, debugging included, cost $0.75 of rented GPU time.
Into the simulator
A simplified collision mesh plus the splat go to USD and into Isaac Sim, where PX4 flies the drone over the twin; locally, PX4 SITL already flies in WSL2. Replicator then inserts objects and exports labels in COCO, converted to YOLO. Training starts on synthetic data and is fine-tuned on real frames.
The constraint that shaped everything
Only commercially usable licences are allowed: Apache, BSD, and Isaac for R&D. That rules out several popular research codebases, and it is worth checking before writing any code, not after.