Cases / Applied AI · 2026
EW-Resilient Target Recognition: Unreal Engine + YOLO
Synthetic jamming simulated in Unreal Engine produces auto-labelled training data for a YOLO detector that still finds a tank through electronic-warfare interference.
Context
Automated target recognition on drone video breaks down when the analog video link is jammed. Electronic warfare (EW) produces horizontal sync glitches and noise bands that slice a target into pieces. Real jammed footage with reliable labels is scarce, so a model cannot simply be trained on it.
The question for this proof of concept: can we generate the training data synthetically, with labels for free, and still get a detector that holds up under jamming?
My role
I defined the pipeline, made the build-versus-skip decisions and wrote the working scripts with AI coding assistance. The focus was a pipeline that other people could repeat and extend, not a one-off model.
What I did
- Synthetic scenes in Unreal Engine. A custom post-process shader recreates analog jamming and sync glitches over a 3D tactical scene.
- Labels without annotators. Movie Render Queue renders two passes in one run: the glitched image and a clean binary mask of the target. A Python/OpenCV script pairs them by frame number, finds the mask contour and writes normalised YOLO bounding boxes. Both passes must be rendered together, otherwise the labels drift — a lesson worth documenting.
- Training. YOLOv8n transfer learning at 1024×1024 for 50 epochs on a 150-render dataset (a 500-frame set is the next increment).
- Reading the failures. Analysed frame by frame how the detector behaves: when a noise band cuts the target it produces several overlapping boxes for one object; under extreme occlusion it still returned a single stable box at about 39% confidence. The fix for duplicates was tuning the confidence and NMS thresholds, not retraining.
- Closing the loop on a live simulation. Tried three ways to feed frames from Unreal into the detector: Spout2 with a render target (blocked by DirectX 12 resource barriers in the editor), Spout2 in standalone mode (broken Python wrapper) and OS-level screen capture. Screen capture won for the PoC because it is independent of engine version and graphics API. WebRTC Pixel Streaming is the documented production path for remote GPUs.
Result
- mAP50 of 96.5% on the synthetic validation set, and a tank detected under heavy jamming at 74% confidence.
- A repeatable synthetic-data pipeline from render to trained weights, with automatic labelling and no manual annotation.
- Honest limits: results are on synthetic data from one scene family and have not been validated on real jammed footage. That is the next milestone.