Neural network replaces traditional video codec with 2.61x compression

Conventional video codecs like H.264 compress pixel data. This paper from Khanal et al. takes a radically different approach: overfit a single sinusoidal representation network (SIREN) to map (x, y, t) coordinates to RGB values and audio amplitudes. The architecture uses separate initialization layers for video and audio, shared fully connected hidden layers, and three output branches — one for video and two Siamese audio branches whose disagreement helps estimate and subtract residual noise.

The overfitted teacher network is compressed via response-based knowledge distillation into a smaller student network, followed by 16-bit symmetric weight quantization and lossless LZMA2 (xz) encoding. On a 6.08 MiB test video, the quantized student achieves video PSNR 28.72 dB (SSIM 0.75) and audio PSNR 24.18 dB (log spectral distance 10.69 dB). The pipeline reduces representation size from 9.05 MiB to 2.33 MiB — a 2.61x compression ratio. A bit-width sweep shows quality saturates at 16 bits. The authors compare against H.264, HEVC, and MP3, noting where their approach falls short, and demonstrate a browser-based prototype that trains, transfers, and decodes models over WebRTC.