Generally, the present disclosure is directed to a neural architecture search process for finding small and fast video processing networks for understanding of video data. The neural architecture search process can automatically design networks that provide comparable video processing performance at a fraction of the computational and storage cost of larger existing models, thereby conserving computing resources such as memory and processor usage.