A system and a method are disclosed for performing a neural architecture search. The method includes sampling a discrete network search space a first time, determining a differential architecture network sampled from a super-network using continuous relaxation of the discrete network search space over operators in the super-network, calculating a reward based on a proxy accuracy or a proxy complexity of the differential architecture network, updating a distribution of the discrete network search space based on the reward, and determining an updated differential architecture network based on the reward.