A controller for controlling a motion of at least one device subject to constraints on the motion, is disclosed. The controller comprises a processor and a memory, where the controller inputs parameters of the task including the state of the at least one device to a neural network trained to output an estimated motion trajectory for performing the task. Further, the controller extracts at least some of the integer values of a solution to a mixed-integer optimization problem for planning an execution of the task that results in the estimated motion trajectory. Further, the controller solves the mixed-integer optimization problem for the parameters of the task with corresponding integer values fixed to the extracted integer values to produce an optimized motion trajectory subject to the constraint and changes the state of the at least one device to track the optimized motion trajectory.