Described herein are embodiments of a neural network-based task planner (TaskNet) for autonomous vehicle. Given a high-level task, the TaskNet planner decomposes it into a sequence of sub-tasks, each of which is further decomposed into task primitives with specifications. TaskNet comprises a first model for predicating the global sequence of working area to cover large terrain, and a second model for determining local operation order and specifications for each operation. The neural models may include convolutional layers for extracting features from grid map-based environment representation, and fully connected layers to combine extracted features with past sequences and predict the next sub-task or task primitive. Embodiments of the TaskNet are trained using an excavation trace generator and evaluate its performance using a 3D physically-based terrain and excavator simulator. Experiment results show TaskNet may effectively learn common task decomposition strategies and generate suitable sequences of sub-tasks and task primitives.