A reinforcement learning neural network system in which internal representations and policies are grounded in visual entities derived from image pixels comprises a visual entity identifying neural network subsystem configured to process image data to determine a set of spatial maps representing respective discrete visual entities. A reinforcement learning neural network subsystem processes data from the set of spatial maps and environmental reward data to provide action data for selecting actions to perform a task.