A neural network system leverages dual attention, specifically both spatial attention and channel attention, to jointly estimate heart rate and respiratory rate of a subject by processing images of the subject. A motion neural network receives images of the subject and estimates heart and breath rates of the subject using both spatial and channel domain attention masks to focus processing on particular feature data. An appearance neural network computes a spatial attention mask from the images of the subject and may indicate that features associated with the subject's face (as opposed to the subject's hair or shoulders) to accurately estimate the heart and/or breath rate. Channel-wise domain attention is learned during training and recalibrates channel-wise feature responses to select the most informative features for processing. The channel attention mask is learned during training and can be used for different subjects during deployment.