Systems and methods for multi-modal, multi-resolution deep learning neural networks for segmentation, outcomes prediction and longitudinal response monitoring to immunotherapy and radiotherapy are detailed herein. A structure-specific Generational Adversarial Network (SSGAN) is used to synthesize realistic and structure-preserving images not produced using state-of-the art GANs and simultaneously incorporate constraints to produce synthetic images. A deeply supervised, Multi-modality, Multi-Resolution Residual Networks (DeepMMRRN) for tumor and organs-at-risk (OAR) segmentation may be used for tumor and OAR segmentation. The DeepMMRRN may combine multiple modalities for tumor and OAR segmentation. Accurate segmentation is may be realized by maximizing network capacity by simultaneously using features at multiple scales and resolutions and feature selection through deep supervision. DeepMMRRN Radiomics may be used for predicting and longitudinal monitoring response to immunotherapy. Auto-segmentations may be combined with radiomics analysis for predicting response prior to treatment initiation. Quantification of entire tumor burden may be used for automatic response assessment.