A system for image quality assessment of non-aligned images includes a first deep path portion of a convolutional neural network having a set of parameters and a second deep path portion of the convolutional neural network sharing a set of parameters with the first deep path convolutional neural network. Weights are shared between the first and second deep path convolutional neural networks to support extraction of a same set of features in each neural network pathway. Non-aligned reference and distorted images are respectively provided to the first and second deep paths of the convolutional neural network for processing. A concatenation layer is connected to both the first and second deep paths convolutional neural network, and a fully connected layer is connected to the concatenation layer to receive input from both the first and second deep paths of the convolutional neural network, generating an image quality assessment as a linear regressor and outputting an image quality score.