A system and method are provided. The system includes an image capture device configured to capture an actual image depicting an object. The system also includes a processor. The processor is configured to render, based on a set of 3D Computer Aided Design (CAD) models, a set of synthetic images with corresponding intermediate shape concept labels. The processor is also configured to form a multi-layer Convolutional Neural Network (CNN) which jointly models multiple intermediate shape concepts, based on the rendered synthetic images. The processor is further configured to perform an intra-class appearance variation-aware and occlusion-aware 3D object parsing on the actual image by applying the CNN to the actual image to output an image pair including a 2D geometric structure and a 3D geometric structure of the object depicted in the actual image.