Methods and apparatus of compression for pre-stitched pictures captured by multiple cameras of a panoramic video capture device are disclosed. At the encoder side, stitching information associated with a stitching process to form the pre-stitched pictures is used to encode a current block according to embodiments of the present invention, where the stitching information comprises calibration data, matching results, seam position, blending level, sensor data, or a combination thereof. In one embodiment, the stitching information corresponds to matching results associated with a projection process, and projection-based Inter prediction is used to encode the current block by projecting a reference block in a reference pre-stitched picture to coordinates of the current block. In another embodiment, the stitching information corresponds to seam information associated with seam detection, and seam-based Inter prediction is used to encode the current block by utilizing the seam information.