The invention is applicable to the image processing field, and provides a panoramic image or video horizontal calibration method, a panoramic image or video horizontal calibration system and a portable terminal. The method comprises steps: images or video frames of two circles after imaging of two cameras with opposite photographing directions are received; a horizontal calibration matrix in the case of photographing is acquired; and according to the horizontal calibration matrix, the images or the video frames of two circles are pasted to a left hemisphere surface and a right hemisphere surface of a spherical panoramic mathematical model in a UV mapping mode, and thus, the panoramic image or the video frame pasted to the spherical panoramic mathematical model is horizontal. When an image or a video frame photographed by a panoramic camera is not horizontal, the panoramic image or the video frame pasted to the spherical panoramic mathematical model is still horizontal.