The invention provides a method, system and computer program product for detecting field order of a video sequence. The method includes processing a top and bottom field picture of a first frame and a second frame, in which the first and second frames are consecutive frames of a video sequence. Difference frames are obtained by calculating the difference of the top and bottom field pictures of the first and second frames. Thereafter, field order is detected using the difference frames.