An image processing apparatus stores a background model in which a feature amount is associated with time information for each state at each position of an image to be a background, extracts a feature amount for each position of an input video image, compares the feature amount in the input video image with that of each state in the background model, to determine the state similar to the input video image, and updates the time information of the state similar to the input video image, determines a foreground area in the input video image based on the time information of the state similar to the input video image, detects a predetermined subject from the foreground area, and updates the time information of the state in the background model.