A system and method for virtual clothes fitting based on video augmented reality in a mobile wireless device is disclosed. In an embodiment, a method for virtual cloth-fitting with video augmentation in a mobile wireless device includes receiving a video stream of an upper body of a user; detecting a face in the video stream; detecting a shoulder contour; determining keypoints of the shoulder and neck portions of the video stream of the upper body of the user according to the detected face and the detected shoulder contour; mapping an image of clothes to the video stream of the upper body of the user according to the keypoints; and displaying an augmented video stream of the upper body of the user with the image of the clothes overlaid over a portion of the video stream of the upper body of the user.