A method and apparatus is capable of processing a sensory effect of image data. The apparatus includes an image analyzer that analyzes depth information and texture information about at least one object included in an image. A motion analyzer analyzes a motion of a user. An image matching processor matches the motion of the user to the image. An image output unit outputs the image to which the motion of the user is matched, and a sensory effect output unit outputs a texture of an object touched by the body of the user to the body of the user.