An image signal process apparatus and a method for controlling thereof are provided. The image signal process apparatus includes a storage unit storing pixel data of an incoming frame image, a scaling filter generating one frame image by scaling the pixel data, and a controller retrieving a part of the pixel data among pixel data of the frame image stored at the storage unit, and transmitting the retrieved pixel data to the scaling filter. As a result, a process of processing a stereoscopic image may be reduced such that deterioration of an image quality may be prevented, and a data processing time may be reduced.