The present invention relates to a breast image analysis method with four mammogram images which are input to a convolutional neural network as one input and a system therefor and the system includes an image receiving unit which receives four mammogram images; an image size adjusting unit which adjusts a size of a mammogram image received from the image receiving unit; a preprocessing unit which performs preprocessing on the mammogram image adjusted by the image size adjusting unit; a convolutional neural network (CNN)-based CNN learning unit which generates learning information by learning the mammogram image preprocessed by the preprocessing unit; and a CNN inference unit which receives the learning information learned from the CNN learning unit and a mammogram image to be classified from the image receiving unit to diagnose a breast abnormality.