Provided are an apparatus and method for automatically generating a visual annotation with respect to a massive image based on a visual language. The apparatus for automatically generating a visual annotation based on a visual language includes an image input unit configured to receive an input image, an image analyzing unit configured to extract feature information of the input image received by the image input unit, a searching unit configured to search a similar image with respect to the input Image and text information included in the similar image by using the feature information extracted by the image analyzing unit, and a visual annotation configuring unit configured to configure a visual annotation with respect to the input image by using the text information searched by the searching unit.