There is provided a surface abnormality detection device, and a system, capable of detecting an abnormal portion having a displacement below the distance measurement accuracy when detecting the abnormal portion on the surface of a structure. A surface abnormality detection device includes a classification means for classifying an object under measurement into one or more clusters having the same structure, based on position information at a plurality of points on a surface of the object under measurement; a determination means for determining a reflection brightness normal value of the cluster based on a distribution of reflection brightness values at a plurality of points on a surface of the cluster; and an identification means for identifying an abnormal portion on the surface of the cluster based on a difference between the reflection brightness normal value and the reflection brightness value at each of the plurality of points.