An autonomous vehicle may be configured to detect objects based on known structures of an environment. The vehicle may be configured to obtain image data from a sensor and be configured to operate in an autonomous mode. The image data may include data indicative of a known structure in the environment. The vehicle may include a computer system. The computer system may determine, based on a first portion of the image data, information indicative of an appearance of the known structure. The computer system may determine, based on a second portion of the image data, information indicative of an appearance of an unknown object in the environment. The computer system may also compare the information indicative of the appearance of the known structure with the information indicative of the appearance of the unknown object and provide instructions to control the vehicle in the autonomous mode based on the comparison.