An autonomous vehicle may be configured to use environmental information for image processing. The vehicle may be configured to operate in an autonomous mode in an environment and may be operating substantially in a lane of travel of the environment. The vehicle may include a sensor configured to receive image data indicative of the environment. The vehicle may also include a computer system configured to compare environmental information indicative of the lane of travel to the image data so as to determine a portion of the image data that corresponds to the lane of travel of the environment. Based on the portion of the image data that corresponds to the lane of travel of the environment and by disregarding a remaining portion of the image data, the vehicle may determine whether an object is present in the lane, and based on the determination, provide instructions to control the vehicle in the autonomous mode in the environment.