An illustrative example elevator control system includes a plurality of sway sensors situated within a hoistway of the building. The sway sensors respectively include a contact surface situated to be contacted by a vertically extending elongated member of an elevator when the elongated member moves laterally in the hoistway. The sway sensors respectively provide an indication of contact between the contact surface and the elongated member. A controller receives an indication of building movement and the indications from the sway sensors. The controller determines whether at least one condition exists in the hoistway based on the indications and implements an adjustment to elevator movement control when the at least one condition exists.