An elevator control system (24) provides elevator dispatch and door control based on passenger data received from a video monitoring system. The video monitoring system includes a video processor (16) connected to receive video input from at least one video camera (12). The video processor (16) tracks objects located within the field of view of the video camera, and calculates passenger data parameters associated with each tracked object. The elevator controller (24) provides elevator dispatch (26), door control (28), and security functions (30) based in part on passenger data provided by the video processor (16). The security functions may also be based in part on data from access control systems (14).