Systems and methods for managing and optimizing mixed fleet worksite operations based on video and or audio data are disclosed. One method includes receiving one or more models relating to a fleet of machines at the worksite, wherein the fleet of machines comprises an in-network machine and an out-of-network machine, receiving first sensor data associated with the out-of-network machine at the worksite, receiving second sensor data associated with the in-network machine at the worksite, determining a machine state of each of the in-network machine and the out-of-network machine based at least on the first sensor data and the second sensor data, comparing the determined machine states to a modeled machine state represented by the received one or more models to classify site operations and/or detect an irregularity in site operations or an inefficiency in site operations, and generating a response based at least on the detected irregularity or inefficiency.