Systems, methods, and non-transitory computer-readable media can determine sensor data captured by at least one sensor of a vehicle while navigating an environment over a period of time. Information describing one or more agents associated with the environment during the period of time can be determined based at least in part on the captured sensor data. A schema-based encoding describing the environment during the period of time can be generated based at least in part on the determined information and a scenario schema, wherein the schema-based encoding provides a structured representation of the environment during the period of time.