The technology relates to assisting large self-driving vehicles, such as cargo vehicles, as they maneuver towards and/or park at a destination facility. This may include a given vehicle transitioning between different autonomous driving modes. Such a vehicles may be permitted to drive in a fully autonomous mode on certain roadways for the majority of a trip, but may need to change to a partially autonomous mode on other roadways or when entering or leaving a destination facility such as a warehouse, depot or service center. Large vehicles such as cargo truck may have limited room to maneuver in and park at the destination, which may also prevent operation in a fully autonomous mode. Here, information from the destination facility and/or a remote assistance service can be employed to aid in real-time semi-autonomous maneuvering.