A remote driving system controls a remote driving vehicle, which is capable of performing a remote driving and an in-vehicle driving. The remote driving system is configured to: sequentially determining occurrence of an abnormality in a remote determination target, the remote determination target being a remote driver or a remote system operated by the remote driver; permit the in-vehicle driving in response to determining occurrence of the abnormality in the remote determination target; and forbid the in-vehicle driving in response to failing to determine occurrence of the abnormality in the remote determination target.