The disclosed system uses a challenge-response authentication protocol for datagram-based remote procedure calls. Using a challenge-response authentication protocol has many advantages over using a conventional authentication protocol. There are two primary components responsible for communication using the challenge-response protocol: a challenge-response protocol component on the client computer (client C-R component) and a challenge-response protocol component on the server computer (server C-R component). In order to start a session using the challenge-response protocol, the client C-R component first generates a session key. The session key is used by both the client C-R component and the server C-R component for encrypting and decrypting messages. After creating the session key, the client C-R component encrypts a message containing a request for a remote procedure call and sends it to the server C-R component. In response, the server C-R component sends a challenge to the client C-R component. The challenge contains a unique identifier generated by the server C-R component. The client C-R component responds to the challenge by sending a challenge response and the session key. The challenge response is the unique identifier contained within the challenge encrypted with the password of the user of the client computer. The session key is also encrypted using this password. Upon receiving the challenge response, the server C-R component uses its copy of the client's password to create its own version of the challenge response and compares it to the version received from the client C-R component. If the two versions of the challenge response are identical, the identity of the user of the client computer has been verified. If the two versions are not identical, an attempted unauthorized access has been detected. After verification, the server C-R component extracts the session key, decrypts the message, and invokes the requested procedure of the server program. Subsequently, the server C-R component will send and receive encrypted messages from the client C-R component, thereby facilitating a remote procedure call.