A mutual authentication method in a Downloadable Conditional Access System (DCAS) is provided. The mutual authentication method may receive authentication-related information about authentication between an authentication unit and a security module (SM) from a Trusted Authority (TA), generate an authentication session key using the authentication-related information, transmit the authentication session key by the authentication unit to the SM through a Cable Modem Termination System (CMTS), and control a Conditional Access System (CAS) software to be downloaded to the SM from the authentication unit, when the authentication is completed by the authentication session key.