System and method for establishing secure conference calls. In one example system, a central conference call server establishes point-to-point connections with accessory devices comprising a secure element and connected to corresponding participant devices. The conference call server includes an interface to a plurality of secure elements configured to perform scrambling and unscrambling of media signals communicated to and from the accessory devices. In another example, one of the participant devices operates as the central conference call server. In other examples, participant devices communicate on a conference call via point-to-point connections between all accessory devices connected to the participant devices. The accessory devices include secure elements for decryption and encryption of media signals communicated between the accessory devices.