An audio customization system operates to enhance a user's audio environment. A user may wear headphones and specify what portion the ambient audio and/or source audio will be transmitted to the headphones or the personal speaker system. The audio signal may be enhanced by application of a spatialized transformation using a spatialization engine such as head-related transfer functions so that at least a portion of the audio presented to the personal speaker system will appear to originate from a particular direction. The direction may be modified in response to movement of the personal speaker system.