A stereo audio signal is processed to determine primary and ambient components by transforming the signal into vectors corresponding to subband signals, and decomposing the left and right channel vectors into ambient and primary components by matrix and vector operations. Principal component analysis is used to determine a primary component unit vector, and ambience components are determined according to a correlation-based cross-fade or an orthogonal basis derivation.