A system and method for controlling a hybrid electric vehicle powertrain having an engine, a generator, and a motor connected via a planetary gear set to detect lockup in the planetary gear set and control the powertrain in response. When torque is distributed in an electric mode of operation with the engine disabled, the generator is disabled based at least upon a difference between actual generator speed and an expected generator speed exceeding a threshold, indicating a lockup in the planetary gear set. When the engine is activated and distributes torque through the powertrain, the engine and the generator are disabled based at least upon a difference between engine acceleration and an expected engine acceleration exceeding a first threshold, and a difference between engine speed and ring gear speed being less than a second threshold.