A method of operating a vehicle includes a vehicle controller receiving an operator-input vehicle control command with an associated torque request, and identifying any propulsion actuator constraints that limit a brake torque capacity available from the vehicle powertrain. Using the propulsion actuator constraint(s) and torque request, the controller determines a propulsion brake torque distribution for the vehicle's road wheels and a maximum brake torque capacity for the powertrain actuator(s). A first brake torque request is determined using the propulsion brake torque distribution and a vehicle control mode of the powertrain system, and a second brake torque request is determined using the maximum brake torque capacity and the vehicle control mode. A friction brake torque command is determined by arbitrating between the first and second brake torque requests. The vehicle controller transmits the friction brake torque command to the friction brake system and a powertrain brake command to the powertrain actuator(s).