A vehicle includes a brake pedal, a master cylinder, a braking circuit with a wheel cylinder, and a brake pressure generator with strokable piston. A pedal feel simulator is coupled to the master cylinder through a switchable valve, the simulator providing a reaction force. An isolation valve closes to isolate the braking circuit from the master cylinder and the simulator circuit. A controller is programmed to place the simulator in fluid communication with an output of the brake pressure generator, and to stroke the piston at a designated diagnostic time. The resulting pressure increase is observed, and the controller checks whether the pressure-increase to piston-stroke relationship is within a predetermined acceptable range for continued operation of a brake-by-wire mode in which the master cylinder is coupled to the simulator circuit and decoupled from the braking circuit.