A valve operating device (100) opens and closes a valve (2) of an internal combustion engine by driving a valve stem axially. A rocker cam (10) that rocks about a drive shaft (21) in synchronization with a rotation of the internal combustion engine includes a driving cam portion (11) and a lift restricting cam portion (12). A cam follower (41) lifts the valve by contacting the driving cam portion (11) slidingly so as to convert the rocking of the rocker cam (10) into an axial direction motion of the valve stem. A restricting member (42) formed integrally with the cam follower (41) in a rocker arm (40) prevents the cam follower (41) from separating from the driving cam portion (11) in cooperation with the lift restricting cam portion (12), and as a result, an irregular valve motion is prevented regardless of a rocking acceleration of the rocker cam (10).