Example implementations relate to punch-in transitions to smoothly transition operation of parts of a robotic system from control tracks to real-time input, such as to smoothly transition a control track rotating a knob at a handheld controller to real-time input received at the handheld controller. Similarly, example implementations relate to punch-out transitions to smoothly transition operation of parts of a robotic system from real-time input to control tracks, such as to smoothly transition from the real-time input received at a handheld controller to a control track rotating a knob at a handheld controller. In particular, an example system may include a handheld haptic controller, a control system, and a robotic component.