Robotic devices, systems, and methods for use in telesurgical therapies through minimally invasive apertures make use of joint-based data throughout much of the robotic kinematic chain, but selectively rely on information from an image capture device to determine location and orientation along the linkage adjacent a pivotal center at which a shaft of the robotic surgical tool enters the patient. A bias offset may be applied to a pose (including both an orientation and a location) at the pivotal center to enhance accuracy. The bias offset may be applied as a simple rigid transformation from the image-based pivotal center pose to a joint-based pivotal center pose.