There is provided an imaging control device, an imaging control method, a program, and an imaging system that enable easy change of a direction in which an image of a subject is captured in a case where imaging is performed using an optical member whose observation optical axis is inclined with respect to a central axis. The imaging control device includes: a rotation axis setting unit that sets a rotation axis on the basis of a pivot point and a point of interest on the subject, the pivot point serving as a fulcrum of the rod-shaped optical member whose observation optical axis is inclined with respect to the central axis and being not positioned on the observation optical axis; and a posture control unit that rotates the optical member around the rotation axis. The present technology can be applied to, for example, an endoscope system.