An optical device for holographic recording or reading is provided and includes: a light emission unit that emits light to be irradiated on an optical recording medium, the light being recording or reading light; a first prism unit that moves in accordance with a moving speed of the optical recording medium and refracts the light to move the light irradiated on the optical recording medium over a distance in a moving direction of the optical recording medium so that the light is irradiated on substantially the same position on the optical recording medium for a period of time; and a second prism unit that moves in accordance with the moving speed of the first prism unit to compensate for an optical path length of the light.