A device for inserting a subcutaneous infusion device into skin of a patient. The device can include a housing, a needle hub including a needle, a sleeve, and a spring engaging the needle hub. The device can also include a cap coupled to the housing, and a retention member configured to maintain the device in a ship state prior to decoupling of the cap from the housing. In one example, the retention member can include a boss coupled to the cap and configured to engage the infusion device to maintain the infusion device in the ship state prior to decoupling of the cap from the housing. In another example, the retention member can include a tab formed by the sleeve, the tab engaging a bead on an internal surface of the cap when the sleeve moves relative to the housing while the device is in the ship state.