A method for assembling a camera for a vision system of a vehicle includes providing a circuit element having an imager disposed thereat, with the imager including a pixelated imaging array having a plurality of photosensing elements. A lens holder is provided that includes a lens, and a camera housing element is provided. The camera housing element includes an electrical connector having a plurality of electrically conductive pins. The lens is aligned and focused relative to the imager and the circuit element is affixed relative to the lens holder when the lens is aligned and focused relative to the imager. A header connector is positioned and attached at the circuit element after the circuit element is affixed relative to the lens holder. The camera housing element is mated with the lens holder, including electrically connecting the electrically conductive pins to respective electrically conductive elements of the header connector.