A crossed roller bearing assembly includes an outer ring provided with an outer V-shaped guide groove, an inner ring provided with an inner V-shaped guide groove and a plurality of rollers arranged in a guide space defined between the outer and inner V-shaped guide grooves in a rolling contact fashion. An insertion hole is formed in one of the outer and inner rings and the insertion hole preferably has a diameter which is larger than the diameter of the rollers, but smaller than a diagonal length in cross section of the rollers. A plug having a shape which substantially corresponds to the shape of the insertion hole is fitted into the insertion hole after insertion of the rollers. No gap or clearance is defined between the insertion hole and the plug so that the rollers can roll along the guide space smoothly.