A polymer comprising recurring units having an acid generator bound to the backbone, and recurring units having an optionally acid labile group-substituted carboxyl group and/or recurring units having an optionally acid labile group-substituted hydroxyl group is obtained by polymerizing corresponding monomers in a solution of a non-polymerizable compound containing a nitrogen atom to which an acid labile group is bound. This prevents deprotection reaction of the acid labile group in the case of positive resist-forming polymer or crosslinking reaction in the case of negative resist-forming polymer.