A negative-working resist composition for electron beams or X-rays comprising (A) a compound generating an acid and/or radical species by the irradiation of electron beams or X-rays, (B) a resin which is insoluble in water and soluble in an alkali aqueous solution, (C) a crosslinking agent causing crosslinking with the resin of component (B) by the action of an acid, and (D) a compound having at least one unsaturated bond capable of being polymerized by an acid and/or a radical, and a negative-working resist composition for electron beams or X-rays comprising (A) a compound generating an acid and/or radical species by the irradiation of electron beams or X-rays, (B′) a resin having at least one unsaturated bond polymerizable by an acid and/or an alkali, which is insoluble in water but soluble in an alkali aqueous solution, and (C) a crosslinking agent causing crosslinking with the resin (B′) by the action of an acid are disclosed.