An actinic ray-sensitive or radiation-sensitive composition, including: (1) a low molecular compound having a molecular weight of 500 to 5,000 and containing (G) an acid-decomposable group; and (2) a compound capable of generating an acid of 305 Å3 or more in volume upon irradiation with an actinic ray or radiation, an actinic ray-sensitive or radiation-sensitive composition, including: a solvent; and (1A) a compound which is a low molecular compound having a molecular weight of 500 to 5,000 and containing (Z) one or more groups capable of decomposing upon irradiation with an actinic ray or radiation to produce an acid, (G) one or more acid-decomposable groups and (S) one or more dissolution auxiliary groups, wherein assuming that the number of the functional groups in one molecule of (Z), (G) and (S) is z, q and s, respectively, q/z≧2 and s/z≧2, and a pattern forming method using the composition are provided.