A pollution-preventing process which comprises (i) controlling a surface moisture content of a first building material selected from the group consisting of an inorganic base material and wood at 0 to 20%, or (ii) coating a penetration-inhibiting primer onto a surface of a second building material having a surface moisture content of 10 to 100% and selected from the group consisting of an inorganic base material and wood to be dried, followed by coating a treating agent consisting of an organic solvent solution of a modified polysiloxane resin having a ladder structure onto a surface of the first building material or the second building material to impregnate the surface with the treating agent and by subjecting to curing treatment to prevent at least one pollutant selected from the group consisting of mold and moss from growing on the surface of the first building material or the second building material.