A process for upgrading crude oil fractions or other hydrocarbon oil feedstreams boiling in the range of 36° to 520° C., and preferably naphtha and gas oil fractions boiling in the range of 36° to 400° C., employs a solid adsorption material to lower sulfur and nitrogen content by contacting the hydrocarbon oil, and optionally a viscosity-reducing solvent, with one or more solid adsorbents such as silica gel or silica, silica alumina, alumina, attapulgus clay and activated carbon in a mixing vessel for a predetermined period of time; passing the resulting slurry to a membrane separation zone, optionally preceded by a primary filtration step (i.e., single stage or multiple stages), to separate the solid adsorption material with the adsorbed sulfur and nitrogen compounds from the treated oil; recovering the upgraded hydrocarbon product having a significantly reduced nitrogen and sulfur content as the membrane permeate; mixing the solid adsorbent material with one or a combination of aromatic solvents such as toluene, benzene, the xylenes and tetrahydrofuran to remove and stabilize the sulfur and nitrogen compounds; transferring the solvent to a fractionation tower to recover the solvent, which can be recycled for use in the process; and recovering the hydrocarbons that are rich in sulfur and nitrogen for processing in a relatively small high-pressure hydrotreating unit or transferring them to a fuel oil pool for blending.