A hydrocracking system involves introducing a heavy oil feedstock and a colloidal or molecular catalyst, or a precursor composition capable of forming the colloidal or molecular catalyst, into a hydrocracking reactor. The colloidal or molecular catalyst is formed in situ within the heavy oil feedstock by intimately mixing a catalyst precursor composition into a heavy oil feedstock and raising the temperature of the feedstock to above the decomposition temperature of the precursor composition to form the colloidal or molecular catalyst. The colloidal or molecular catalyst catalyzes upgrading reactions between the heavy oil feedstock and hydrogen and eliminates or reduces formation of coke precursors and sediment. At least a portion of a resid fraction containing residual colloidal or molecular catalyst is recycled back into the hydrocracking reactor to further upgrade the recycled resid fraction portion and provide recycled colloidal or molecular catalyst within the hydrocracking reactor.