The present invention discloses a catalyst and process for hydrocracking of heavy hydrocarbon oils having majority portion boiling above 525° C. in the presence of hydrogen. A process comprising first step of converting heavy oil into lighter products in the presence of catalyst and hydrogen in slurry phase is disclosed. The process further comprises recycling of part of liquid products (HVGO) along with fresh heavy oil for improving the product selectivity. This recycled HVGO is having high concentrations of aromatics compounds. The separation of particles generated during the reaction at reactor exit also avoids the chances of choking of downstream sections.