Embodiments of the present disclosure relate to controlling generation of 3D objects using point clouds and text. Systems and methods are disclosed that leverage a pre-trained text-to-image diffusion model to reconstruct a complete 3D model of an object from a sensor-captured incomplete point cloud for the object and a textual description of the object. The complete 3D model of the object may be represented as a neural surface (signed distance function), polygonal mesh, radiance field (neural surface and volumetric coloring function), and the like. The signed distance function (SDF) measures the distance of any 3D point from the nearest surface point, where positive or negative signs indicate that the point is outside or inside the object respectively. The SDF enables use of the incomplete point cloud for constraining the surface location by simply encouraging the signed distance function to be zero in the point cloud locations.