Embodiments of mechanisms of a backside illuminated image sensor device structure are provided. The method for manufacturing a backside illuminated image sensor device structure includes providing a substrate and forming a polysilicon layer over the substrate. The method further includes forming a buffer layer over the polysilicon layer and forming an etch stop layer over the buffer layer. The method further includes forming a hard mask layer over the etch stop layer and patterning the hard mask layer to form an opening in the hard mask layer. The method further includes performing an implant process through the opening of the hard mask layer to form a doped region in the substrate and removing the hard mask layer by a first removing process. The method further includes removing the etch stop layer by a second removing process and removing the buffer layer by a third removing process.