A method includes forming Magnetic Tunnel Junction (MTJ) stack layers, which includes depositing a bottom electrode layer; depositing a bottom magnetic electrode layer over the bottom electrode layer; depositing a tunnel barrier layer over the bottom magnetic electrode layer; depositing a top magnetic electrode layer over the tunnel barrier layer; and depositing a top electrode layer over the top magnetic electrode layer. The method further includes patterning the MTJ stack layers to form a MTJ; and performing a passivation process on a sidewall of the MTJ to form a protection layer. The passivation process includes reacting sidewall surface portions of the MTJ with a process gas comprising elements selected from the group consisting of oxygen, nitrogen, carbon, and combinations thereof.