A method of fabricating a semiconductor device includes the following steps. A plurality of doped regions are formed in a substrate. A first dielectric layer is formed on the substrate. A plurality of first contacts and second contacts are formed in the first dielectric layer to be connected to the plurality of doped regions. A memory element is formed on the first dielectric layer. The memory element is electrically connected to the second contact. A second dielectric layer is formed on the first dielectric layer. The second dielectric layer surrounds the memory element. A conductive line is formed in the second dielectric layer. A top surface of the conductive line is at a same level as a top surface of the memory element, and the conductive line is electrically connected to the plurality of first contacts.