According to one embodiment, a magnetic memory device includes a stacked structure including a first magnetic layer having a variable magnetization direction, a second magnetic layer having a fixed magnetization direction, and a nonmagnetic layer provided between the first magnetic layer and the second magnetic layer, and containing magnesium (Mg) and oxygen (O). The nonmagnetic layer further contains an additive element selected from fluorine (F), sulfur (S), hydrogen (H) and lithium (Li).