A seat belt tongue retaining device which comprises a buckle unit having an insertion passage into which a tongue is inserted, and a latch pawl capable of getting into and out of the insertion passage. A tongue has a latch hole for allowing the latch pawl to engage therewith. Slots formed in the tongue extend in the inserting direction. Arms disposed turnably in the buckle unit have leading ends capable of protruding and moving in the tongue inserting direction. The arms engage with the slots of the tongue in the insertion passage and pull the tongue deep into the passage wherein the tongue is latched automatically.