A device for purifying water comprises a filter unit (10) containing an osmotic membrane and provided with an inlet (11), a filtrate outlet (12), a return conduit (14) for a recirculating large flow of unfiltered water and having an outlet (22) for a small flow of reject water,According to the invention, two ion exchange units (17, 18) are arranged in parallel and connected to the filter unit (10) in such way that raw water entering the device is first passed through one of said ion exchange units to be softened, and that the reject water is passed in the opposite direction through the other ion exchange unit to regenerate the same. Valve means (19, 24) are provided for alternately shifting the ion exchange units between the functions of softening and regeneration.