A video coder compares a first sample of a current block to a first set of neighbor samples to determine a first difference value; determines a first offset for the first sample based on the first difference value; compares a second sample of the current block to a second set of neighbor samples to determine a second difference value that has a same sign as the first difference value; determines an offset for the second sample, that is different than the offset for the first sample, based on the second difference value; and applies the first offset to the first sample value and the second offset to the second sample value to generate a filtered image.