An adaptive filter to use in connection with prediction-based pixel block encoding and decoding is determined independently at the encoder and decoder side through a template-based procedure. A pixel block (12) has an identified reference pixel block (22) in a reference frame (20). A template (16) of multiple pixels (18) adjacent the pixel block (12) and a reference template (26) of multiple pixels (28) adjacent the reference pixel block (22) are used for determining the filter parameters of the adaptive filter. The determined adaptive filter is then applied to the reference pixel block (22) and is used for generating an encoded representation (40) of the pixel block (12) during encoding and or generating a decoded representation of the pixel block (12) during decoding.