Methods and apparatus of video coding using sub-block based affine mode are disclosed. According to this method, control-point motion vectors (MVs) associated with the affine mode are determined for a block. A sub-block MV is derived for a target sub-block of the block from the control-point MVs for the block. A prediction offset is determined for a target pixel of the target sub-block using information comprising a pixel MV offset from the sub-block MV for the target pixel according to Prediction Refinement with Optical Flow (PROF). The target pixel of the target sub-block is encoded or decoded using a modified predictor. The modified prediction is generated by clipping the prediction offset to a target range and combining the clipped prediction offset with an original predictor.