A processor-implemented method of performing convolution operations in a neural network includes generating a plurality of first sub-bit groups and a plurality of second sub-bit groups, respectively from at least one pixel value of an input feature map and at least one predetermined weight, performing a convolution operation on a first pair that includes a first sub-bit group including a most significant bit (MSB) of the at least one pixel value and a second sub-bit group including an MSB of the at least one predetermined weight, based on the plurality of second sub-bit groups, obtaining a maximum value of a sum of results for convolution operations of remaining pairs excepting the first pair, and based on a result of the convolution operation on the first pair and the maximum value, determining whether to perform the convolution operations of the remaining pairs.