A method and apparatus for joint sign prediction of transform coefficients. At the encoder side, a maximum number is determined according to coding context of the current block. A set of jointly predicted coefficient signs associated with a set of selected transform coefficients is determined where a total number of the set of jointly predicted coefficient signs is equal to or smaller than the maximum number. Joint sign prediction for the set of jointly predicted coefficient signs is determined by selecting one hypothesis from a group of hypotheses corresponding to combinations of the set of jointly predicted coefficient signs that achieves a minimum cost.