A method and apparatus for significance map context selection are disclosed. According to the present invention, the TUs are divided into sub-blocks and at least two context sets are used. Non-DC transform coefficients in each sub-block are coded based on the same context, context set, or context formation. The context, context set, or context formation for each sub- block can be determined based on sub-block index in scan order, horizontal sub-block index, vertical sub-block index, video component type, TU width, TU height, or any combination of the above. In one embodiment, the sum of the horizontal and the vertical sub-block indexes is used to classify each sub-block into a class and the context, context set, or context formation is then determined according to the class.