In various embodiments, a significance map of a matrix of video data coefficients is encoded or decoded using context-based adaptive binary arithmetic coding (CABAC). The significance map scanned line-by-line along a scanning pattern. Each line may be a vertical, horizontal, or diagonal section of the scanning pattern. Context models for each element processed in a particular line are chosen based on values of neighboring elements that are not in the line. The neighboring elements may be limited to those contained within one or two other scanning lines. Avoiding reliance on neighbors that are in the same scanning line facilitates parallel processing.