In various embodiments, a significance map of a matrix of video data coefficients is encoded or decoded using context-based adaptive binary arithmetic coding (CABAC). The significance map scanned line-by-line along a scanning pattern. Each line may be a vertical, horizontal, or diagonal section of the scanning pattern. Context models for each element processed in a particular line are chosen based on values of neighboring elements that are not in the line. Avoiding reliance on neighbors that are in the same line facilitates parallel processing.