A video coder can determine a most probable mode for a block of video data using non-square partitions based on the direction of the non-square partitions. When the direction of the non-square partitions is vertical, an intra prediction mode of a left-neighboring block can be selected as the most probable intra prediction mode, and when the direction of the non-square partitions is horizontal, an intra prediction mode of an above-neighboring block can be selected as the most probable intra prediction mode.