A technical solution to the technical problem of how to improve performance when performing SpMV multiplication uses sparse matrix row similarity to schedule SpMV multiplication operations. CSR representation metadata is generated for a CSR representation and indicates the locations of non-zero values in the rows of the corresponding sparse matrix or the cache locations of column data needed for SpMV multiplication operations. The CSR representation metadata is used to determine the similarity of rows in the sparse matrix based upon Cosine similarity, Jaccard similarity, Locality Sensitive Hashing (LSH) that approximates Jaccard similarity, or other measures of similarity. The row similarity is used to schedule SpMV multiplication operations to increase data locality, reduce cache misses, reduce time stalling on memory accesses, and reduce bandwidth consumption. Implementations include the use of similarity thresholds to schedule SpMV multiplication operations on particular threads and processing elements and load balancing to further improve performance.