Sparse matrix operations using processing-in-memory is described. In accordance with the described techniques, a processing-in-memory component of a memory module receives a request for a vector element stored at a first location in memory of the memory module. The processing-in-memory component identifies an index value for a non-zero element in a sparse matrix using a representation of the sparse matrix stored at a second location in the memory. The processing-in-memory component then outputs a result that includes the vector element by retrieving the vector element from the first location in memory using the index value.