The disclosed embodiments relate to a computing system that facilitates performing prefetching for scatter/gather operations. During operation, the system receives a scatter/gather prefetch instruction at a processor core, wherein the scatter/gather prefetch instruction specifies a virtual base address, and a plurality of offsets. Next, the system performs a lookup in a translation-lookaside buffer (TLB) using the virtual base address to obtain a physical base address that identifies a physical page for the base address. The system then sends the physical base address and the plurality of offsets to a cache. This enables the cache to perform prefetching operations for the scatter/gather instruction by adding the physical base address to the plurality of offsets to produce a plurality of physical addresses, and then prefetching cache lines for the plurality of physical addresses into the cache.