A methodology for designing a distributed shared-memory system, which can incorporate adaptation or selection of cache protocols during operation, guarantees semantically correct processing of memory instructions by the multiple processors. A set of rules includes a first subset of “mandatory” rules and a second subset of “voluntary” rules such that correct operation of the memory system is provided by application of all of the mandatory rules and selective application of the voluntary rules. A policy for enabling voluntary rules specifies a particular coherent cache protocol. The policy can include various types of adaptation and selection of different operating modes for different addresses and at different caches. A particular coherent cache protocol can make use of a limited capacity directory in which some but not necessarily all caches that hold a particular address are identified in the directory. In another coherent cache protocol, various caches hold an address in different modes which, for example, affect communication between a cache and a shared memory in processing particular memory instructions.