Various examples of decoders and physical layout designs for non-volatile flash memory arrays in an analog neural system are disclosed. In one example, a system comprises a plurality of vector-by-matrix multiplication arrays in an analog neural memory system, each vector-by-matrix multiplication array comprising an array of non-volatile memory cells organized into rows and columns, wherein each memory cell comprises a word line terminal; a plurality of read row decoders, each read row decoder coupled to one of the plurality of vector-by-matrix multiplication arrays for applying a voltage to one or more selected rows during a read operation; and a shared program row decoder coupled to all of the plurality of vector-by-matrix multiplication arrays for applying a voltage to one or more selected rows in one or more of the vector-by-matrix multiplication arrays during a program operation.