A neural network device with synapses having memory cells each having source and drain regions in a semiconductor substrate with a channel region extending there between, a floating gate over an entirety of the channel region, and a first gate over the floating gate. First lines each electrically connect together the first gates in one of the memory cell rows, second lines each electrically connect together the source regions in one of the memory cell rows, and third lines each electrically connect together the drain regions in one of the memory cell columns. The synapses are configured to receive a first plurality of inputs as electrical voltages on the first lines or on the second lines, and to provide a first plurality of outputs as electrical currents on the third lines.