A method and apparatus are provided. The method includes configuring a plurality of long short term memory (LSTM) networks, wherein each of the plurality of LSTM networks is at a different network layer, configuring a plurality of memory cells in a spatial domain of the plurality of LSTM networks, configuring the plurality of memory cells in a temporal domain of the plurality of LSTM networks, controlling an output of each of the plurality of LSTM networks based on highway connections to outputs from at least one previous layer and at least one previous time of the plurality of LSTM networks, and controlling the plurality of memory cells based on highway connections to memory cells from the at least one previous time.