A system and method of implementing a neural network with a non-linear activation function is disclosed. A Universal Coordinate Rotation Digital Computer (CORDIC) is used to implement the activation function. Advantageously, the CORDIC is also used during training for back propagation. Using a CORDIC, activation functions such as hyperbolic tangent and sigmoid may be implemented without the use of a multiplier. Further, the derivatives of these functions, which are needed for back propagation, can also be implemented using the CORDIC.