System and method are disclosed for approximating unknown safety constraints during reinforcement learning of an autonomous agent. A controller for directing the autonomous agent includes a reinforcement learning (RL) algorithm configured to define a policy for behavior of the autonomous agent, and a control barrier function (CBF) algorithm configured to calculate a corrected policy that relocates policy states to an edge of a safety region. Iterations of the RL algorithm safely learn an optimal policy where exploration remains within the safety region. CBF algorithm uses standard least squares to derive estimates of coefficients for linear constraints of the safe region. This overcomes inaccurate estimation of safety region constraints caused by one or more noisy observations of constraints received by sensors.