A method and system for secure speaker authentication between a caller device and a first device using an authentication server are provided. The system comprises extracting features into a feature matrix from an incoming audio call; generating a partial i-vector, wherein the partial i-vector includes a first low-order statistic; sending the partial i-vector to the authentication server; and receiving from the authentication server a match score generated based on a full i-vector and another i-vector being stored on the authentication server, wherein the full i-vector is generated from the partial i-vector.