A health condition or change in health condition of a person may be determined by processing the person's speech with a neural network. Speech from more than one time period may be processed and, in some implementations, speech from a time period may be associated with a health condition label. For each time period, a feature vector may be computed from the speech and the feature vector may be processed with a neural network to obtain a speech embedding vector. In some implementations, feature vector may include word-piece encodings and the neural network may be a transformer neural network. The speech embedding vectors may be processed with a mathematical model to determine a change in a health condition between two time periods or to determine a health condition label for a specific time period. In some implementations, the mathematical model may be a regression model or a fully-connected neural network.