Systems, devices, and techniques are disclosed for recommendations using a transformer neural network. User activity data including items and actions associated with users and a catalog including descriptions of the items may be received. User vectors for the users, item vectors for the items and action vectors the actions may be generated by applying singular vector decomposition to the user activity data. Sequence vectors may be generated based on item vectors and the action vectors. Transformer vectors may be generated by applying a text-to-text transferring transformer to descriptions of the items. Similarity vectors may be generated based on the transformer vectors. Merged vectors may be generated by merging the sequence vector, transformer vector, and similarity vector for items. A set of probabilities may be determined by inputting the user vector for the user, merged vectors for the items, and sequence vectors for the actions to a transformer neural network.