This patent disclosure provides various embodiments of combining multiple modalities of non-text surgical data of different formats, in particular in forms of videos, images, and audios in a meaningful manner so that the combined data from the multiple modalities are compatible with text data. In some embodiments, prior to combining the multiple modalities of surgical data, multiple segmentation engines are used to segment and convert a corresponding modality of surgical data into a corresponding set of metrics and parameters. The multiple sets of metrics and parameters corresponding to the multiple modalities are then combined to generate a combined feature set. The combined feature set can be provided to a data analytics tool for performing comprehensive data analyses on the combined feature set to generate one or more predictions for the surgical procedure.