公开(公告)号：US20240176959A1

公开(公告)日：2024-05-30

申请号：US18200778

申请日：2023-05-23

Applicant: ELECTRONICS AND TELECOMMUNICATIONS RESEARCH INSTITUTE

Inventor： Jeong Heo ,  Young-Ae SEO ,  Jin SEONG ,  Jong Hun SHIN ,  Ki Young Lee ,  Soojong LIM ,  Young Kil Kim ,  Jihee Ryu

IPC: G06F40/40 ,  G06F40/30 ,  G06V20/70

CPC classification number: G06F40/40 ,  G06F40/30 ,  G06V20/70

Abstract: Provided is a method of generating a language model using crossmodal information. The method includes: receiving language-based first modality information and non-language-based second modality information; converting the first modality information into a first byte sequence; converting the second modality information into a second byte sequence; converting the first and second byte sequences into a first embedding vector and a second embedding vector by applying an embedding technique for each modality; generating semantic association information between first and second modality information by inputting the first and second embedding vectors to a crossmodal transformer; and learning the language model by setting the generated semantic association information as training data.