Query Based Video Segment Retrieval Using Knowledge Graphs

The growth in the amount of videos causes problems in finding relevant data in videos by text search. The current state-of-the-art approach uses vector searches based on semantics and demonstrates poor multimodal data understanding capabilities. The presented paper describes the method for retrieving segments of videos that utilizes semantic vectors and knowledge graphs to make accurate video search. The proposed method includes multimodal analysis like speech recognition, automatic recognition of characters in the text, and then video contextual segmentation. Knowledge relations between objects are found by relation extraction models, which allow storing structured data in a graph representation. Relations in graphs can be used to query data and retrieve semantically related information. Efficient ranking with a combination of graph and vector scores is developed. The experiments demonstrate the effectiveness of the proposed technique in terms of mean reciprocal rank (MRR) and Recall@K metrics.