Design of an Integrated Method Using Transformer-Based Sequence Models and RankNet for Video Transcript Processing

Abstract

The increasing consumption of educational videos translates into requirements for fast and correct multimedia processing of video transcripts, especially in educational domains. Traditional methods usually fail to keep up with the amount of data produced through these sources, thereby affecting transcription accuracy, semantic understanding, content classification, and relevance ranking. Specific to these methods is their reliance on models in isolation, which all capture only a subset of the complicated relationships between textual and visual data, hence often leading to less optimal performance across these tasks. This work provides an integrated, holistic way to incorporate multiple state-of-the-art methodologies within one framework for those limitations. Proposed work take advantage of a T5 (Text-to-Text Transfer Transformer)/BART(Bidirectional and Auto- Regressive Transformers) based Transformer sequence-to-sequence model in transcript pre-processing and segmentation to further bring down Word error rate (WER) by 15-20% and improve context segmentation accuracy up to about 25% when applied on Massive Open Online Courses (MOOCs) dataset samples. This work then utilizes Sentence-BERT (SBERT) for enhanced semantic understanding, where in semantically meaningful sentence embeddings are created that improve the average cosine similarity score by about 20% over the baseline models. This work focuses first on the multiple modal fusion model, which concatenates video features from a pre-trained Convolutional Neural Network(CNN) and text features from SBERT to increase around 10-15% in classification accuracy. At the end, this work has a pairwise ranking algorithm known as RankNet that integrates all these feature improvements in previous modules to produce accurate ranking of the top-10 most relevant videos, thereby achieving 18% improvement inMean Reciprocal Rank (MRR). The main novelty of this research is the Unified Transformer-Based Multiple Task Learning Framework. In a single pass, it performs transcription, semantic similarity, classification, and ranking. This reduces computational costs by 25%, improves overall accuracy by 15%, and decreases inference time by 20%. Our model sets a new standard for efficient, accurate processing of video transcripts with broad applications across a wide array of fields.