Bangla NLP Architecture Guide: Pre-trained Transformers vs. Frontier LLMs

Table Of Links

Abstract

I. INTRODUCTION

II. RELATED WORKS

III. BACKGROUND STUDY

IV. CORPUS CREATION

V. IMPLEMENTATION DETAILS

VI. RESULT ANALYSIS & DISCUSSION

VII. FUTURE RESEARCH DIRECTIONS

VIII. CONCLUSION AND REFERENCES

A. Pre-trained Language Models on Bangla

Developing language-specific models presents significant hurdles, especially for languages like Bengali that have limited resources. Despite this, recent improvements have resulted in an increase in pretrained language models gaining popularity. These models have demonstrated SOTA performance across diverse downstream tasks. In this context, the efficacy of such models is briefly explored below.

1. ELECTRA base: ELECTRA by Google employs “replaced token detection” to distinguish real tokens from substitutes, enhancing both comprehension and computational efficiency. By focusing on altered tokens, it refines context understanding and language semantics during training. BanglaBERT [2] generator, a derivative of ELECTRA, utilizes masked language modeling (MLM) on extensive Bengali corpora for pre-training.

2. BERT base: BERT (Bidirectional Encoder Representations from Transformers) is a cutting-edge pre-trained language model created by Google researchers. It transformed NLP by proposing a bidirectional method to context comprehension. Unlike prior models that processed text sequentially, BERT evaluates both the left and right contexts at the same time, capturing deeper semantic significance. Bangla BERT Base [11] and mBERT [3] are pre-trained Bengali language models based on BERT’s pioneering mask language modeling framework.

3. ALBERT large: ALBERT emphasizes flexibility and performance, shrinking model size and computational requirements without sacrificing efficiency. It surpasses conventional BERT models through strategies like parameter reduction and layer parameter sharing. ALBERT’s Lite design, featuring parameter sharing and factorized embedding, maintains quality with fewer parameters. SahajBERT4 , an ALBERT variant, is trained for Bengali using MLM.

4. RoBERTa base: XLM-RoBERTa [12], an extension of RoBERTa, is a multilingual model trained on an extensive corpus covering over 100 languages, enabling it to process diverse information sources. Operating through unsupervised learning, it autonomously learns from vast amounts of text data without human labeling. Employing masked language modeling, it predicts missing elements within text, fostering a deep understanding of word and concept relationships. Additionally, XLM-RoBERTa possesses automatic language detection capabilities, facilitating seamless multilingual processing without external cues.

B. Large Language Models on Bangla

1. GPT-3.5 Turbo: GPT-3.5 Turbo [13] has considerable advances in comprehending and executing instructions, making it ideal for activities that need particular formatting or outputs, such as creative content development. Its fine-tuning capability enables developers to adjust its behavior to specific requirements, hence improving performance for a variety of apps. For example, the model can be fine-tuned to consistently employ a specific language or to simplify cues for desirable replies.

   These advancements elevate GPT-3.5 Turbo to the top of its series, providing a cost-effective and adaptable solution for text generating workloads. With a broad context window of 16,385 tokens and enhanced precision in formatting, it successfully tackles encoding concerns for non-English language functions while providing speedy answers, limited at 4,096 output tokens.

2. Gemini 1.5 Pro: Gemini 1.5 Pro [14], a member of Google DeepMind’s Gemini series, is a cutting-edge multimodal model adept at processing text, audio, and video, expanding its utility across various tasks. Its standout feature lies in its ability to grasp long-context information, fostering nuanced understanding and insightful responses.

   While specifics on multilingual capabilities are somewhat limited, Gemini Pro likely excels in processing text across multiple languages. Boasting a substantial input token limit of 30,720, an output token limit of 2,048, along with stringent safety measures and a 60 requests per minute rate limit, it emerges as a versatile tool for diverse linguistic endeavors, ensuring both efficiency and effectiveness.