Fine-Tuning Transformers vs. Few-Shot LLMs for Bangla NLP

Table Of Links

Abstract

I. INTRODUCTION

II. RELATED WORKS

III. BACKGROUND STUDY

IV. CORPUS CREATION

V. IMPLEMENTATION DETAILS

VI. RESULT ANALYSIS & DISCUSSION

VII. FUTURE RESEARCH DIRECTIONS

VIII. CONCLUSION AND REFERENCES

A. Political Sentiment Analysis using PLMs

We provide a thorough method for detecting subtle sentiment patterns in political conversation, which includes preparing the data, model fine-tuning, and performance evaluation. This approach provides a solid foundation for analyzing public opinion processes in the field of politics. Our research implementations are accessible to the public at: GitHub5

Step 1) Text Preprocessing: Our dataset “Motamot” undergoes preprocessing to ensure compatibility with PLMs. A series of normalization steps are applied, specifically designed for Bengali text. These steps include handling whitespace, commas, URLs, Unicode characters, and correcting quotation marks throughout the dataset.

Step 2) Fine-tuning Procedure: During the fine-tuning process, the initialized PLMs are trained on the Political Sentiment Analysis dataset using transfer learning techniques. Model parameters are updated via gradient descent optimization algorithms, with the objective of minimizing a suitable loss function, such as cross-entropy loss. We employ the AdamW optimizer and CrossEntropyLoss as the loss function.

Step 3) Training Settings: Hyperparameters play a crucial role in shaping the learning process and influencing model performance. Key hyperparameters such as learning rate, batch size, and epochs are adjusted during fine-tuning to optimize performance and prevent overfitting. Table I provides detailed guidance for hyperparameter optimization to enhance model performance.

Step 4) Performance Assessment: The performance of finetuned PLMs is evaluated on a held-out test set using predefined evaluation criteria. Metrics such as Accuracy, Precision, Recall, and F1 score are utilized to objectively assess model performance. Table II presents comprehensive performance metrics, on the effectiveness of fine-tuned models for Political Sentiment Analysis tasks.

B. Political Sentiment Analysis using LLMs

Data Selection: We randomly selected 300 data points from our “Motamot” dataset to assess the effectiveness of Zero Shot and Few Shot prompts. Specifically, we focused on data instances with a “short description” column containing more than 25 words. Each label category, including Positive and Negative sentiments, comprises 100 instances. These samples were extracted from the training subset of the “Motamot” dataset. Considering the cost implications associated with using the OpenAI GPT-3.5 Turbo API and Gemini 1.5 Pro API, we opted to limit our experiments to this subset of the dataset rather than the entire corpus.

Prompting Template: In LLMs, a prompt directs text generation by specifying input parameters, crucial for task customization and output refinement. Its form varies based on the intended task, guiding LLM behavior effectively.

• Zero-shot prompting for Political Sentiment Analysis: Zero-shot prompting in Political Sentiment Analysis involves guiding the model to predict the sentiment label (Positive or Negative) associated with a given text, without prior training on such sentiments. For instance, in a Positive sentiment scenario, the text may express support or agreement with a political figure or policy.

Conversely, in a Negative sentiment scenario, the text may convey criticism or dissatisfaction. The model aims to accurately classify the sentiment based on the textual content alone. Figure 2 displays Zero-shot prompting with “Motamot” dataset, showing premise-text pairs labeled Positive or Negative. Additionally, it highlights Gemini 1.5 Pro’s Zero-shot accuracy in Political Sentiment Analysis without explicit training.

• Few-shot prompting for Political Sentiment Analysis:

In Political Sentiment Analysis, few-shot prompting is a technique aimed at enhancing a model’s ability to discern sentiment in text with minimal training data. In a 5-shot scenario, the model is provided with just five labeled examples, allowing it to grasp the nuances of sentiment expression. Similarly, in a 10-shot scenario, the model’s exposure to labeled data doubles, refining its understanding further.

Extending on this concept in a 15-shot scenario, the model is provided with fifteen labeled samples, providing a wider range of emotion expressions for learning. With each increase in shots, the model’s ability to generalize and properly predict emotion in new, unlabeled texts improves. This strategy finds a compromise between data efficiency and prediction performance, allowing the model to make more accurate judgments about political emotions as it fulfills more labeled samples.