MSc Computer Science thesis · Pokhara University
Image Captioning in Nepali
Generating Nepali descriptions of images with a Swin Transformer encoder and GPT-2 decoder, with experiments in language-specific tokenisation, model architecture, and dataset scale.
Research paper co-authored with Dr. Basanta Joshi.
- Architectures compared
- 2
- Experiment configurations
- 11
- Captioning datasets
- 2
The problem
Image captioning connects visual recognition with language generation. This research investigates generating captions in Nepali, using adapted Flickr8k and Flickr30k image–caption datasets. The challenge is to represent visual content and generate useful descriptions in a language with limited captioning resources and different tokenisation needs from English.
My contribution
- Developed Model A using a Swin Transformer visual encoder and a GPT-2 text decoder connected through cross-attention.
- Investigated a Nepali-trained tokenizer and compared it with the default GPT-2 tokenizer, including caption scores and validation time.
- Compared Model A with a ConvNeXtV2–Swin–GPT-2 hybrid, varying encoder depths, attention heads, decoder layers, and dataset size across 11 configurations.
- Evaluated generated captions using BLEU-1 through BLEU-4 and METEOR, and examined qualitative examples alongside reported literature results.
Approach
Prepare image–caption pairs
Use Nepali adaptations of Flickr8k and Flickr30k, with five reference captions per image. Clean caption text, handle beginning/end/padding tokens, and resize input images to 224 × 224 pixels. The reported training/validation/test proportions are 75% / 12.5% / 12.5%.
Connect vision and language
Model A uses Swin to encode visual features and GPT-2 to generate captions autoregressively through cross-attention. Model B adds ConvNeXtV2 feature extraction before Swin, keeping the language-generation approach comparable.
Adapt tokenisation and architecture
Train a tokenizer on Nepali caption text and compare it with the default tokenizer. Explore four-, six-, and twelve-layer decoder configurations, encoder depth, and attention heads. The selected Model A configuration uses six decoder layers.
Train and evaluate
Implement experiments with Hugging Face and TensorFlow/Keras in Kaggle. Tune learning rate and batch size, then compare configurations across 10, 20, 40, and 60 epochs using caption metrics and generated examples. GPU memory limits constrain the configurations explored.
Results & findings
On Flickr30k, Swin + GPT-2 recorded BLEU-1 of 0.69, BLEU-4 of 0.35, and METEOR of 0.22. The hybrid matched its BLEU-2, BLEU-3, BLEU-4, and METEOR scores, with a slightly lower BLEU-1 of 0.68. On Flickr8k, the two architectures showed different strengths across the BLEU metrics.
| Model / dataset | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | METEOR |
|---|---|---|---|---|---|
| Swin + GPT-2 · Flickr8k | 0.59 | 0.43 | 0.32 | 0.23 | 0.19 |
| Swin + GPT-2 · Flickr30k | 0.69 | 0.54 | 0.44 | 0.35 | 0.22 |
| ConvNeXtV2 + Swin + GPT-2 · Flickr8k | 0.58 | 0.44 | 0.34 | 0.26 | 0.19 |
| ConvNeXtV2 + Swin + GPT-2 · Flickr30k | 0.68 | 0.54 | 0.44 | 0.35 | 0.22 |
Best reported configurations for each architecture and dataset. These are caption-evaluation scores, rather than classification accuracy percentages.
- For Swin + GPT-2, moving from Flickr8k to Flickr30k accompanied an increase in BLEU-1 from 0.59 to 0.69 and BLEU-4 from 0.23 to 0.35. The datasets and batch sizes differ, so this comparison does not isolate dataset size as the only cause.
- On Flickr8k, the hybrid scored slightly lower on BLEU-1 but higher on BLEU-2 through BLEU-4 than Swin + GPT-2. Adding a CNN feature extractor did not produce a uniform advantage across datasets and metrics.
- Compared with the reported CNN–Transformer reference on Flickr8k, Swin + GPT-2 improved BLEU-1 and BLEU-2, while the reference retained higher BLEU-3 and BLEU-4. This supports a mixed comparison rather than a claim of improvement on every metric.
What changed with Nepali tokenisation?
| Tokenizer | BLEU-1 | BLEU-4 | Validation / epoch |
|---|---|---|---|
| Default GPT-2 | 0.52 | 0.16 | 7m 31s |
| Nepali-trained | 0.55 | 0.20 | 3m 16s |
Minimal Swin + GPT-2 configurations on Flickr8k, comparing caption scores at 60 epochs. Validation timings are reported per epoch for these experiments. They are not general inference-speed benchmarks.
Comparison with the reported CNN–Transformer reference
| Flickr8k model | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 |
|---|---|---|---|---|
| CNN–Transformer · Subedi et al. | 0.52 | 0.42 | 0.37 | 0.34 |
| Swin + GPT-2 | 0.59 | 0.43 | 0.32 | 0.23 |
| ConvNeXtV2 + Swin + GPT-2 | 0.58 | 0.44 | 0.34 | 0.26 |
Reference scores are from Subedi et al. Differences in preprocessing and evaluation may affect comparability.
Experiments & analysis
Select a figure to view it at full size.
Limitations & what I learned
- The caption datasets focus on human activities and use Nepali adaptations of existing English datasets. Translation noise and limited linguistic and visual coverage can affect generalisation.
- BLEU and METEOR capture aspects of overlap with reference captions. They do not fully establish semantic correctness, fluency, grounding, or usefulness to Nepali-speaking readers.
- Architecture and training choices were constrained by available GPU memory. The reported comparisons do not establish uncertainty across repeated runs or a universal advantage for one architecture.
Where I would take this next
- Add human evaluation by Nepali speakers and complementary metrics such as CIDEr, SPICE, ROUGE-L, and BERTScore.
- Evaluate more diverse image domains and caption styles, and audit image-level splits and tokenizer-training boundaries when reproducing the experiments.
- Measure inference latency, memory use, and caption quality together before considering deployment in resource-constrained settings.