Loss decreases fast, but predictions are not improving accordingly

## Summary of my problem I am trying to fine-tune Donut (`naver-clova-ix/donut-base`) with my own dataset for information extraction from business invoices. The training executes well, but I notice that the loss decreases really fast during the first epochs, but if I look at the predictions compared to the ground-truth, I don't really understand this decrease, as the results are bad. After 5/10 epochs, the loss is so small that I don't think the model will keep updating its weights anymore, so even if I run for 100+ epochs, results will stay the same. ## Environment details Based on other opened issues, I am using the following environment: **python version**: 3.8.20 **dependencies** ``` transformers==4.24.0 timm==0.6.13 torch==2.0.1 datasets==3.1.0 ``` **conf.yaml** ``` resume_from_checkpoint_path: null # only used for resume_from_checkpoint option in PL result_path: "./result" pretrained_model_name_or_path: "naver-clova-ix/donut-base" # loading a pre-trained model (from moldehub or path) dataset_name_or_paths: ["../data-donut"] # loading datasets (from moldehub or path) sort_json_key: False # cord dataset is preprocessed, and publicly available at https://huggingface.co/datasets/naver-clova-ix/cord-v2 train_batch_sizes: [2] val_batch_sizes: [1] input_size: [1280, 960] # when the input resolution differs from the pre-training setting, some weights will be newly initialized (but the model training would be okay) max_length: 768 align_long_axis: False num_nodes: 1 seed: 2022 lr: 3e-5 warmup_steps: 300 # 800/8*30/10, 10% num_training_samples_per_epoch: 800 max_epochs: 50 max_steps: -1 num_workers: 4 val_check_interval: 1.0 check_val_every_n_epoch: 1 gradient_clip_val: 1.0 verbose: True special_tokens: - "<s_data-donut>" - "<s_company>" - "</s_company>" - "<s_date>" - "</s_date>" - "<s_address>" - "</s_address>" - "<s_total>" - "</s_total>" ``` ## About my data **total**: 1980 samples - **train**: 1584 samples (80%) - **validation**: 198 samples (10%) - **test**: 198 samples (10%) ## Some graphics: <img width="751" height="480" alt="Image" src="https://github.com/user-attachments/assets/31836440-0b10-4a2e-b6ef-c3220cc4e116" /> <img width="758" height="480" alt="Image" src="https://github.com/user-attachments/assets/dab504e4-6080-4563-a3ec-7de4adb9d4da" /> ## Core issue Could it be a problem related to how the loss is computed? I know that a high learning rate could be causing the fast loss "improvement", but I also tried with different values, and the problem is still the same: at some point, the loss is too low and results stop improving. Have anyone else noticed a similar issue? Were you able to solve it somehow? I would really appreciate any kind of response. Thanks in advance.