# PaddleNLP PEFT API

PaddleNLP PEFT API provides single-card/distributed LoRA and Prefix-Tuning. Users can quickly adapt models for low-parameter fine-tuning by defining models, datasets, and corresponding configurations.

## Prerequisites
### LoRA
<div align="center">
<img src=https://github.com/PaddlePaddle/PaddleNLP/assets/37530985/63d56558-247a-4a8d-a6ca-121c820f7534 width=30% height=30% />
</div>

In large model networks, there are many linear layers that require intensive matrix multiplication computations, which typically have full rank and are difficult to optimize. The LoRA paper demonstrates that randomly projecting input representations into smaller subspaces can still enable effective learning while saving significant computational memory requirements. Implementation: For pre-trained weight matrices, introduce two low-rank matrices $AB$ (the two orange matrices in the figure) to approximate the weight update process $W_0+\Delta W=W_0+BA$, where $B \in \mathbb{R}^{d \times r}, A \in \mathbb{R}^{r \times k}$, with $r$ being much smaller than the original weight matrix's rank. During training, $W_0$ parameters remain frozen, while only gradients for matrices $\mathrm{A}$ and $\mathrm{B}$ are updated. The forward propagation formula is:

$$
h=W_{0}x+BAx
$$

Due to reduced training parameters, this decreases the storage of intermediate variables during training, thereby saving substantial training memory. For more details, refer to the LoRA [paper](https://arxiv.org/abs/2106.09685).

### Prefix-tuning

<div align="center">
<img src=https://github.com/PaddlePaddle/PaddleNLP/assets/37530985/8baf6943-4540-4c02-8540-35f977acc077 width=40% height=40% />
</div>

Prefix-tuning is a lightweight fine-tuning solution for NLG-type downstream tasks. Inspired by prompt learning, it incorporates additional prefix embeddings as continuous prompts. The prefix embeddings are generated by a dedicated prefix encoder network and are inserted before the hidden_state at each layer of the language model as past_key_values. Similar to LoRA, it freezes all parameters of the pre-trained model and only updates gradients for the prefix embeddings, resulting in training parameters being just 0.1% of regular SFT. Prefix-tuning achieves comparable performance to SFT with full data samples and even surpasses SFT in few-shot scenarios. For details, refer to the Prefix-tuning [paper](https://arxiv.org/abs/2101.00190).

## Quick Start
### LoRA

1. To fine-tune a model with LoRA, first define the LoRAConfig, then construct the LoRAModel using this configuration. Freeze the backbone parameters via mark_only_lora_as_trainable:

```python
from paddlenlp.peft import LoRAConfig, LoRAModel, mark_only_lora_as_trainable

# Define LoRA parameters
lora_config = LoRAConfig(
    target_modules=["query_proj", "value_proj"],
    r=8,
    lora_alpha=16,
    merge_weights=True
)

# Create LoRA model
model = LoRAModel(lora_config, model)

# Freeze non-LoRA parameters
mark_only_lora_as_trainable(model)
```
```python
    from paddlenlp.peft import LoRAConfig, LoRAModel
    from paddlenlp.transformers import AutoModelForCausalLM

    model = AutoModelForCausalLM.from_pretrained('facebook/llama-7b')
    target_modules = [".*q_proj.*", ".*v_proj.*", ".*k_proj.*"]
    lora_rank = 8
    lora_config = LoRAConfig(
        target_modules=target_modules,
        r=lora_rank,
        lora_alpha=2 * lora_rank,
    )
    model = LoRAModel(model, lora_config)
    model.mark_only_lora_as_trainable()
    model.print_trainable_parameters()
```

2. Model Saving and Loading

The saving and loading of LoRAModel is similar to regular models, both using save_pretrained/from_pretrained calls
```python
    # Save
    model.save_pretrained('lora_path')
```
Paddle will save the LoRAModel's matrix AB weights as lora_mode_state.pdparams file, and LoRAConfig configuration as lora_config.json file in the lora_path directory.

When loading the model weights for inference later, simply use from_pretrained:
```python
    from paddlenlp.transformers import AutoModelForCausalLM
    + from paddlenlp.peft import LoRAModel, LoRAConfig

    # Load
    + config = LoRAConfig.from_pretrained('lora_path')
      model = AutoModelForCausalLM.from_pretrained('facebook/llama-7b')
    + model = LoRAModel.from_pretrained(model, 'lora_path')
      model.eval()
```

### class LoRAConfig
```text
Parameters:

    --r
                        Default is 8, the rank of the LoRA A/B matrices.

    --target_modules
                        Specifies which modules to apply LoRA to. Accepts a list of module names
                        or regular expressions. E.g., ['q', 'v'] or '.*decoder.*(SelfAttention|EncDecAttention).*(q|v)$'

    --trainable_modules
                        Specifies additional modules (beyond LoRA parameters) that require gradient updates.
                        Accepts a list of module names or regular expressions. E.g., ['q', 'v'] or '.*decoder.*(SelfAttention|EncDecAttention).*(q|v)$'

    --lora_alpha
                        Default is 8, the alpha value for LoRA (int type).

    --lora_dropout
                        Default is 0.0, dropout rate (float type).

    --merge_weights
                        Default is False. This parameter will be deprecated. Please use model.merge() or model.unmerge() instead.

    --trainable_bias
                        Specifies trainable biases. Options: ['lora', 'all'].

    --enable_lora_list
                        Specifies whether to use `MergedLoRALinear`. If not specified, `LoRALinear` will be used by default.

    --tensor_parallel_degree
                        Default is -1. Control parameter for multi-GPU tensor parallelism. Must match the base model's tensor_parallel_degree configuration.

    --dtype
                        Data type configuration for LoRA matrix parameters.

    --head_dim
                        Head dimension for multi-head attention. Only used by `LoRAMergedLinear` and
                        `ColumnParallelLoRAMergedLinear`.
```

### class LoRAModel

```text
LoRA model class that ... [Original Chinese description truncated; please provide full text for complete translation]
```
```text
Parameters:

    --model
                        Specify the base model, which must be an object of type nn.Layer

    --lora_config
                        Specify LoRAConfig to configure LoRAModel

Key functions:

    - mark_only_lora_as_trainable()

        Marks LoRA-related layers in the model as trainable while keeping other layers frozen.


    - save_pretrained(save_directory, merge_tensor_parallel)
        --save_directory
                        Path to save directory
        --merge_tensor_parallel
                        Whether to merge tensor parallel parameters, default is True

        If merge_tensor_parallel is True and the model's tensor_parallel_degree > 1, retrieves the trainable state_dict and merges parameters using _merge_trainable_tensor_parallel. When tensor_parallel_degree > 1, only the main process performs saving.


    - from_pretrained(model, lora_path)
        --model
                        The model object to load LoRA weights into
        --lora_path
                        Path containing saved LoRA weights and config

        Loads LoRA weight parameters from pretrained model and applies them to the given model for subsequent tasks.


    - print_trainable_parameters()

        Iterates through all parameters, counts those requiring gradient updates, and prints statistics.

```

### Prefix-tuning
1. Setting Prefix-tuning parameters
```python
    from paddlenlp.transformers import AutoModelForCausalLM

    model = AutoModelForCausalLM.from_pretrained('facebook/llama-7b')

    prefix_config = PrefixConfig(
        num_prefix_tokens=64,
        num_attention_heads=model.config.n_head,
        num_hidden_layers=model.config.n_layer,
        hidden_size=model.config.hidden_size,
        prefix_projection=False,
        prefix_projection_hidden_size=model.config.hidden_size
    )
    model = PrefixModelForCausalLM(model=model, prefix_config=prefix_config)
    model.mark_only_prefix_as_trainable()
    model.print_trainable_parameters()
```

2. Model Saving and Loading

Consistent with LoRAModel, use save_pretrained/from_pretrained APIs
```python
    # Save
    model.save_pretrained('prefix_path')
```
Paddle will save the prefix_encoder (containing Embedding layer and Linear layers) model weights and PrefixConfig configuration as prefix_config.json in the prefix_path directory. When loading the model weights for inference later, simply use from_pretrained:
```python
      from paddlenlp.transformers import AutoModelForCausalLM
    + from paddlenlp.peft import PrefixModel, PrefixConfig

    # Load
    + config = PrefixConfig.from_pretrained('prefix_path')
      model = AutoModelForCausalLM.from_pretrained('facebook/llama-7b')
    + model = PrefixModelForCausalLM.from_pretrained(model, 'prefix_path')
      model.eval()
```

### class PrefixConfig
Parameters:

    --prefix_dropout
                        Default is 0.0, prefix projection dropout rate, float type

    --num_prefix_tokens
                        Number of prefix tokens, int type

    --num_attention_heads
                        Number of attention heads, int type

    --multi_query_group_num
                        Number of multi-query groups, int type

    --num_hidden_layers
                        Number of hidden layers in base model, int type

    --hidden_size
                        Hidden size of base model, int type

    --prefix_projection
                        Default is False, whether to apply projection to prefix tokens, bool type

    --prefix_projection_hidden_size
                        If prefix_projection is set to True, specifies the hidden size
                        for projection operation, int type

    --tensor_parallel_degree
                        Default is -1, tensor parallel degree for multi-GPU parallelism control

    --dtype
                        Data type for prefix embeddings parameters
```text
Parameters:

    --model
                        Specify the base model, which must be an object of type nn.Layer

    --prefix_config
                        Specify PrefixConfig to configure PrefixModelForCausalLM

    --postprocess_past_key_value
                        Specify the function for post-processing past_key_value

    --pad_attention_mask
                        Specify the function to process pad_attention_mask for new prefix embeddings

Key Functions

    - mark_only_prefix_as_trainable()

        Marks only the Prefix embeddings and Prefix projection layers as trainable while freezing parameters in other layers.

    - save_pretrained(save_directory, merge_tensor_parallel)
        --save_directory
                        Path to save directory
        --merge_tensor_parallel
                        Whether to merge tensor parallel parameters, default True

        If merge_tensor_parallel is True and model's tensor parallel degree > 1, retrieves trainable state_dict and uses _merge_trainable_tensor_parallel to merge tensor parallel trained state_dict. When merge_tensor_parallel is True and tensor parallel degree > 1, only main process will perform save operation.

    - from_pretrained(model, prefix_path, postprocess_past_key_value, pad_attention_mask)
        --model
                        Model object to load Prefix weights into
        --prefix_path
                        Path containing Prefix weights and config files
        --postprocess_past_key_value
                        Same as constructor parameter in PrefixModelForCausalLM
        --pad_attention_mask
                        Same as constructor parameter in PrefixModelForCausalLM

        Loads Prefix weights from pretrained model into given model, enabling subsequent predictions or training.

    - print_trainable_parameters()

        Iterates through all weight parameters, counts parameters with gradient updates, and prints statistics.
```
For more detailed usage, please refer to the [finetuning script](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/llm/run_finetune.py) version, as well as the corresponding launch script writing method (found in the [README.md](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/llm/README.md) file).
