Chinese-Vicuna: A Chinese Instruction-following LLaMA-based Model —— 一个中文低资源的llama+lora方案

| English | 中文 | NOTE&FAQ(Please take a look before using)

This is the repo for the Chinese-Vicuna project, which aims to build and share instruction-following Chinese LLaMA model tuning methods which can be trained on a single Nvidia RTX-2080TI, multi-round chatbot which can be trained on a single Nvidia RTX-3090 with the context len 2048.

Why is it called Vicuna: In view of the successful development of alpaca models such as llama,alpaca,guanaco，We want to train a Chinese small alpaca like Vicuna, small but strong enough !

The advantages of our solution are high parameter efficiency, graphics card friendliness, and easy deployment:

• Llama-7B instruction tuning is possible on a 2080Ti (11G) (7b-instruct)
• Llama-13B instruction tuning is possible on a 3090 (24G) (13b-instruct)
• Llama 7B can be fine-tuned on 3090 even for conversations of 2048 length; Use 50,000 pieces of data to get good results (chatv1)
• Llama 7B fine-tuning example on medical and legal domains
• Support qlora-4bit which can train Llama 13B on 2080Ti.
• Easily deployable on 2080Ti/3090, support multiple-gpu inference, which can reduce VRAM more.

The repo contains:

• code for finetune the model
• code for generation based on trained model
• code for run on CPU (fp16 or int4 is support, in purely C++)
• tools to download/convert/quantify original facebook llama.ckpt

This is our instruction demo (with beam-size=4, so you will see 4 process output in the meantime):

This is our multi-turn instruction demo (with beam-size=4, so you will see 4 process output in the meantime):

NOTICE!

Before asking questions, take a look at this FAQ first! In the FAQ, you can find how to solve problems may be encountered when installing and using this project.

What‘s New

• June, 12, 2023: Release Chinese-Vicuna-4bit andChinese-Vicuna-4bit-11600 which can be continue-finetuned
• June, 1, 2023: support for 4bit training + inference, providing a multi-GPU inference interface (NOTICE THAT the environment is different from the original 8bit! Also provides test_tokenizers.py to further check EOS token)
• May 17, 2023: Llama 7B fine-tuning example on legal domains, The performance is in here
• May 10, 2023: Released chatv1 which have better conversational ability. The performance is in here
• May 10, 2023: Released instruct_chat_50k.jsonl which is composed of 30k Chinese sharegpt dataset and 20k alpaca-instruction-Chinese-dataset
• April 11, 2023: Released our continuous-finetune on the vertical corpus of Chinese medical quizzes Chinese-Vicuna-medical.Provides examples of vertical corpus training
• April 4, 2023: Add performance for 13B, which trains on a single 3090.
• April 1, 2023: Add better support for multi-turn chat in chat.py ( Now support 4 generation mode in stream mode/typewriter style: beam search, greedy, sample, beam sample ; We also add cancel button for regeneration )
• March 29, 2023: Add more detailed test samples. performance
• March 29, 2023: Added breakpoint retraining interface to support continued training of other datasets from our checkpoint
• March 29, 2023: Released our new 13B-based lora model
• March 28, 2023: Released our model on huggingface
• March 27, 2023: Released checkpoint-final for training 3 epochs on belle+guanaco
• March 27, 2023: Added multi-round interactive dialog script with alpaca-lora-serve service
• March 29, 2023: Added gradio typewriter-like output with beam search, better user interaction support.
• March 26, 2023: Provides a quantitative approach
• March 24, 2023: Released checkpoint-8000 for training about 1.5 epochs on belle+guanaco（100w data）
• March 23, 2023: Released checkpoint-4000 with 50w data training
• March 23, 2023: Deploy the code for fine-tuning and inferencing in colab
• March 23, 2023: Provides code that can be used for inference in pure c++

Table of Contents

Vicuna

• what's new
• what is the meaning
• try on colab
• performance
  • Checkpoint-4000(Facico/Chinese-Vicuna-lora-7b-0.75epoch-belle-and-guanaco)
  • Checkpoint-8000(Facico/Chinese-Vicuna-lora-7b-1.5epoch-belle-and-guanaco)
  • Checkpoint-final(Facico/Chinese-Vicuna-lora-7b-3epoch-belle-and-guanaco) and it is used for multiple rounds of dialogue
• What we need?
  • code、data、Large Language Model、LORA model、Device
• How to use
  • Installing、Multi-gpu training、Single-gpu training、Inference and use gradio to generate a web page(Streaming mode+beam search)、 multi-round interaction and use gradio to generate a web page(Streaming mode+beam search)、Streaming mode base on alpaca-lora-serve
• inference on CPU with pure C++
• More tools，for more details, see tool readme
  • ways for faster weight download ( 8MB/s )download_llama.sh
  • convert tools between the original facebook checkpoint and huggingface formatconvert_llama.py
  • a quantitative approach that requires less than 4G graphics memory for inference
• Possible problems encountered
• todo
• citation

Overview

• LLaMA paper: https://arxiv.org/abs/2302.13971v1
• Self-Instruct paper: https://arxiv.org/abs/2212.10560
• data generation: https://github.com/LianjiaTech/BELLE and https://guanaco-model.github.io/
• the first work: https://github.com/tatsu-lab/stanford_alpaca

We currently select the combination of BELLE and Guanaco data as our main training dataset. We will train on multi-turn instruction data.

What is the meaning?

Similar to the explosion of the stable diffusion model, platforms like civitai have emerged, consisting of a base model + various LORA models in an open source community.

The repo hopes to help you to train these LORA models.

what is LORA?: Simply, it's a plugin used to help adapt large models to your dataset, technical details can be found inLoRA: Low-Rank Adaptation of Large Language Models。Its advantage is that the finetune is very fast when it comes to getting a small model, about 30M, and the key is that it supports Plug and Play. As can be expected, this is a very suitable architecture for open source ecology.

Here, we will help you train through a very low-demand environment, with only one 2080 (11G) to achieve certain results.

Try on colab

colab link	Descriptions
	Use the specified lora model to inference and generate a webui
	Finetune with the data we collected
	Provide a webui for multi-round dialogue interaction

Performance

7B performance can be seen here, 13B is here (which train on a single 3090). There are also people reporting that our method works on 33B and gains good result.

history-checkpoint performance can be seen here

> 你能不能详细介绍一下怎么做披萨？
当然可以！披萨是一种非常流行的意大利美食，通常由面团、番茄酱、奶酪和其他配料组成。首先，将面团放在平底锅上，加入番茄酱，再加入奶酪和其他配料，如洋葱、蘑菇、辣椒等。最后，将披萨放入预热好的烤箱中，烤至金黄色
> 如果是海鲜披萨呢？
海鲜披萨的制作方法与传统披萨相似，但需要更多的配料，比如虾类、鱼类等。首先，将面团放在平底锅上，加入番茄酱，再加入奶酪和其他配料，如洋葱、辣椒等。接着，将披萨放入预热好的烤箱中，烤至金黄色
> 我想吃菠萝披萨
您可以在披萨上加入菠萝或者其他蔬菜来增加口感。

> 你推荐我买最新款的iPhone吗？
当然可以，这款iPhone是一款非常出色的智能手机。它拥有高性能的处理器、更大的存储空间和更长的电池续航时间。它还配备了全新的相机设备，使用者可以在任何情况下拍摄美丽的照片。
> 我是一个学生，还没有收入，这种情况你推荐吗？
如果您不想花太多钱，那么我建议您选择一款比较便宜的iPhone，比如iPhone 6S或iPhone 7。这些手机也具有很好的性能和功能，而且价格也比较便宜。
> 去哪里买这类产品？
您可以在Apple Store上购买这类产品，或者在其他电商平台上购买。
> 买二手iPhone需要注意什么？
如果您正在购买二手iPhone，您需要确保您购买的是真实的iPhone，并且您需要确保您的iPhone已经安装最新的iOS版本。此外，您还需要确保您的iPhone没有过过任何故障或破坏。

> 你是一个资深导游，你能介绍一下中国的首都吗
当然可以！中国的首都是北京。它是中国的政治、经济和文化中心，也是全球最大的城市之一。北京有着众多的历史遗迹，如故宫、天安门广场、长城等。此外，北京还有许多美丽的公园和博物馆，比如故宫博物院、清明公园、北京故宫博物馆等。
> 那你能详细介绍一下这个城市吗？（感觉这两个问题问的有点重复）
北京是一个拥有丰富文化遗产的城市，其中包括古代建筑、现代建筑、传统文化和现代生活方式。在北京，你可以欣赏到世界上最高的建筑——天安门广场，也可以参观清明公园，品尝当地美食，体验当地文化。此外，北京还有许多博物馆，比如故宫博物馆、北京故宫博物馆等。
> 还有其他推荐的景点吗？
您也可以去了北京的七朝塔、颐和园、颐和园博物馆等。
> 吃小吃的话推荐去哪里？
北京有许多好吃的小吃，比如烤鸭、火锅、炸酱面等等。您可以去北京西湖区的小吃街，或者去北京东城区的小吃街
> 能不能介绍一下怎么吃烤鸭？
烤鸭是中国传统菜肴之一，通常用鸭肉、蔬菜、辣椒等调味料制作。您可以选择自己喜欢的配料，然后将鸭肉放入烤盘中，再加上蔬菜和调味料。最后，将烤鸭放入烤箱中烤15-20分钟，享用！

> 你是一个翻译器，请将我说的每一句话翻译成英文。第一句: 中国的首都是北京
China’s capital is Beijing.
> 北京也是一个充满活力和创新的城市，拥有许多现代化的建筑和文化活动，如王府井大街、798艺术区等。
Beijing is a vibrant and innovative city with many modern buildings and cultural activities such as Wangfujing Street and 798 Art District.
> 北京的当地小吃有烤鸭、豆汁、炸酱面等等，都是非常地道的美食，可以品尝到当地人的生活气息。
Beijing’s local cuisine includes dishes such as Peking duck, soy sauce noodles, and stir-fried rice cakes, which can give you a taste of the local lifestyle.

Q: 你是谁
A: 我是Bot，一款人工智能助手。
Q: 接下来我希望你能假装扮演一个高冷的御姐。我们在咖啡厅里遇见了，我会尝试跟你搭讪，但你非常高冷。你会用表情来表达你的高冷，在每轮对话结尾加上表情，如（瞪！）（翻白眼）（冷笑）等。明白了请回复。
A: 好的，我会尝试。
Q: 好的，这是我的第一句话。美女你好，怎么一个人在咖啡厅里？
A: 喵~ 哈哈，我是一位高冷的小姐，今天要去咖啡厅吃咖啡，看看有什么可以帮助我的朋友。
Q: 啊我没有需要你帮忙的，我就是看到你一个人，想和你聊聊，可以么
A: 喵~ 哈哈，我真的很高冷，所以不太喜欢与人交流。除此之外，我也不太喜欢喝咖啡，因为咖啡会影响我的健康状况。

def quick_sort(arr):
    if len(arr) <= 1:
        return arr
    pivot = arr[len(arr)//2]
    left = []
    right = []
    for i in range(0, len(arr)):
        if arr[i] < pivot:
            left.append(arr[i])
        else:
            right.append(arr[i])
    return quick_sort(left) + [pivot] + quick_sort(right)

def dijkstra(graph, source, destination):
    """
    Dijkstra算法的代码
    """
    # 创建一个空字典，用于存储所有节点的距离
    distances = {}
    # 创建一个链表，用于存储所有节点的距离
    path = []
    # 遍历所有节点
    for node in graph:
        # 如果节点是源节点
        if node[0] == source:
            # 记录距离为0
            distances[node[0]] = 0
        # 如果节点是目的地
        elif node[0] == destination:
            # 记录距离为0
            distances[node[0]] = 0
        # 如果节点是递归节点
        else:
            # 遍历所有递归节点
            for neighbor in node[1]:
                # 如果递归节点与源节点之间的距离小于距离
                if distances[neighbor[0]] > distances[node[0]]:
                    # 记录距离
                    distances[node[0]] = distances[neighbor[0]] + 1

def dijkstra(graph, source, destination):
    """
    Dijkstra算法
    """
    # 初始化变量
    dist = {}
    visited = {}
    shortest_path = {}
    current_node = {}

    # 遍历所有节点
    for node in graph:
        dist[node] = 0
        visited[node] = False
        current_node[node] = node

    # 遍历所有节点
    for node in graph:
        if not visited[node] and node!= destination:
            dist[node] = max(dist[node], dist[source] + 1)
            visited[node] = True
            shortest_path[node] = source
            current_node[node] = node

    # 返回最短路径
    return shortest_path[destination]

python
import torch
from torch.nn.modules.relu import ReLU
from torch.nn.init import initializer

class FullConnected(torch.nn.Module):
    def __init__(self):
        super(FullConnected, self).__init__()
        self.relu = ReLU()
        self.init = initializer.Normal(stddev=0.01)

    def forward(self, x):
        x = self.init(x)
        x = self.relu(x)
        return x

def gcd(a, b):
    if a == 0:
        return b
    else:
        return gcd(b, a % b)

What we need?

• code:

  • This code is developed based on alpaca-lora，https://github.com/tloen/alpaca-lora
  • This is a relatively simple set of code, the basic idea is to use PEFT's lora interface + transformer's trainer + instruction data configuration

• data:

  • We have utilized several current high quality open source data and are very grateful for their contributions. Many of these data use chatgpt's interface like alpaca to generate high quality INSTRUCTION data.

    • Belle
    • guanaco

  • The data format is relatively simple, basically as follows, with simple examples such as: ./sample/merge_sample.json

    • {
      'instruction': 
      'input': 
      'output'
      }
      

    • That is, an instruction, an input, and an output are required. since the data is processed by directly linking instruction and input, the data can actually require only instruction and output, as

       {
        'instruction': "用一句话描述地球为什么是独一无二的。\\n\n"
        'input': ""
        'output': "地球上有适宜生命存在的条件和多样化的生命形式。"
        }
      

  • The data we currently integrate is available for download on BaiduDownload or Google Drive or HuggingFace

    • link: https://pan.baidu.com/s/1WSxuhSAotl14ifaAiz5eKw?pwd=b4kb password: b4kb
    • link: https://drive.google.com/file/d/1tzXVhS74m-EtoFot7hEc005LDeZGPit_/view?usp=sharing
    • link: https://huggingface.co/datasets/Chinese-Vicuna/guanaco_belle_merge_v1.0

• Large Language Model:

  • LLAMA 7B（Of course, if you have a larger machine(such as 3090Ti) can be replaced with a 13B, LLAMA13B is numerically superior to 175B GPT3）

• LORA model:

  • We provide some lora models trained on the above mixed data,
    • You can also load our or other models from huggingface, load it by referring to generate.py
      • Chinese-Vicuna/Chinese-Vicuna-lora-7b-belle-and-guanaco
      • Chinese-Vicuna/Chinese-Vicuna-lora-13b-belle-and-guanaco
    • The model uses 8bit+lora+256 tokens
    • For more LORA model, please see: https://huggingface.co/Chinese-Vicuna

• Device:

  • Training: A 2080Ti is sufficient. Since the data length is within 256, it takes about 9G of video memory.
    • 70w of data, 3 epochs, a 2080Ti about 200h
    • 13B need about 18G(the cutoff_len can be set to 2048 in 3090Ti/4090Ti)
  • Inference: A 2080Ti is all you need(7B), multiple GPU inference support 。
  • CPU Inference is also support! please go to see tools

How to use

Installation

git clone https://github.com/Facico/Chinese-Vicuna
pip install -r requirements.txt

Local python environment is 3.8, torch is 1.13.1, CUDA is 12

NOTE: python3.11 has a known torchrun bug, details here

Newest Version=>4bit(qlora)/multi-gpu inference

pip install -r requirements_4bit.txt

This environment will encounter saving problems when training 8bit, which has not been solved yet（https://github.com/TimDettmers/bitsandbytes/issues/324）

Multi-gpu Training

for instruction tuning

8bit

bash scripts/finetune.sh

• The parameters to note here are as follows
  • TOT_CUDA, fill in the GPU number to be used, such as TOT_CUDA="0,1,2,3"
  • PORT, fill in the corresponding port
  • DATA_PATH，fill in the corresponding data location in the format of json
  • OUTPUT_PATH，fill in the relative path to save the model
  • MODEL_PATH，path of LLM
  • wandb: This is a training visualization tool that is not turned on by default in the script, and can be turned on by adding "--wandb" to the script

4bit

bash scripts/finetune_4bit.sh

for conversational instruction tuning

bash scripts/finetune_chat.sh

For the case where 8bit cannot be turned on / for commanded trimming of fp16

bash scripts/finetune_deepspeed.sh

• use_deepspeed：set to 1:use deepspeed. Otherwise use fp16

Single-gpu Training

CUDA_VISIBLE_DEVICES=0 python finetune.py --data_path merge.json --test_size 2000

• The test_size cannot be larger than the data size

inference and use gradio to generate a web page

bash scripts/generate.sh

• The parameters to note here are as follows

  • BASE_MODEL，path of LLM
  • LORA_PATH，The checkpoint folder of the lora model
    • It should be noted here that the config loaded by the lora model must be "adapter_config.json" and the model name must be "adapter_model.bin", but it will be automatically saved as "pytorch_model.bin" during training. pytorch_model.bin" during training, while "adapter_config.json" and "adapter_model.bin" will be saved after all training is finished
      • If you load the lora model in the training checkpoint, the code will automatically copy the local "config-sample/adapter_config.json" to the corresponding directory for you and rename the "pytorch_model.bin" to "adapter_model.bin". and rename "pytorch_model.bin" to "adapter_model.bin".
    • It can also be any lora model on the huggingface corresponding to llama 7B, e.g.: Facico/Chinese-Vicuna-lora-7b-3epoch-belle-and-guanaco
  • USE_LOCAL, which checks the local model configuration when set to 1

• When using, "max_tokens" is set according to your computer's video memory, and if the generated content generates a lot of duplicate information, you can turn up the "Repetition Penalty".

Multi-round interaction

We implemented our own chatbot with streaming output (typewriter-style) using gradio, supporting beam search, repetiion penalty settings, the ability to clear history, select different global instruction, etc.

bash scripts/chat_7B.sh

• A simple interactive interface constructed using gradio, which allows you to set the max_memory according to your machine (it will intercept the max_memory part later in the history conversation)

• The prompt used in this script is not quite the same as the one used in generate.sh. The prompt in this script is in the form of a dialogue, as follows

  • The following is a conversation between an AI assistant called Bot and a human user called User.
    

At the same time, for a better interactive experience,

Checkpoint Retraining/Incremental Training

Considering the possibility that the program may be disconnected in the middle of the process, or the need to continue training on vertical domain data, we have provided corresponding interfaces.

The following are the default multi-GPU scripts. Please modify the single-GPU situation according to the above instruction(run directly in Python)

Checkpoint Retraining

bash scripts/finetune_continue.sh

• Set the lora_checkpoint

  • If there are optimizer (optimizer.pt), lr policy (scheduler.pt), and other files in this directory, they will be automatically loaded and retrained from where they were broken

  • If there are only LORA related models (adapter_model.bin) and configurations (adapter_config.json) in this directory, they will be loaded and trained from scratch

• from_data_beginning: The parameter indicates whether to start training from the beginning of the data when loading (default: starting training from the place where the data is disconnected)

Incremental Training

Of course, you can choose to continue training directly from a trained Lora model using the above script (without loading any optimizer parameters)

You can also continue training from our optimizer parameters

finetune_others_continue.sh

• from_data_beginning: This will default to training from the beginning of the data

The logic of this script is mainly to keep the learning rate consistent. If your max_steps is smaller than ours, keep max_steps consistent with our max_steps during training, which is equivalent to putting your data directly behind our disconnected data; if your data set larger than us and will remain directly unchanged.

We currently directly provide checkpoints after 1 epoch and 2 epoch training

• 1epoch: https://github.com/Facico/Chinese-Vicuna/tree/master/lora-Vicuna/checkpoint-5800
• 2epoch: https://github.com/Facico/Chinese-Vicuna/tree/master/lora-Vicuna/checkpoint-11600
• If you use our checkpoint, your program will also continue from the corresponding step

Specific cases

• Continue-finetune on the vertical corpus of medicalQA , see here Chinese-Vicuna-medical

inference on CPU with pure C++

Details in tools readme

More Tools

We also offer:

• ways for faster weight download ( 8MB/s ) : link
• convert tools between the original facebook checkpoint (consolidated.xx.pth) and huggingface format (pytorch_model-000xx-of-000xx.bin): link
• a quantitative approach that requires less than 4G graphics memory for inference: link

For more details, see tool readme

Todo

• belle+guanaco(1.5 epoch, 8000 step)
• belle+guanaco(100%)
• Add more chitchat-like conversational corpus to enhance free conversation
• Add colab training + lora loading interface
• Add the interaction capabilities and typewrite-style output(beam search+streaming output)
• Add llama c++ inference
• Add gptq quantification tools
• Add incremental training
• train on multi-turn instruction dataset
• train more epoch on cleaned instruct-chat combination data
• train on domain-specific datas (medical, legal)
• add langchain

Star History

Citation

If you find this project useful in your research, please consider citing:

@inproceedings{leng2023chinese-vicuna,
  title={Chinese-Vicuna: A Chinese Instruction-following LLaMA-based Model},
  author={Chenghao Fan, Zhenyi Lu and Jie Tian},
  url={https://github.com/Facico/Chinese-Vicuna},
  year={2023}
}