PetGPT
Thanks to LLaMA and Alpaca-LoRA, you can train your own PetGPT at home. This GitHub repository offers a step by step guide in its README.md file.
LLaMA (Large Language Model Meta AI)
This repository assumes that you have already found a way to download the checkpoints and tokenizer for LLaMA (e.g., by filling out this google form). You should create a subdirectory (named LLaMA
) within this repository having a structure similar to the following tree.
LLaMA
βββ 13B
β βββ checklist.chk
β βββ consolidated.00.pth
β βββ consolidated.01.pth
β βββ params.json
βββ 30B
β βββ checklist.chk
β βββ consolidated.00.pth
β βββ consolidated.01.pth
β βββ consolidated.02.pth
β βββ consolidated.03.pth
β βββ params.json
βββ 65B
β βββ checklist.chk
β βββ consolidated.00.pth
β βββ consolidated.01.pth
β βββ consolidated.02.pth
β βββ consolidated.03.pth
β βββ consolidated.04.pth
β βββ consolidated.05.pth
β βββ consolidated.06.pth
β βββ consolidated.07.pth
β βββ params.json
βββ 7B
β βββ checklist.chk
β βββ consolidated.00.pth
β βββ params.json
βββ llama.sh
βββ tokenizer_checklist.chk
βββ tokenizer.model
4 directories, 26 files
Clone the LLaMA repository to make sure that everything works as expected.
git clone https://github.com/facebookresearch/llama.git
The LLaMA repository is already included here for reproducibility purposes in the folder named llama
. You can now run the following commands.
cd llama
torchrun --nproc_per_node 1 example.py --ckpt_dir ../LLaMA/7B --tokenizer_path ../LLaMA/tokenizer.model
Converting LLaMA to Hugging Face
Create an empty directory within this repository called LLaMA_HF
. The following two scripts will then help you convert the LLaMA checkpoints and tokenizer to the Hugging Face format.
convert_llama_tokenizer_to_hf.ipynb
convert_llama_weights_to_hf.ipynb
These two scripts are simplied versions of convert_llama_weights_to_hf.py for pedagogical purposes.
This should result in a subdirectory (named LLaMA_HF
) within this repository having a structure similar to the following tree.
LaMA_HF
βββ config.json
βββ generation_config.json
βββ pytorch_model-00001-of-00002.bin
βββ pytorch_model-00002-of-00002.bin
βββ pytorch_model.bin.index.json
βββ special_tokens_map.json
βββ tokenizer_config.json
βββ tokenizer.model
Exploratory Data Analysis
Clone the Alpaca-LoRA repository.
git clone https://github.com/tloen/alpaca-lora.git
The Alpaca-LoRA repository is already included here for reproducibility purposes in the folder named alpaca-lora
. Within this folder, there is a file called alpaca_data_cleaned.json
. This file contains a cleaned and curated version of the dataset used to train the original Alpaca. The following script will help you explore this dataset and build some intuition.
exploratory_data_analysis.ipynb
Fine Tuning
You can now use the following commands to finetune the LLaMA model on the alpaca_data_cleaned.json
dataset.
cd alpaca-lora
mkdir output
WORLD_SIZE=4 CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --nproc_per_node=4 --master_port=1234 finetune.py --base_model ../LLaMA_HF --data_path alpaca_data_cleaned.json --output_dir output
The command given above works for a machine with 4 GPUs. You should adjust it according to your own setup.
Once the finetuning process is finished, it will write the model artifacts in a folder called output
.
alpaca-lora/output/
βββ adapter_config.json
βββ adapter_model.bin
βββ checkpoint-1000
β βββ optimizer.pt
β βββ pytorch_model.bin
β βββ rng_state_0.pth
β βββ rng_state_1.pth
β βββ rng_state_2.pth
β βββ rng_state_3.pth
β βββ scaler.pt
β βββ scheduler.pt
β βββ trainer_state.json
β βββ training_args.bin
βββ checkpoint-600
β βββ optimizer.pt
β βββ pytorch_model.bin
β βββ rng_state_0.pth
β βββ rng_state_1.pth
β βββ rng_state_2.pth
β βββ rng_state_3.pth
β βββ scaler.pt
β βββ scheduler.pt
β βββ trainer_state.json
β βββ training_args.bin
βββ checkpoint-800
βββ optimizer.pt
βββ pytorch_model.bin
βββ rng_state_0.pth
βββ rng_state_1.pth
βββ rng_state_2.pth
βββ rng_state_3.pth
βββ scaler.pt
βββ scheduler.pt
βββ trainer_state.json
βββ training_args.bin
3 directories, 32 files
Generating Text
Here is how you can interact with the finetuned model.
python generate.py --load_8bit --base_model ../LLaMA_HF --lora_weights output --share_gradio False
The final screen hosted on http://0.0.0.0:7860 will look like the following image.