HuatuoGPT (华佗GPT), Towards Taming Language Models To Be a Doctor.

✨ Latest News

• [06/30/2023]: Evaluation data of HuatuoGPT released in the eval/ folder.
• [06/30/2023]: Release the code, model weights of HuatuoGPT-7B and HuatuoGPT-13B
• [05/25/2023]: Release the tech report and the HuatuoGPT demo.

⚡ Introduction

Welcome to the repository of HuatuoGPT, a large language model (LLM) trained on a vast Chinese medical corpus. Our objective with HuatuoGPT is to construct a more professional ‘ChatGPT’ for medical consultation scenarios.

Here is a list of what has been released:

1. HuatuoGPT-SFT-data: A hybrid SFT data capitalizing on both strengths to endow the model with Doctor-like and Patient-friendly characteristics.
2. HuatuoGPT model: HuatuoGPT model weights(HuatuoGPT-7B and HuatuoGPT-13B) and the online demo. HuatuoGPT-7B is trained on Baichuan-7B and HuatuoGPT-13B is trained on Ziya-LLaMA-13B-Pretrain-v1.
3. Medical evaluation benchmark: an evaluation method used to evaluate LLMs in medical scenarios.

💭 Motivation

• To address the growing demand for quick medical consultations both online and in hospitals that do not necessarily require deep medical knowledge. We believe that LLMs like HuatuoGPT can be effectively utilized to meet these demands, freeing up physicians’ time and energy for more complex cases.
• To provide open data for training medical LLMs. Building high-quality instruction training data for LLMs is essential, but it can be also challenging. We have constructed medical instruction data using various methods and made it publicly available. This dataset can be combined with other datasets to train one's own medical 'ChatGPT'.
• To emphasize the importance of carefully evaluating the ability of medical LLMs before using them to offer medical assistance to patients. We recognize the potential benefits of LLMs in the medical field, but also acknowledge the need for thorough evaluation and testing to ensure patient safety and accurate diagnoses.

📚 Data

Overview

To leverage the best of both distilled data (from ChatGPT) and real-world data (from Doctors), we firstly fine-tune HuatuoGPT using four types of data:

Download

• HuatuoGPT-sft-data-v1: The data used in the Supervised Fine-Tuning (SFT) stage of HuatuoGPT.

👨‍⚕️ Model

Model Access

Note that due to that HuatuoGPT-13B-delta is a LLaMA based model, we only release the delta of weights. You can download LLaMA-13B weights and use apply_delta.py to convert:

python apply_delta.py \
--base-model-path $LLaMA_Base_Path \
--target-model-path $Save_Path \
--delta-path $Delta_Path

Deploy

Firstly, you should install all required packages

pip install -r requirements.txt

Please make sure you have download our model weights and run

python -m huatuo_cli_demo_stream.py --model-name $model_dir

🚀 Demo

Try our model in https://www.huatuogpt.cn/. Note that it is still in progressing.

🧐 Evaluations

Evaluation by GPT-4 and Doctors

We invite GPT-4 and doctors to compare responses from HuatuoGPT(13B version) and other LLMs. Evaluation data is available in the eval/ folder. Results are as below:

• Single turn evaluation

• Multi turn evaluation

Benchmark Evaluation

⚒️ Training

Prepare the Data

You can download the SFT data from HuatuoGPT-sft-data-v1 or buld your SFT data as the same schema.

Training

You can train the model by:

accelerate launch \
	--config_file scripts/sft.yaml \
	--num_processes 8 \
	--num_machines 1 \
	--machine_rank 0 \
	--deepspeed_multinode_launcher standard scripts/finetune.py \
    --experiment_name HuatuoGPT \
	--model_path /path/to/your/model \
    --gradient_accumulation_steps 8 \
    --max_ckpts 3 \
    --max_seq_len 2048 \
	--data_dir /path/to/your/data \
	--output_dir ./ckpts \
	--log_dir ./train_logs \
	--n_epochs 3 \
	--train_bsz_per_gpu 2 \
	--eval_bsz_per_gpu 2 \
	--learning_rate 5e-5 \
	--eval_step -1 \
	--save_step -1 \
    --gradient_checkpointing

🤖 Limitations

Our goal with HuatuoGPT is to address the need for quick medical consultations, rather than replace doctors or provide full medical support to patients. However, our model does have several limitations that must be taken into consideration:

• Misunderstandings: As with all language models, there is a risk of misunderstandings or misinterpretations, especially when dealing with medical jargon or complex conditions. In this scenario, our models may give wrong answers.
• Hallucinations: Large language models can sometimes generate responses that do not make sense or are completely unrelated to the given input. These "hallucinations" can be especially problematic when users are not familiar with the concepts being discussed, as they may not be able to easily recognize the errors in the model's output. These "hallucinations" can be a challenge to detect and avoid.
• Bias: LLMs are trained on large datasets, which can inadvertently introduce bias into the model's responses. Additionally, care should be taken to ensure that the model is not used to perpetuate biases in medical treatment.

Acknowledgement

We are aware that our works are inspired by the following works, including but not limited to

• IDEA-CCNL/Ziya-LLaMA-13B-Pretrain-v1: https://huggingface.co/IDEA-CCNL/Ziya-LLaMA-13B-Pretrain-v1
• Baichuan-7B: https://huggingface.co/baichuan-inc/baichuan-7B
• LLaMA: https://arxiv.org/abs/2302.13971
• Self-instruct: https://github.com/yizhongw/self-instruct

Without these, nothing could happen in this repository.

Citation

@article{huatuogpt-2023,
  title={HuatuoGPT, Towards Taming Language Models To Be a Doctor},
  author={Hongbo Zhang and Junying Chen and Feng Jiang and Fei Yu and Zhihong Chen and Jianquan Li and Guiming Chen and Xiangbo Wu and Zhiyi Zhang and Qingying Xiao and Xiang Wan and Benyou Wang and Haizhou Li},
  journal={arXiv preprint arXiv:2305.15075},
  year={2023}
}

We are from the School of Data Science, the Chinese University of Hong Kong, Shenzhen (CUHKSZ) and the Shenzhen Rsearch Institute of Big Data (SRIBD).

Star History