Taiwan-LLM: Language Models for Taiwanese Culture

✍️ Online Demo • 🤗 Model Collection • 🐦 Twitter • 📃 Paper • 👨️ Yen-Ting Lin

🎉🎉🎉Taiwan-LLM v2: We are excited to release Taiwan-LLM v2, including the 7B and 13B models, now available on twllm.com and on our HuggingFace collection🚀

Overview

Taiwan-LLM is a full parameter fine-tuned model based on Meta/LLaMa-2 for Traditional Mandarin applications.

Taiwan-LLM v2.0 13B pretrained on over 30 billion tokens and instruction-tuned on over 1 million instruction-following conversations both in traditional mandarin.

Taiwan-LLM v2.0 7B pretrained on over 30 billion tokens and instruction-tuned on over 1 million instruction-following conversations both in traditional mandarin.

Taiwan-LLM v1.0 13B pretrained on over 5 billion tokens and instruction-tuned on over 490k conversations both in traditional mandarin.

Demo

A live demonstration of the model can be accessed at TWLLM.com.

Key Features

1. Traditional Mandarin Support: The model is fine-tuned to understand and generate text in Traditional Mandarin, making it suitable for Taiwanese culture and related applications.

2. Instruction-Tuned: Further fine-tuned on conversational data to offer context-aware and instruction-following responses.

3. Performance on TC-Eval: Taiwan-LLM v2 13B shows a slight edge over ChatGPT-3 and achieves around 92% of ChatGPT-4's performance in zh-tw benchmarks.

Work in progress

• Improved pretraining: A refined pretraining process (e.g. more data from Taiwan, training strategies) is under development, aiming to enhance model performance for better Taiwanese culture.
• Extended Length: Our demo on twllm.com is served in 8k context length.
• 3B / 1B model: In the pipeline are scaled-down versions of the Taiwan-LLM with 3 billion and 1 billion parameters, aimed at providing more accessible options for researchers and developers with limited computational resources.
• Vision: We are exploring multi-modal integration into Taiwan-LLM, enabling the model to understand content that combines text and image.
• Mixture of Experts: Planning to incorporate a Mixture of Experts architecture to improve the model's efficiency and capacity.

Taiwanese Culture Examples

Model

We provide a number of model checkpoints that we trained. Please find them on Hugging Face here. Here are some quick links to the checkpoints that are finetuned from LLaMa 2:

Data

Architecture

Taiwan-LLaMa is based on LLaMa 2, leveraging transformer architecture, flash attention 2, and bfloat16.

It includes:

• Pretraining Phase: Pretrained on a vast corpus of over 5 billion tokens, extracted from common crawl in Traditional Mandarin.
• Fine-tuning Phase: Further instruction-tuned on over 490k multi-turn conversational data to enable more instruction-following and context-aware responses.

Evaluating "Taiwan LLM" on TC-Eval

How to deploy the model on my own machine?

We recommend hosting models with 🤗 Text Generation Inference. Please see their license for details on usage and limitations.

bash run_text_generation_inference.sh "yentinglin/Taiwan-LLaMa-v1.0" NUM_GPUS DIR_TO_SAVE_MODEL PORT MAX_INPUT_LEN MODEL_MAX_LEN

Taiwan LLm Prompt Template:

How to properly fomat my prompt?

from transformers import AutoTokenizer
# system message is optional
chat = [
  # {"role": "system", "content": "你講中文"},
  {"role": "user", "content": "Hello, how are you?"},
  {"role": "assistant", "content": "I'm doing great. How can I help you today?"},
  {"role": "user", "content": "I'd like to show off how chat templating works!"},
]
# This applies to all Taiwan-LLM series.
tokenizer = AutoTokenizer.from_pretrained("yentinglin/Taiwan-LLM-7B-v2.0.1-chat")
prompt_for_generation = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)
print(prompt_for_generation)

Version 2 is more robust to different system prompt or none.

你是人工智慧助理，以下是用戶和人工智能助理之間的對話。你要對用戶的問題提供有用、安全、詳細和禮貌的回答。USER: {user} ASSISTANT:

Taiwan LLm v1 Prompt Template:

A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: {user} ASSISTANT:

Setup development environment

conda create -n taiwan-llama python=3.10 -y 
conda activate taiwan-llama
pip install -r requirements.txt

常見問題

網頁Demo與程式碼執行結果有落差

可以參考這個 #19 (comment)。

能否商用化？

關於模型能不能商用，我建議您自行尋求法律意見。

模型作者 (Meta 與我) 都願意開放商用，但是『可以商用的模型”訓練在“有著作權法保護的資料上”，是否可以商用』需要您的判斷。

台灣沒有相關法案保護模型訓練在有著作權的資料上，但就我的理解，我們模型雖訓練在著作權資料上，但並沒有抄襲著作權人的意思表示，所以模型是可以商用的。

以上是我諮詢律師的結論，為求謹慎還請您尋求更專業的法律意見。

請問訓練此模型時使用的機器規格

Pretraining: 8 x A100 80G for 2 weeks
Instruction finetuning: 8 x H100 for 12 hrs

English Version

Web Demo and Code Execution Results Differ

Refer to this #19 (comment).

Can it be Commercialized?

For questions on commercial use, consult legal advice.

Both the model authors (Meta and I) are open to commercial use. However, whether a "commercially usable model" trained on "copyrighted data" can be used commercially is for you to decide.

To my understanding, although the model is trained on copyrighted data, it does not plagiarize. Therefore, it can be commercialized.

This is based on legal advice; for caution, consult a legal expert.

Machine Specifications for Training This Model

Pretraining: 8 x A100 80G for 2 weeks
Instruction finetuning: 8 x H100 for 12 hrs

Citations

If you use our code, data, or models in your research, please cite this repository. You can use the following BibTeX entry:

@misc{lin2023taiwan,
      title={Taiwan LLM: Bridging the Linguistic Divide with a Culturally Aligned Language Model}, 
      author={Yen-Ting Lin and Yun-Nung Chen},
      year={2023},
      eprint={2311.17487},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Collaborate With Us

If you are interested in contributing to the development of Traditional Mandarin language models, exploring new applications, or leveraging Taiwan-LLaMa for your specific needs, please don't hesitate to contact us. We welcome collaborations from academia, industry, and individual contributors.

License

The code in this project is licensed under the Apache 2.0 License - see the LICENSE file for details.

The models included in this project are licensed under the LLAMA 2 Community License. See the LLAMA2 License for full details.

OpenAI Data Acknowledgment

The data included in this project were generated using OpenAI's models and are subject to OpenAI's Terms of Use. Please review OpenAI's Terms of Use for details on usage and limitations.

Acknowledgements

We thank Meta LLaMA team and Vicuna team for their open-source efforts in democratizing large language models.