Contrastive Decoding
Contrastive Decoding: Open-ended Text Generation as Optimization
Arxiv Link: https://arxiv.org/abs/2210.15097
Setup
pip install -e transformers Run contrastive decoding on a specified prompt:
cd text-generation;
python run_generation.py --model_name_or_path gpt2-xl --model_type gpt2 --length 256 --prompt "<|endoftext|> A version of Sonic the Hedgehog was developed by Ancient and released in 1991" --student_name_or_path gpt2 --st_coef 1.0 --student_temperature 0.5 --outfile outputs/temp_out.json --ignore_prefix noRun contrastive decoding on dataset (see submit_decoding.py for detail):
python run_generation.py --model_name_or_path gpt2-xl --model_type gpt2 --length 256 --prompt_file wikitext --student_name_or_path gpt2 --st_coef 1.0 --student_temperature 0.5 --outfile outputs/temp_out.json --ignore_prefix noThis code is used for producing all results in the paper. We will release a cleaner version of the code soon;