DecryptPrompt

如果LLM的突然到来让你感到沮丧，不妨读下主目录的Choose Your Weapon Survival Strategies for Depressed AI Academics 持续更新以下内容，Star to keep updated~

目录顺序如下

1. 国内外，垂直领域大模型
2. Agent和指令微调等训练框架
3. 开源指令，预训练，rlhf，对话，agent训练数据梳理
4. AIGC相关应用
5. prompt写作指南和5星博客等资源梳理
6. Prompt和LLM论文细分方向梳理

My blogs & ChatGPT应用

• 解密Prompt系列1. Tunning-Free Prompt：GPT2 & GPT3 & LAMA & AutoPrompt
• 解密Prompt系列2. 冻结Prompt微调LM： T5 & PET & LM-BFF
• 解密Prompt系列3. 冻结LM微调Prompt: Prefix-tuning & Prompt-tuning & P-tuning
• 解密Prompt系列4. 升级Instruction Tuning：Flan/T0/InstructGPT/TKInstruct
• 解密prompt系列5. APE+SELF=自动化指令集构建代码实现
• 解密Prompt系列6. lora指令微调扣细节-请冷静,1个小时真不够~
• 解密Prompt系列7. 偏好对齐RLHF-OpenAI·DeepMind·Anthropic对比分析
• 解密Prompt系列8. 无需训练让LLM支持超长输入:知识库 & Unlimiformer & PCW & NBCE
• 解密Prompt系列9. 模型复杂推理-思维链基础和进阶玩法
• 解密Prompt系列10. 思维链COT原理探究
• 解密Prompt系列11. 小模型也能COT，先天不足后天补
• 解密Prompt系列12. LLM Agent零微调范式 ReAct & Self Ask
• 解密Prompt系列13. LLM Agent指令微调方案: Toolformer & Gorilla
• 解密Prompt系列14. LLM Agent之搜索应用设计：WebGPT & WebGLM & WebCPM
• 解密Prompt系列15. LLM Agent之数据库应用设计：DIN & C3 & SQL-Palm & BIRD

LLMS

模型评测

大模型评估尚未出现北极星指标，整体上全面性有余，泛化性不足，类perplexity的指标还未出现，期待ing ~

国外模型

国内模型

垂直领域模型&进展

Tool and Library

指令微调，预训练，rlhf框架

LLM Agent工具

Training Data

AIGC

• NexusGPT: : AutoGPT可以出来工作了，第一个全AI Freelance平台
• cognosys: 全网最火的web端AutoGPT，不过咋说呢试用了下感觉下巴要笑掉了，不剧透去试试你就知道
• godmode：可以进行人为每一步交互的的AutoGPT
• agentgpt: 基础版AutoGPT ⭐
• do Anything: AutoGPT Like的to Do List生成器
• ChatMind: chatgpt生成思维导图，模板很丰富，泛化性也不错，已经被XMind收购了~ ⭐
• New Bing：需要连外网否则会重定向到bing中国，需要申请waitlist ⭐
• Perplexity.ai: 同样需要科学上网，感觉比Bing做的更好的接入ChatGPT的神奇搜索引擎，在Bing之外还加入了相关推荐和追问 ⭐
• BingGPT: NewBing开源桌面客户端，可以将聊天记录导出
• DocsGPT: 把ChatGPT开放域问答转化成封闭域问答的通用方案，试用垂类领域问答场景,可以试用定制的ChatBot ⭐
• langchain-ChatGLM: 基于ChatGLM的本地知识问答，和上面的DocsGPT相似，不过可以本地部署⭐
• ChatPDF: 国内的ChatPDF, 上传pdf后，会给出文章的Top5可能问题，然后对话式从文档中进行问答和检索，10s读3万字
• ChatDoc:ChatPDF升级版，增加了表格类解析，和完善的索引引用加跳转加对应文章内容高亮，哈哈我准备自己整一个
• ChatPaper: 根据输入关键词，自动在arxiv上下载最新的论文，并对论文进行摘要总结，可以在huggingface上试用！
• OpenRead: 面向论文写作，阅读场景，可以帮助生成文献综述，以及提供和NotionAI相似的智能Markdown用于写作
• researchgpt: 和ChatPDF类似，支持arivx论文下载，加载后对话式获取论文重点
• BriefGPT: 日更Arxiv论文，并对论文进行摘要，关键词抽取，帮助研究者了解最新动态, UI不错哟
• ChatGPT-academic: 又是一个基于gradio实现的paper润色，摘要等功能打包的实现
• feishu-chatgpt: 飞书chatgpt，和365copilot相似也是多组件集成, 有点全！
• AI Topiah: 聆心智能AI角色聊天，和路飞唠了两句，多少有点中二之魂在燃烧
• chatbase: 情感角色聊天，还没尝试
• Vana: virtual DNA, 通过聊天创建虚拟自己！概念很炫
• WriteSonic：AI写作，支持对话和定向创作如广告文案，商品描述, 支持Web检索是亮点，支持中文
• copy.ai: WriteSonic竞品，亮点是像论文引用一样每句话都有对应网站链接，可以一键复制到右边的创作Markdown，超级好用！ ⭐
• NotionAI：智能Markdown，适用真相！在创作中用command调用AI辅助润色，扩写，检索内容，给创意idea
• Jasper: 同上，全是竞品哈哈
• copy.down: 中文的营销文案生成，只能定向创作，支持关键词到文案的生成
• ChatExcel: 指令控制excel计算，对熟悉excel的有些鸡肋，对不熟悉的有点用
• ChatPPT: 使用ChatGPT进行PPT制作
• BibiGPT: Bilibli视频内容一键总结，多模态文档
• Copilot: 要付费哟
• Fauxpilot: copilot本地开源替代
• CodeGex: 国内替代品，还没试过
• Codeium: Copilot替代品，有免费版本支持各种plugin
• sql translate: text2sql，利用 OpenAI 的 API 实现的一个很简单的工具，sql到文字，文字到sql
• ai2sql: text2sql老牌公司，相比sqltranslate功能更全面，支持SQL 语法检查、格式化和生成公式
• chat2query: text2sql 相比以上两位支持更自然的文本指令，以及更复杂的数据分析类的sql生成 ⭐
• OuterBase: text2sql 设计风格很吸睛！电子表格结合mysql和dashboard，更适合数据分析宝宝
• Wolverine: 代码自我debug的python脚本
• dreamstudio.ai: 开创者，Stable Difussion， 有试用quota
• midjourney: 开创者，艺术风格为主
• Dall.E: 三巨头这就凑齐了
• ControlNet: 为绘画创作加持可控性
• GFPGAN: 照片修复
• Visual ChatGPT: 微软发布图像ChatGPT，对话方式进行图像生成编辑，问答 ⭐
• gemo.ai: 多模态聊天机器人，包括文本，图像，视频生成
• storybird: 根据提示词生成故事绘本，还可以售卖

Resources

教程类

• OpenAI Cookbook: 提供OpenAI模型使用示例 ⭐
• OpenAI 接口被墙解决办法: 使用腾讯云搭建代理，亲测非常好用且手残党也可以轻松上手
• PromptPerfect:用魔法打败魔法，输入原始提示词，模型进行定向优化，试用后我有点沉默了，可以定向支持不同使用prompt的模型如Difussion，ChatGPT， Dalle等
• ClickPrompt: 为各种prompt加持的工具生成指令包括Difussion，chatgptdeng, 需要OpenAI Key
• ChatGPT ShortCut：提供各式场景下的Prompt范例，范例很全，使用后可以点赞！ ⭐
• Full ChatGPT Prompts + Resources: 各种尝尽的prompt范例，和以上场景有所不同
• learning Prompt: prompt engineering超全教程，和落地应用收藏，包括很多LLM调用Agent的高级场景 ⭐
• The art of asking chatgpt for high quality answers: 如何写Prompt指令出书了，链接是中文翻译的版本，比较偏基础使用
• Prompt-Engineer-Guide: 同learnig prompt类的集成教程，互相引用可还行？！分类索引做的更好些 ⭐
• OpenAI 应用汇总指南: 纯应用类的汇总指南
• AI 导航: 包括但不限于ChatGPT的应用汇总网站，更新很快，发现了一些新大陆
• AI Alignment Forum: RLHF等对齐相关最新论文和观点的讨论论坛
• Langchain: Chat with your data:吴恩达LLM实践课程
• 构筑大语言模型应用：应用开发与架构设计: 一本关于 LLM 在真实世界应用的开源电子书
• Large Language Models: Application through Production: 大模型应用Edx出品的课程

书籍博客类

• OpenAI ChatGPT Intro
• OpenAI InstructGPT intro
• AllenAI ChatGPT能力解读：How does GPT Obtain its Ability? Tracing Emergent Abilities of Language Models to their Sources ⭐
• Huggingface ChatGPT能力解读：The techniques behind ChatGPT: RLHF, IFT, CoT, Red teaming, and more
• Stephen Wolfram ChatGPT能力解读: What Is ChatGPT Doing and Why Does It Work?
• Chatgpt相关解读汇总
• 麻省理工科技采访OpenAI工程师
• AGI历史与现状
• 张俊林 通向AGI之路：大型语言模型（LLM）技术精要
• 知乎回答 OpenAI 发布 GPT-4，有哪些技术上的优化或突破?
• 追赶ChatGPT的难点与平替
• 压缩即泛化，泛化即智能 ⭐
• 陆奇最新演讲实录：我的大模型世界观｜第十四期
• LLM Powered Autonomous Agents ⭐
• All You Need to Know to Build Your First LLM App ⭐
• GPT-4 Architecture, Infrastructure, Training Dataset, Costs, Vision, MoE
• 为什么伟大不能被计划: OpenAI研究员出书
• 拾象投研机构对LLM的调研报告（文中有两次PPT的申请链接）: 对大模型应用给出了很全面的总结梳理
• 启明创投State of Generative AI 2023: 最近发现应用落地才是LLM真正产生价值的核心，开始更多关注一些投研的分析报告
• How to Use AI to Do Stuff: An Opinionated Guide
• Llama 2: an incredible open LLM
• Wolfram语言之父新书：这就是ChatGPT
• 谷歌出品：对大模型领悟能力的一些探索很有意思 Do Machine Learning Models Memorize or Generalize? ⭐
• OpenAI首席科学家最新讲座解读LM无监督预训练学了啥 An observation on Generalization ⭐
• The Complete Beginners Guide To Autonomous Agents: Octane AI创始人 Matt Schlicht发表的关于人工智能代理的一些思考
• An Initial Exploration of Theoretical Support for Language Model Data Engineering. Part 1: Pretraining: 符尧大佬系列新作，通过了解大模型背后的数据工程来了解模型本质，第一篇预训练数据

Papers

paper List

• https://github.com/dongguanting/In-Context-Learning_PaperList
• https://github.com/thunlp/PromptPapers
• https://github.com/Timothyxxx/Chain-of-ThoughtsPapers
• https://github.com/thunlp/ToolLearningPapers

综述

• A Survey of Large Language Models
• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing ⭐
• Paradigm Shift in Natural Language Processing
• Pre-Trained Models: Past, Present and Future
• What Language Model Architecture and Pretraining objects work best for zero shot generalization ⭐
• Towards Reasoning in Large Language Models: A Survey
• Reasoning with Language Model Prompting: A Survey ⭐
• An Overview on Language Models: Recent Developments and Outlook ⭐
• A Survey of Large Language Models[6.29更新版]
• Unifying Large Language Models and Knowledge Graphs: A Roadmap
• Augmented Language Models: a Survey ⭐
• Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey
• Challenges and Applications of Large Language Models

大模型能力探究

• In Context Learning
  • LARGER LANGUAGE MODELS DO IN-CONTEXT LEARNING DIFFERENTLY
  • How does in-context learning work? A framework for understanding the differences from traditional supervised learning
  • Why can GPT learn in-context? Language Model Secretly Perform Gradient Descent as Meta-Optimizers ⭐
  • Rethinking the Role of Demonstrations What Makes incontext learning work? ⭐
  • Trained Transformers Learn Linear Models In-Context
• 涌现能力
  • Sparks of Artificial General Intelligence: Early experiments with GPT-4
  • Emerging Ability of Large Language Models ⭐
• 能力评估
  • IS CHATGPT A GENERAL-PURPOSE NATURAL LANGUAGE PROCESSING TASK SOLVER?
  • Can Large Language Models Infer Causation from Correlation?
  • Holistic Evaluation of Language Model
  • Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond
  • Theory of Mind May Have Spontaneously Emerged in Large Language Models
  • Beyond The Imitation Game: Quantifying And Extrapolating The Capabilities Of Language Models
  • Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations
  • Demystifying GPT Self-Repair for Code Generation
  • Evidence of Meaning in Language Models Trained on Programs
  • Can Explanations Be Useful for Calibrating Black Box Models
  • On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective
  • Language acquisition: do children and language models follow similar learning stages?

Prompt Tunning范式

• Tunning Free Prompt
  • GPT2: Language Models are Unsupervised Multitask Learners
  • GPT3: Language Models are Few-Shot Learners ⭐
  • LAMA: Language Models as Knowledge Bases?
  • AutoPrompt: Eliciting Knowledge from Language Models
• Fix-Prompt LM Tunning
  • T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
  • PET-TC(a): Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference ⭐
  • PET-TC(b): PETSGLUE It’s Not Just Size That Matters Small Language Models are also few-shot learners
  • GenPET: Few-Shot Text Generation with Natural Language Instructions
  • LM-BFF: Making Pre-trained Language Models Better Few-shot Learners ⭐
  • ADEPT: Improving and Simplifying Pattern Exploiting Training
• Fix-LM Prompt Tunning
  • Prefix-tuning: Optimizing continuous prompts for generation
  • Prompt-tunning: The power of scale for parameter-efficient prompt tuning ⭐
  • P-tunning: GPT Understands Too ⭐
  • WARP: Word-level Adversarial ReProgramming
• LM + Prompt Tunning
  • P-tunning v2: Prompt Tuning Can Be Comparable to Fine-tunning Universally Across Scales and Tasks
  • PTR: Prompt Tuning with Rules for Text Classification
  • PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domains
• Fix-LM Adapter Tunning
  • LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS ⭐
  • LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning
  • Parameter-Efficient Transfer Learning for NLP
  • INTRINSIC DIMENSIONALITY EXPLAINS THE EFFECTIVENESS OF LANGUAGE MODEL FINE-TUNING

主流LLMS

• GLM-130B: AN OPEN BILINGUAL PRE-TRAINED MODEL
• LLaMA: Open and Efficient Foundation Language Models
• PaLM: Scaling Language Modeling with Pathways
• PaLM 2 Technical Report
• GPT-4 Technical Report
• Backpack Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models

指令微调&对齐 (instruction_tunning)

• 经典方案
  • Flan: FINETUNED LANGUAGE MODELS ARE ZERO-SHOT LEARNERS ⭐
  • Flan-T5: Scaling Instruction-Finetuned Language Models
  • ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning
  • Instruct-GPT: Training language models to follow instructions with human feedback ⭐
  • T0: MULTITASK PROMPTED TRAINING ENABLES ZERO-SHOT TASK GENERALIZATION
  • Natural Instructions: Cross-Task Generalization via Natural Language Crowdsourcing Instructions
  • Tk-INSTRUCT: SUPER-NATURALINSTRUCTIONS: Generalization via Declarative Instructions on 1600+ NLP Tasks
  • ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-shot Generalization
  • Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor
  • INSTRUCTEVAL Towards Holistic Evaluation of Instrucion-Tuned Large Language Models
• 更少，质量更高、更多样的指令数据带来质变
  • LIMA: Less Is More for Alignment ⭐
  • Maybe Only 0.5% Data is Needed: A Preliminary Exploration of Low Training Data Instruction Tuning
  • Textbooks Are All You Need ⭐
  • AlpaGasus: Training A Better Alpaca with Fewer Data
  • InstructionGPT-4: A 200-Instruction Paradigm for Fine-Tuning MiniGPT-4
  • Instruction Mining: High-Quality Instruction Data Selection for Large Language Models
  • Visual Instruction Tuning with Polite Flamingo
• 新对齐/微调方案
  • WizardLM: Empowering Large Language Models to Follow Complex Instructions
  • Becoming self-instruct: introducing early stopping criteria for minimal instruct tuning
  • Self-Alignment with Instruction Backtranslation ⭐
  • Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models
  • Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks
  • PROMPT2MODEL: Generating Deployable Models from Natural Language Instructions
  • OpinionGPT: Modelling Explicit Biases in Instruction-Tuned LLMs
• 微调经验/实验报告
  • BELLE: Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Cases
  • Baize: Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data
  • A Comparative Study between Full-Parameter and LoRA-based Fine-Tuning on Chinese Instruction Data for Large LM
  • Exploring ChatGPT’s Ability to Rank Content: A Preliminary Study on Consistency with Human Preferences
  • Towards Better Instruction Following Language Models for Chinese: Investigating the Impact of Training Data and Evaluation

对话模型

• LaMDA: Language Models for Dialog Applications
• Sparrow: Improving alignment of dialogue agents via targeted human judgements ⭐
• BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AI
• Enhancing Chat Language Models by Scaling High-quality Instructional Conversations
• DiagGPT: An LLM-based Chatbot with Automatic Topic Management for Task-Oriented Dialogue

思维链 (prompt_chain_of_thought)

• 基础&进阶用法
  • [zero-shot-COT] Large Language Models are Zero-Shot Reasoners ⭐
  • [few-shot COT] Chain of Thought Prompting Elicits Reasoning in Large Language Models ⭐
  • SELF-CONSISTENCY IMPROVES CHAIN OF THOUGHT REASONING IN LANGUAGE MODELS
  • LEAST-TO-MOST PROMPTING ENABLES COMPLEX REASONING IN LARGE LANGUAGE MODELS ⭐
  • Tree of Thoughts: Deliberate Problem Solving with Large Language Models
  • Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models
  • Decomposed Prompting A MODULAR APPROACH FOR Solving Complex Tasks
  • Successive Prompting for Decomposing Complex Questions
  • Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Framework
  • Beyond Chain-of-Thought, Effective Graph-of-Thought Reasoning in Large Language Models
  • Tree-of-Mixed-Thought: Combining Fast and Slow Thinking for Multi-hop Visual Reasoning
  • LAMBADA: Backward Chaining for Automated Reasoning in Natural Language
• 分领域COT [Math, Code, Tabular, QA]
  • Solving Quantitative Reasoning Problems with Language Models
  • SHOW YOUR WORK: SCRATCHPADS FOR INTERMEDIATE COMPUTATION WITH LANGUAGE MODELS
  • Solving math word problems with processand outcome-based feedback
  • CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
  • T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Large Language Model Signals for Science Question Answering
  • LEARNING PERFORMANCE-IMPROVING CODE EDITS
  • Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoning
  • Tab-CoT: Zero-shot Tabular Chain of Thought
• 原理分析
  • Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters ⭐
  • TEXT AND PATTERNS: FOR EFFECTIVE CHAIN OF THOUGHT IT TAKES TWO TO TANGO
  • Towards Revealing the Mystery behind Chain of Thought: a Theoretical Perspective
  • Large Language Models Can Be Easily Distracted by Irrelevant Context
• 小模型COT蒸馏
  • Specializing Smaller Language Models towards Multi-Step Reasoning ⭐
  • Teaching Small Language Models to Reason
  • Large Language Models are Reasoning Teachers
  • Distilling Reasoning Capabilities into Smaller Language Models
  • The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning
• COT样本自动构建/选择
  • STaR: Self-Taught Reasoner Bootstrapping ReasoningWith Reasoning
  • AutoCOT：AUTOMATIC CHAIN OF THOUGHT PROMPTING IN LARGE LANGUAGE MODELS
  • Large Language Models Can Self-Improve
  • Active Prompting with Chain-of-Thought for Large Language Models
  • COMPLEXITY-BASED PROMPTING FOR MULTI-STEP REASONING
• others
  • OlaGPT Empowering LLMs With Human-like Problem-Solving abilities
  • Challenging BIG-Bench tasks and whether chain-of-thought can solve them
  • Large Language Models are Better Reasoners with Self-Verification
  • ThoughtSource A central hub for large language model reasoning data
  • Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs

RLHF

• Deepmind
  • Teaching language models to support answers with verified quotes
  • sparrow, Improving alignment of dialogue agents via targetd human judgements ⭐
• openai
  • PPO: Proximal Policy Optimization Algorithms ⭐
  • Deep Reinforcement Learning for Human Preference
  • Fine-Tuning Language Models from Human Preferences
  • learning to summarize from human feedback
  • InstructGPT: Training language models to follow instructions with human feedback ⭐
  • Scaling Laws for Reward Model Over optimization ⭐
• Anthropic
  • A General Language Assistant as a Laboratory for Alignmen
  • Red Teaming Language Models to Reduce Harms Methods,Scaling Behaviors and Lessons Learned
  • Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback ⭐
  • Constitutional AI Harmlessness from AI Feedback ⭐
  • Pretraining Language Models with Human Preferences
  • The Capacity for Moral Self-Correction in Large Language Models
• AllenAI, RL4LM：IS REINFORCEMENT LEARNING (NOT) FOR NATURAL LANGUAGE PROCESSING BENCHMARKS
• 改良方案
  • RRHF: Rank Responses to Align Language Models with Human Feedback without tears
  • PRM：Let's verify step by step
  • Chain of Hindsight Aligns Language Models with Feedback
  • AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
  • Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
  • RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment

LLM Agent 让模型使用工具 (llm_agent)

• 基于prompt通用方案
  • ReAct: SYNERGIZING REASONING AND ACTING IN LANGUAGE MODELS ⭐
  • Self-ask: MEASURING AND NARROWING THE COMPOSITIONALITY GAP IN LANGUAGE MODELS ⭐
  • MRKL SystemsA modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning
  • PAL: Program-aided Language Models
  • ART: Automatic multi-step reasoning and tool-use for large language models
  • ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models ⭐
  • Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
  • Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models ⭐
  • Faithful Chain-of-Thought Reasoning
  • Reflexion: Language Agents with Verbal Reinforcement Learning ⭐
  • Search-in-the-Chain: Towards Accurate, Credible and Traceable Large Language Models for Knowledge-intensive Tasks
  • Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Framework
  • RestGPT: Connecting Large Language Models with Real-World RESTful APIs
• 基于微调通用方案
  • TALM: Tool Augmented Language Models
  • Toolformer: Language Models Can Teach Themselves to Use Tools ⭐
  • Tool Learning with Foundation Models
  • Tool Maker：Large Language Models as Tool Maker
  • TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs
• 检索增强方案
  • WebGPT：Browser-assisted question-answering with human feedback
  • WebGLM: Towards An Efficient Web-Enhanced Question Answering System with Human Preferences
  • WebCPM: Interactive Web Search for Chinese Long-form Question Answering ⭐
  • REPLUG: Retrieval-Augmented Black-Box Language Models
  • Query Rewriting for Retrieval-Augmented Large Language Models
  • RETA-LLM: A Retrieval-Augmented Large Language Model Toolkit
  • Atlas: Few-shot Learning with Retrieval Augmented Language Models
  • RRAML: Reinforced Retrieval Augmented Machine Learning
• 调用模型方案
  • HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace
  • Gorilla：Large Language Model Connected with Massive APIs ⭐
  • OpenAGI: When LLM Meets Domain Experts
• 垂直领域
  • WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
  • ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings
  • ChemCrow Augmenting large language models with chemistry tools
  • Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflow
  • GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Information
  • PointLLM: Empowering Large Language Models to Understand Point Clouds
• 评估
  • Evaluating Verifiability in Generative Search Engines
  • Mind2Web: Towards a Generalist Agent for the Web
  • Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions
  • API-Bank: A Benchmark for Tool-Augmented LLMs
  • ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs
  • FACTSCORE: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
• 智能体
  • Generative Agents: Interactive Simulacra of Human Behavior
  • CAMEL: Communicative Agents for "Mind" Exploration of Large Scale Language Model Society
  • LLM+P: Empowering Large Language Models with Optimal Planning Proficiency
• 其他
  • Inference with Reference: Lossless Acceleration of Large Language Models
  • RecallM: An Architecture for Temporal Context Understanding and Question Answering

指令数据生成 (instruction_data_gen)

• APE: LARGE LANGUAGE MODELS ARE HUMAN-LEVEL PROMPT ENGINEERS ⭐
• SELF-INSTRUCT: Aligning Language Model with Self Generated Instructions ⭐
• iPrompt: Explaining Data Patterns in Natural Language via Interpretable Autoprompting
• Flipped Learning: Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learners
• Fairness-guided Few-shot Prompting for Large Language Models
• Instruction induction: From few examples to natural language task descriptions.
• Baize An Open-Source Chat Model with Parameter-Efficient Tuning on self-Chat Data
• SELF-QA Unsupervised Knowledge Guided alignment.
• GPT Self-Supervision for a Better Data Annotator
• The Flan Collection Designing Data and Methods
• Self-Consuming Generative Models Go MAD
• InstructEval: Systematic Evaluation of Instruction Selection Methods
• Overwriting Pretrained Bias with Finetuning Data

预训练数据(pretrain_data)

• DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• CCNet: Extracting High Quality Monolingual Datasets fromWeb Crawl Data
• WanJuan: A Comprehensive Multimodal Dataset for Advancing English and Chinese Large Models
• CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Model

领域模型 (domain_llms)

• MedGPT: Medical Concept Prediction from Clinical Narratives
• BioGPT：Generative Pre-trained Transformer for Biomedical Text Generation and Mining
• Galactia：A Large Language Model for Science
• PubMed GPT: A Domain-specific large language model for biomedical text ⭐
• BloombergGPT： A Large Language Model for Finance
• ChatDoctor：Medical Chat Model Fine-tuned on LLaMA Model using Medical Domain Knowledge
• Med-PaLM：Large Language Models Encode Clinical Knowledge[V1,V2] ⭐
• Augmented Large Language Models with Parametric Knowledge Guiding
• XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters
• ChatLaw Open-Source Legal Large Language Model ⭐
• MediaGPT : A Large Language Model For Chinese Media
• SMILE: Single-turn to Multi-turn Inclusive Language Expansion via ChatGPT for Mental Health Support
• KITLM: Domain-Specific Knowledge InTegration into Language Models for Question Answering
• FinVis-GPT: A Multimodal Large Language Model for Financial Chart Analysis
• EcomGPT: Instruction-tuning Large Language Models with Chain-of-Task Tasks for E-commerce

LLM超长文本处理 (long_input)

• Parallel Context Windows for Large Language Models
• Structured Prompting: Scaling In-Context Learning to 1,000 Examples
• 苏剑林, NBCE：使用朴素贝叶斯扩展LLM的Context处理长度 ⭐
• Vcc: Scaling Transformers to 128K Tokens or More by Prioritizing Important Tokens
• Unlimiformer: Long-Range Transformers with Unlimited Length Input
• Scaling Transformer to 1M tokens and beyond with RMT
• RECURRENTGPT: Interactive Generation of (Arbitrarily) Long Text
• TRAIN SHORT, TEST LONG: ATTENTION WITH LINEAR BIASES ENABLES INPUT LENGTH EXTRAPOLATION ⭐
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• Extending Context Window of Large Language Models via Positional Interpolation
• LongNet: Scaling Transformers to 1,000,000,000 Tokens
• https://kaiokendev.github.io/til#extending-context-to-8k
• 苏剑林,Transformer升级之路：10、RoPE是一种β进制编码 ⭐
• 苏剑林,Transformer升级之路：11、将β进制位置进行到底
• 苏剑林,Transformer升级之路：12、无限外推的ReRoPE？
• Focused Transformer: Contrastive Training for Context Scaling
• Lost in the Middle: How Language Models Use Long Contexts ⭐

NL2SQL

• 大模型方案
  • DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction ⭐
  • C3: Zero-shot Text-to-SQL with ChatGPT ⭐
  • SQL-PALM: IMPROVED LARGE LANGUAGE MODEL ADAPTATION FOR TEXT-TO-SQL
  • BIRD Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQL ⭐
  • A Case-Based Reasoning Framework for Adaptive Prompting in Cross-Domain Text-to-SQL
  • ChatDB: AUGMENTING LLMS WITH DATABASES AS THEIR SYMBOLIC MEMORY
  • A comprehensive evaluation of ChatGPT’s zero-shot Text-to-SQL capability
  • Few-shot Text-to-SQL Translation using Structure and Content Prompt Learning
• Domain Knowledge Intensive
  • Towards Knowledge-Intensive Text-to-SQL Semantic Parsing with Formulaic Knowledge
  • Bridging the Generalization Gap in Text-to-SQL Parsing with Schema Expansion
  • Towards Robustness of Text-to-SQL Models against Synonym Substitution
  • FinQA: A Dataset of Numerical Reasoning over Financial Data
• others
  • RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL
  • MIGA: A Unified Multi-task Generation Framework for Conversational Text-to-SQL

降低模型幻觉 (reliability)

• Survey of Hallucination in Natural Language Generation
• Trusting Your Evidence: Hallucinate Less with Context-aware Decoding ⭐
• SELF-REFINE:ITERATIVE REFINEMENT WITH SELF-FEEDBACK ⭐
• PROMPTING GPT-3 TO BE RELIABLE
• Enhancing Self-Consistency and Performance of Pre-Trained Language Models through Natural Language Inference
• On the Advance of Making Language Models Better Reasoners
• Progressive-Hint Prompting Improves Reasoning in Large Language Models
• ASK ME ANYTHING: A SIMPLE STRATEGY FOR PROMPTING LANGUAGE MODELS ⭐
• Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
• Reflexion: an autonomous agent with dynamic memory and self-reflection
• Self-consistency for open-ended generations
• Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback
• Factuality Enhanced Language Models for Open-Ended Text Generation
• Adaptive Chameleon or Stubborn Sloth: Unraveling the Behavior of Large Language Models in Knowledge Clashes
• Rethinking with Retrieval: Faithful Large Language Model Inference
• KoLA: Carefully Benchmarking World Knowledge of Large Language Models
• RefGPT: Reference → Truthful & Customized Dialogues Generation by GPTs and for GPTs

推理速度优化(fast_inference)

• Fast Transformer Decoding: One Write-Head is All You Need
• Fast Inference from Transformers via Speculative Decoding
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
• Skeleton-of-Thought: Large Language Models Can Do Parallel Decoding
• SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference

模型知识编辑黑科技(model_edit)

• ROME：Locating and Editing Factual Associations in GPT
• Transformer Feed-Forward Layers Are Key-Value Memories
• MEMIT: Mass-Editing Memory in a Transformer
• MEND：Fast Model Editing at Scale
• Editing Large Language Models: Problems, Methods, and Opportunities

Other Prompt Engineer(prompt_engineer)

• Calibrate Before Use: Improving Few-Shot Performance of Language Models
• In-Context Instruction Learning
• LEARNING PERFORMANCE-IMPROVING CODE EDITS
• Boosting Theory-of-Mind Performance in Large Language Models via Prompting
• Generated Knowledge Prompting for Commonsense Reasoning
• RECITATION-AUGMENTED LANGUAGE MODELS
• kNN PROMPTING: BEYOND-CONTEXT LEARNING WITH CALIBRATION-FREE NEAREST NEIGHBOR INFERENCE
• EmotionPrompt: Leveraging Psychology for Large Language Models Enhancement via Emotional Stimulus
• Causality-aware Concept Extraction based on Knowledge-guided Prompting

Multimodal

• InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
• Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models
• PaLM-E: An Embodied Multimodal Language Model
• LLava Visual Instruction Tuning
• MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
• TabLLM: Few-shot Classification of Tabular Data with Large Language Models
• BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions
• mPLUG-Owl : Modularization Empowers Large Language Models with Multimodality
• LVLM eHub: A Comprehensive Evaluation Benchmark for Large VisionLanguage Models

Others

• Learnware: Small Models Do Big
• The economic potential of generative AI
• A PhD Student’s Perspective on Research in NLP in the Era of Very Large Language Models