dair-ai/Prompt-Engineering-Guide: Guides, papers, lecture, and resources for prompt engineering

Created
Mar 7, 2023 6:43 AM
URL
https://github.com/dair-ai/Prompt-Engineering-Guide
Type
image

Prompt Engineering Guide

This guide contains a set of recent papers, learning guides, and tools related to prompt engineering. The repo is intended as a research and educational reference for practitioners and developers.

Announcements:

  • Prompt Engineering Lecture is live here! It Includes notebook and slides.
    • image

  • Join our Discord

  • Follow us on Twitter

Table of Contents

  • Lecture
  • Guides
  • Papers
  • Tools & Libraries
  • Datasets
  • Blog, Guides, Tutorials and Other Readings

Lecture

We have published a 1 hour lecture that provides a comprehensive overview of prompting techniques, applications, and tools.

  • Video Lecture
  • Notebook with code
  • Slides

Guides

The following are a set of guides on prompt engineering developed by us. Guides are work in progress.

  • Prompt Engineering - Introduction
  • Prompt Engineering - Basic Prompting
  • Prompt Engineering - Advanced Prompting
  • Prompt Engineering - Adversarial Prompting
  • Prompt Engineering - Miscellaneous Topics

Papers

The following are the latest papers (sorted by release date) on prompt engineering. We update this on a daily basis and new papers come in. We incorporate summaries of these papers to the guides above every week.

      • Surveys / Overviews:

    1. Augmented Language Models: a Survey (Feb 2023)
    2. A Survey for In-context Learning (Dec 2022)
    3. Towards Reasoning in Large Language Models: A Survey (Dec 2022)
    4. Emergent Abilities of Large Language Models (Jun 2022)
    5. A Taxonomy of Prompt Modifiers for Text-To-Image Generation (Apr 2022)
    6. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (Jul 2021)

      • Approaches/Techniques:

    1. Scalable Prompt Generation for Semi-supervised Learning with Language Models (Feb 2023)
    2. Bounding the Capabilities of Large Language Models in Open Text Generation with Prompt Constraints (Feb 2023)
    3. À-la-carte Prompt Tuning (APT): Combining Distinct Data Via Composable Prompting (Feb 2023)
    4. GraphPrompt: Unifying Pre-Training and Downstream Tasks for Graph Neural Networks (Feb 2023)
    5. The Capacity for Moral Self-Correction in Large Language Models (Feb 2023)
    6. SwitchPrompt: Learning Domain-Specific Gated Soft Prompts for Classification in Low-Resource Domains (Feb 2023)
    7. Evaluating the Robustness of Discrete Prompts (Feb 2023)
    8. Compositional Exemplars for In-context Learning (Feb 2023)
    9. Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery (Feb 2023)
    10. Multimodal Chain-of-Thought Reasoning in Language Models (Feb 2023)
    11. Large Language Models Can Be Easily Distracted by Irrelevant Context (Feb 2023)
    12. Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Models (Feb 2023)
    13. Progressive Prompts: Continual Learning for Language Models (Jan 2023)
    14. Batch Prompting: Efficient Inference with LLM APIs (Jan 2023)
    15. On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning (Dec 2022)
    16. Constitutional AI: Harmlessness from AI Feedback (Dec 2022)
    17. Successive Prompting for Decomposing Complex Questions (Dec 2022)
    18. Discovering Language Model Behaviors with Model-Written Evaluations (Dec 2022)
    19. Structured Prompting: Scaling In-Context Learning to 1,000 Examples (Dec 2022)
    20. PAL: Program-aided Language Models (Nov 2022)
    21. Large Language Models Are Human-Level Prompt Engineers (Nov 2022)
    22. Ignore Previous Prompt: Attack Techniques For Language Models (Nov 2022)
    23. Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods (Nov 2022)
    24. Teaching Algorithmic Reasoning via In-context Learning (Nov 2022)
    25. Enhancing Self-Consistency and Performance of Pre-Trained Language Models through Natural Language Inference (Nov 2022)
    26. Ask Me Anything: A simple strategy for prompting language models (Oct 2022)
    27. ReAct: Synergizing Reasoning and Acting in Language Models (Oct 2022)
    28. Prompting GPT-3 To Be Reliable (Oct 2022)
    29. Decomposed Prompting: A Modular Approach for Solving Complex Tasks (Oct 2022)
    30. Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought (Oct 2022)
    31. Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examples (Sep 2022)
    32. Promptagator: Few-shot Dense Retrieval From 8 Examples (Sep 2022)
    33. On the Advance of Making Language Models Better Reasoners (June 2022)
    34. Large Language Models are Zero-Shot Reasoners (May 2022)
    35. MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning (May 2022)
    36. Toxicity Detection with Generative Prompt-based Inference (May 2022)
    37. Learning to Transfer Prompts for Text Generation (May 2022)
    38. The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning (May 2022)
    39. A Taxonomy of Prompt Modifiers for Text-To-Image Generation (Apr 2022)
    40. PromptChainer: Chaining Large Language Model Prompts through Visual Programming (Mar 2022)
    41. Self-Consistency Improves Chain of Thought Reasoning in Language Models (March 2022)
    42. Training language models to follow instructions with human feedback
    43. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (Feb 2022)
    44. Chain of Thought Prompting Elicits Reasoning in Large Language Models (Jan 2022)
    45. Show Your Work: Scratchpads for Intermediate Computation with Language Models (Nov 2021)
    46. Generated Knowledge Prompting for Commonsense Reasoning (Oct 2021)
    47. Multitask Prompted Training Enables Zero-Shot Task Generalization (Oct 2021)
    48. Reframing Instructional Prompts to GPTk's Language (Sep 2021)
    49. Design Guidelines for Prompt Engineering Text-to-Image Generative Models (Sep 2021)
    50. Making Pre-trained Language Models Better Few-shot Learners (Aug 2021)
    51. Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity (April 2021)
    52. BERTese: Learning to Speak to BERT (April 2021)
    53. The Power of Scale for Parameter-Efficient Prompt Tuning (April 2021)
    54. Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm (Feb 2021)
    55. Calibrate Before Use: Improving Few-Shot Performance of Language Models (Feb 2021)
    56. Prefix-Tuning: Optimizing Continuous Prompts for Generation (Jan 2021)
    57. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts (Oct 2020)
    58. Language Models are Few-Shot Learners (May 2020)
    59. How Can We Know What Language Models Know? (July 2020)

      • Applications:

    1. Grimm in Wonderland: Prompt Engineering with Midjourney to Illustrate Fairytales (Feb 2023)
    2. LabelPrompt: Effective Prompt-based Learning for Relation Classification (Feb 2023)
    3. Prompt Tuning of Deep Neural Networks for Speaker-adaptive Visual Speech Recognition (Feb 2023)
    4. The Capacity for Moral Self-Correction in Large Language Models (Feb 2023)
    5. Prompting for Multimodal Hateful Meme Classification (Feb 2023)
    6. PLACES: Prompting Language Models for Social Conversation Synthesis (Feb 2023)
    7. Commonsense-Aware Prompting for Controllable Empathetic Dialogue Generation (Feb 2023)
    8. Crawling the Internal Knowledge-Base of Language Models (Jan 2023)
    9. Legal Prompt Engineering for Multilingual Legal Judgement Prediction (Dec 2022)
    10. Investigating Prompt Engineering in Diffusion Models (Nov 2022)
    11. Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering (Sep 2022)
    12. Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language (Oct 2022)
    13. Piloting Copilot and Codex: Hot Temperature, Cold Prompts, or Black Magic? (Oct 2022)
    14. Plot Writing From Scratch Pre-Trained Language Models (July 2022)

      • Collections:

    1. Chain-of-ThoughtsPapers
    2. Papers with Code
    3. Prompt Papers

Tools & Libraries

(Sorted by Name)

  • AI Test Kitchen
  • betterprompt
  • DreamStudio
  • DUST
  • Dyno
  • EveryPrompt
  • GPT Index
  • GPTTools
  • hwchase17/adversarial-prompts
  • Interactive Composition Explorer
  • LangChain
  • LearnGPT
  • Lexica
  • loom
  • Metaprompt
  • OpenAI Playground
  • OpenPrompt
  • Playground
  • Prodia
  • Prompt Base
  • Prompt Engine
  • Prompt Generator for OpenAI's DALL-E 2
  • Promptable
  • PromptInject
  • Prompts.ai
  • PromptSource
  • Scale SpellBook
  • sharegpt
  • ThoughtSource
  • Visual Prompt Builder

Datasets

(Sorted by Name)

  • Anthropic's Red Team dataset, (paper)
  • Awesome ChatGPT Prompts
  • DiffusionDB
  • Midjourney Prompts
  • P3 - Public Pool of Prompts
  • PartiPrompts
  • Real Toxicity Prompts
  • Stable Diffusion Dataset
  • WritingPrompts

Blog, Guides, Tutorials and Other Readings

(Sorted by Name)

  • 3 Principles for prompt engineering with GPT-3
  • A beginner-friendly guide to generative language models - LaMBDA guide
  • A Complete Introduction to Prompt Engineering for Large Language Models
  • A Generic Framework for ChatGPT Prompt Engineering
  • AI Content Generation
  • Awesome ChatGPT Prompts
  • Best 100+ Stable Diffusion Prompts
  • Best practices for prompt engineering with OpenAI API
  • Building GPT-3 applications — beyond the prompt
  • ChatGPT, AI and GPT-3 Apps and use cases
  • CMU Advanced NLP 2022: Prompting
  • Curtis64's set of prompt gists
  • DALL·E 2 Prompt Engineering Guide
  • DALL·E 2 Preview - Risks and Limitations
  • DALLE Prompt Book
  • DALL-E, Make Me Another Picasso, Please
  • Diffusion Models: A Practical Guide
  • Exploiting GPT-3 Prompts
  • Exploring Prompt Injection Attacks
  • Extrapolating to Unnatural Language Processing with GPT-3's In-context Learning: The Good, the Bad, and the Mysterious
  • Generative AI with Cohere: Part 1 - Model Prompting
  • Giving GPT-3 a Turing Test
  • GPT3 and Prompts: A quick primer
  • How to Draw Anything
  • How to get images that don't suck
  • How to write good prompts
  • Introduction to Reinforcement Learning with Human Feedback
  • In defense of prompt engineering
  • Language Models and Prompt Engineering: Systematic Survey of Prompting Methods in NLP
  • Learn Prompting
  • Methods of prompt programming
  • Mysteries of mode collapse
  • NLP for Text-to-Image Generators: Prompt Analysis
  • NLP with Deep Learning CS224N/Ling284 - Lecture 11: Promting, Instruction Tuning, and RLHF
  • Notes for Prompt Engineering by sw-yx
  • OpenAI Cookbook
  • OpenAI Prompt Examples for several applications
  • Pretrain, Prompt, Predict - A New Paradigm for NLP
  • Prompt Engineering 101 - Introduction and resources
  • Prompt Engineering 101: Autocomplete, Zero-shot, One-shot, and Few-shot prompting
  • Prompt Engineering 101
  • Prompt Engineering - A new profession ?
  • Prompt Engineering by co:here
  • Prompt Engineering by Microsoft
  • Prompt Engineering: The Career of Future
  • Prompt engineering davinci-003 on our own docs for automated support (Part I)
  • Prompt Engineering Guide: How to Engineer the Perfect Prompts
  • Prompt Engineering in GPT-3
  • Prompt Engineering Template
  • Prompt Engineering Topic by GitHub
  • Prompt Engineering: From Words to Art
  • Prompt Engineering with OpenAI's GPT-3 and other LLMs
  • Prompt injection attacks against GPT-3
  • Prompt injection to read out the secret OpenAI API key
  • Prompting in NLP: Prompt-based zero-shot learning
  • Prompting Methods with Language Models and Their Applications to Weak Supervision
  • Prompts as Programming by Gwern
  • Reverse Prompt Engineering for Fun and (no) Profit
  • So you want to be a prompt engineer: Critical careers of the future
  • Simulators
  • Start with an Instruction
  • Talking to machines: prompt engineering & injection
  • the Book - Fed Honeypot
  • The ChatGPT Prompt Book
  • Using GPT-Eliezer against ChatGPT Jailbreaking
  • What Is ChatGPT Doing … and Why Does It Work?

Feel free to open a PR if you think something is missing here. Always welcome feedback and suggestions.