Stable Diffusion Dynamic Prompts extension
A custom extension for AUTOMATIC1111/stable-diffusion-webui that implements an expressive template language for random or combinatorial prompt generation along with features to support deep wildcard directory structures.
Looking for ComfyUI nodes? Find them here.
Table of Contents
- Basic Usage
- Online resources
- Installation
- Configuration
- Troubleshooting
- Compatible Scripts
- Template syntax
- Combinatorial Generation
- Fixed seed
- Magic Prompt
- I'm feeling lucky
- Attention grabber
- Write prompts to file
- Jinja2 templates
- WILDCARD_DIR
- Collections
- Dynamic Prompts and Random Seeds
Basic Usage
Using this script, the prompt:
A {house|apartment|lodge|cottage} in {summer|winter|autumn|spring} by {2$$artist1|artist2|artist3}
Will produce any of the following prompts:
- A house in summer by artist1, artist2
- A lodge in autumn by artist3, artist1
- A cottage in winter by artist2, artist3
- ...
This is especially useful if you are searching for interesting combinations of artists and styles.
You can also pick a random string from a file. Assuming you have the file seasons.txt in WILDCARD_DIR (see below), then:
__seasons__ is coming
Might generate the following:
- Winter is coming
- Spring is coming
- ...
You can also use the same wildcard twice
I love __seasons__ better than __seasons__
- I love Winter better than Summer
- I love Spring better than Spring
More complete documentation can be found here.
Prefer a tutorial? Click here
Need a wildcard library? We've got you covered.
Online resources
You can find a list of tutorials and wildcard packs here
Installation
The extension can be installed directly from within the Extensions tab within the Webui
You can also install it manually by running the following command from within the webui directory:
git clone https://github.com/adieyal/sd-dynamic-prompting/ extensions/dynamic-prompts
Configuration
You can find various settings to change Dynamic Prompt's behaviour in the Settings tab in the Dynamic Prompts section.
Changing syntax
In case of a syntax clash with another extension, Dynamic Prompts allows you to change the definition of variant start and variant end. By default these are set to { and } respectively. , e.g. {red|green|blue}. In the settings tab, you can change these two any string, e.g. <red|green|blue> or even ::red|green|blue::.
By default, wildcards start with __(double underscore) and end with __. You can change this in the settings tab under wildcard wrap.
Wildcard settings
Dynamic Prompts automatically de-duplicates and sorts wildcard files before using them. If you would prefer to disable this functionality, you can uncheck the checkboxes in the settings tab.
Checking the "shuffle wildcards" checkbox will randomize the order of the wildcards, ensuring that running the combinatorial model will produce different images on different runs.
Troubleshooting
If you encounter an issue with Dynamic Prompts, follow these steps to resolve the problem:
-
Check that you have installed the latest version of both the Dynamic Prompts extension and library. To check the installed versions, open the Need Help? accordion in the Dynamic Prompts section of txt2image. You can find the latest version number of the extension here and the library here.
-
If the versions do not match, update the extension in the extensions tab and restart the webui. The extension should automatically update the library.
-
If the above step does not work, you might need to manually update the library using the following command:
python -m pip install -U dynamicprompts[attentiongrabber,magicprompt]- Restart the webui and check. If the webui uses a different python binary, find the correct path to the python binary and run:
/path/to/python/binary/python -m pip install -U dynamicprompts[attentiongrabber,magicprompt]-
If the Wildcard UI does not show, it could be due to an outdated library version. Check for errors in the terminal and update the library as described in step 3.
-
If you get an error message saying "No values found for wildcard some/wildcard", ensure that the file wildcard.txt is in extensions/sd-dynamic-prompts/wildcards/some/. The full path is required, as relative paths are not currently supported.
-
If the issue persists, search for solutions in the issues section on GitHub and the discussion forum. If you cannot find a solution, create a new issue and give it a descriptive name, such as "Wildcard values are being ignored in prompt templates". Provide the necessary context, including the versions of the Dynamic Prompts extension and library, and mention the operating system or colab being used. If there is an error in the terminal, copy and paste the entire text or take a screenshot.
-
Finally, it is essential to test and apply any fixes we release. Your feedback is valuable, as an issue that works in our environment may not work in yours.
Compatible Scripts
Dynamic Prompts works particularly well with X/Y Plot - setting Dynamic Prompts to combinatorial mode while using X/Y Plot, lets you exhaustively test prompt and paramter variations simultaneously.
Template syntax
Documentation can be found here
Fuzzy Glob/recursive wildcard file/directory matching
In addition to standard wildcard tokens such as __times__ -> times.txt, you can also use globbing to match against multiple files at once.
For instance:
__colors*__ will match any of the following:
- WILDCARD_DIR/colors.txt
- WILDCARD_DIR/colors1.txt
- WILDCARD_DIR/nested/folder/colors1.txt
__light/**/*__ will match:
- WILDCARD_DIR/nested/folder/light/a.txt
- WILDCARD_DIR/nested/folder/light/b.txt
but won't match
- WILDCARD_DIR/nested/folder/dark/a.txt
- WILDCARD_DIR/a.txt
You can also used character ranges [0-9] and [a-z] and single wildcard characters ?. For more examples see this article.
Combinatorial Generation
Instead of generating random prompts from a template, combinatorial generation produced every possible prompt from the given string. For example:
I {love|hate} {New York|Chicago} in {June|July|August}
will produce:
- I love New York in June
- I love New York in July
- I love New York in August
- I love Chicago in June
- I love Chicago in July
- I love Chicago in August
- I hate New York in June
- I hate New York in July
- I hate New York in August
- I hate Chicago in June
- I hate Chicago in July
- I hate Chicago in August
If a __wildcard__ is provided, then a new prompt will be produced for every value in the wildcard file. For example:
My favourite season is __seasons__
will produce:
- My favourite season is Summer
- My favourite season is August
- My favourite season is Winter
- My favourite season is Sprint
You also arbitrarily nest combinations inside wildcards and wildcards in combinations.
Combinatorial generation can be useful if you want to create an image for every artist in a file. It can be enabled by checking the Combinatorial generation checkbox in the ui. In order to prevent accidentially producing thousands of images, you can limit the total number of prompts generated using the Max Generations slider. A value of 0 (the default) will not set any limit.
Combinatorial Batches
The combinatorial batches slider lets you repeat the same set of prompts a number of times with different seeds. The default number of batches is 1.
Increasing the maximum number of generations
By default, the Batch count silder of automatic1111 has a maximum value of 100. This can limit the maximum number of generations when using combinatorial generation. You can change the maximum value of this slider by editing ui-config.json and change:
txt2img/Batch count/maximum": 100
to something larger like:
txt2img/Batch count/maximum": 1000
Fixed seed
Select this if you want to use the same seed for every generated image. If there are no wildcards then all the images will be identical. It is useful if you want to test the effect of a particular modifier. For example:
A beautiful day at the beach __medium/photography/filmtypes__
That way you can isolate the effect of each film type on a particular scene. Here are some of the results:
Magic Prompt
Use one of a number of prompt generation models to spice up your prompt.
Using Gustavosta's MagicPrompt model, Trained on 80,000 prompts from Lexica.art, it can help give you interesting new prompts on a given subject. Here are some automatically generated variations for "dogs playing football":
dogs playing football, in the streets of a japanese town at night, with people watching in wonder, in the style of studio ghibli and makoto shinkai, highly detailed digital art, trending on artstation
dogs playing football, in the background is a nuclear explosion. photorealism. hq. hyper. realistic. 4 k. award winning.
dogs playing football, in the background is a nuclear explosion. photorealistic. realism. 4 k wideshot. cinematic. unreal engine. artgerm. marc simonetti. jc leyendecker
This is compatible with the wildcard syntax described above.
Other models
- daspartho/prompt-extend (~500mb)
- succinctly/text2image-prompt-generator (~600mb) - Trained on Midjourney prompts
- microsoft/Promptist (~500mb) - Read the paper here
- AUTOMATIC/promptgen-lexart (~300mb) - Finetuned using 134,819 prompts from lexica.art
- AUTOMATIC/promptgen-majinai-safe (~300mb) - 1,654 prompts from majinai.art
- AUTOMATIC/promptgen-majinai-unsafe (~300mb) - 825 prompts from majinai.art (NSFW)
- Gustavosta/MagicPrompt-Dalle
- kmewhort/stable-diffusion-prompt-bolster (~500mb),
- Ar4ikov/gpt2-650k-stable-diffusion-prompt-generator (~500mb),
- Ar4ikov/gpt2-medium-650k-stable-diffusion-prompt-generator (~1.4gb),
- crumb/bloom-560m-RLHF-SD2-prompter-aesthetic (~1.1gb),
- Meli/GPT2-Prompt (~500mb),
- DrishtiSharma/StableDiffusion-Prompt-Generator-GPT-Neo-125M (~550mb)
The first time you use a model, it is downloaded. It is approximately 500mb and so will take some time depending on how fast your connection is. It will also take a few seconds on first activation as the model is loaded into memory. Note, if you're low in VRAM, you might get a Cuda error. My GPU uses less than 8GB but YMMV.
You can control the maximum prompt length with the Max magic prompt length slider. Magic prompt creativity can adjust the generated prompt but you will need to experiment with this setting.
Use the Magic prompt blocklist regex to filter out keywords. For example, if you want to avoid prompts containing Greg Rutkowski, add his name to this field.
If you are generating many prompts using Magic Prompt, then increasing the Magic Prompt batch size can improve significantly improve prompt generation speed. This may only be noticeable if you are not generating images as well since image generation is much slower than prompt generation.
I'm feeling lucky
Use the lexica.art API to create random prompts. Useful if you're looking for inspiration, or are simply too lazy to think of your own prompts. When this option is selected, the prompt in the main prompt box is used as a search string. For example, prompt "Mech warrior" might return:
- A large robot stone statue in the middle of a forest by Greg Rutkowski, Sung Choi, Mitchell Mohrhauser, Maciej Kuciara, Johnson Ting, Maxim Verehin, Peter Konig, final fantasy , 8k photorealistic, cinematic lighting, HD, high details, atmospheric,
- a beautiful portrait painting of a ( ( ( cyberpunk ) ) ) armor by simon stalenhag and pascal blanche and alphonse mucha and nekro. in style of digital art. colorful comic, film noirs, symmetry, brush stroke, vibrating colors, hyper detailed. octane render. trending on artstation
- symmetry!! portrait of a robot astronaut, floral! horizon zero dawn machine, intricate, elegant, highly detailed, digital painting, artstation, concept art, smooth, sharp focus, illustration, art by artgerm and greg rutkowski and alphonse mucha, 8 k
Leaving the prompt box blank returns a list of completely randomly chosen prompts.
Attention grabber
This option randomly selects a keyword in your prompt and adds a random amount of emphasis. Below is an example of how this affects the prompt:
a portrait an anthropomorphic panda mage casting a spell, wearing mage robes, landscape in background, cute, dnd character art portrait, by jason felix and peter mohrbacher, cinematic lighting
Tick the Fixed seed checkbox under Advanced options to see how emphasis changes your image without changing seed.
Write prompts to file
Check the write prompts to file checkbox in order to create a file with all generated prompts. The generated file is a slugified version of the prompt and can be found in the same directory as the generated images, e.g. outputs/txt2img-images
Jinja2 templates
Jinja2 templates is an experimental feature that enables you to define prompts imperatively. This is an advanced feature and is only recommended for users who are comfortable writing scripts.
To enable, open the advanced accordion and select Enable Jinja2 templates.
You can read about them in more detail here
WILDCARD_DIR
The extension looks for wildcard files in WILDCARD_DIR. The default location is /path/to/stable-diffusion-webui/extensions/sd-dynamic-prompts/wildcards. It can also be manually defined in the main webui config.json under wildcard_dir. When in doubt, the help text for the extension in the webui lists the full path to WILDCARD_DIR
Collections
The collections directory contains modifier libraries that you can use as is or to bootstrap your own. To get started, either use the Wildcard Manager tab to copy a one or more collections to your wildcards folder, or you can manually copy the files across. Three collections are bundled with the dynamic prompts extension.
Jumbo is a very large collection of wildcards across many categories including aesthetics, appearance, artists, medium, style, and time. It is a work in progress, but aims to provide good coverage of various modifier categories.
Parrotzone is a far smaller and more manageable collection sourced from https://proximacentaurib.notion.site/e28a4f8d97724f14a784a538b8589e7d?v=42948fd8f45c4d47a0edfc4b78937474.
Devilkkw focuses more on character building, clothes, gestures, food, etc
If you're using a Unix/Linux O/S, you can easily create a symlink to the relevant collection rather than copying it across if you don't plan to alter it. E.g.
ln -sr collections/parrotzone wildcards/
You can also download additional extensions by running python _tools/download_collections.py from within the extension's root directory, i.e. extensions/sd-dynamic-prompts/
Dynamic Prompts and Random Seeds
Random seeds play an important role in controlling the randomness of the generated outputs. Let's discuss how Dynamic Prompts works with random seeds in different scenarios.
Without Dynamic Prompts Enabled
- If the seed is set to -1: A random seed is picked. This seed is used to generate the first image, then the next image is created using seed + 1, and this pattern continues for subsequent images.
- If the seed is set to a specific number greater than -1: The process is similar to the one above, but starts with the user-specified seed.
- If the variation seed is defined, but variation strength is zero: The process remains the same as in the previous two points.
- If the variation seed is set to a number greater than 0: Every image is generated using the same initial seed (randomly selected or set by the user). The variation seed is either random (if set to -1) or the value chosen by the user. The first image is generated with the variation seed, the next with variation seed + 1, and so on.
Using With Dynamic Prompts Enabled in Random/Standard Mode:
- If the seed is set to -1: The process is similar to the first point in the previous section. However, the prompt is also selected using the same seed (if the random prompt generator is used).
- If the seed is set to a number greater than -1: The process is similar to the second point in the previous section. However, the difference is that a random prompt is also generated using the chosen seed (if the prompt generator is used).
- If the fixed seed checkbox is checked: The same seed is used for all images and prompts. This means the same image is generated repeatedly (this is useful for combinatorial generation).
- If both the fixed seed and unlink seed from prompt checkboxes are checked: A random seed is used for the prompt, but the same seed is used for all images. This setting can be useful if you want to see how different prompts affect the generation of the same image.
Variation Seeds with Dynamic Prompts
- Variation strength set to 0: Variations are ignored.
- Variation set to a number > 0: A variation seed is assigned to every image, incrementing by one each time. However, only 1 prompt is generated since you are looking for variations of the same image.
Combinatorial Mode with Variation Strength > 0
In this case, it only generates the first image for you, which is probably not what you want. To get the desired results, you might need to adjust the settings or use a different mode.