• Stars
    star
    278
  • Rank 148,454 (Top 3 %)
  • Language
    Python
  • License
    MIT License
  • Created about 1 year ago
  • Updated about 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

a1111 implementation of https://github.com/ChenyangSi/FreeU

sd-webui-freeu

implementation of FreeU as an a1111 sd webui extension

At each of the 3 stages of the UNet decoder:

  • Apply a scalar on a window over the features of the backbone
  • Tame the frequencies of the skip connection

Settings

  • Start At Step: do not apply FreeU until this sampling step is reached
  • Stop At Step: apply FreeU until this sampling step is reached
  • Transition Smoothness: see $k_{smooth}$ in this desmos graph
  • Backbone n Scale: scalar applied to the backbone window during UNet stage n
  • Backbone n Offset: offset of the window, 1 is the same as 0 as the window wraps around the downsampled latent features
  • Backbone n Width: width of the window applied to the backbone
  • Skip n Scale: scalar applied to the low frequencies (low end) of the skip connection during UNet stage n
  • Skip n High End Scale: scalar applied to the high frequencies (high end) of the skip connection
  • Skip n Cutoff: ratio that separates low from high frequencies, 0 means to control the single lowest frequency with "Skip n Scale" and 1 means scale all frequencies with "Skip n Scale"

API

You can pass a single dict as the alwayson script args when making API calls:

{
    "alwayson_scripts": {
        "freeu": {
            "args": [{
                "enable": true,
                "start_ratio": 0.1,
                "stop_ratio": 0.9,
                "transition_smoothness": 0.1,
                "stage_infos": [
                    {
                        "backbone_factor": 1.2,
                        "backbone_offset": 0.5,
                        "backbone_width": 0.75,
                        "skip_factor": 0.9,
                        "skip_high_end_factor": 1.1,
                        "skip_cutoff": 0.3
                    },
                    {
                        "backbone_factor": 1.4,
                        "backbone_offset": 0.5,
                        "backbone_width": 0.75,
                        "skip_factor": 0.2,
                        "skip_high_end_factor": 1.1,
                        "skip_cutoff": 0.3
                    },
                    {
                        "backbone_factor": 1.1,
                        "backbone_offset": 0.5,
                        "backbone_width": 0.75,
                        "skip_factor": 0.9,
                        "skip_high_end_factor": 1.1,
                        "skip_cutoff": 0.3
                    }
                ]
            }]
        }
    }
}

It is possible to omit any of the entries. For example:

{
    "alwayson_scripts": {
        "freeu": {
            "args": [{
                "start_ratio": 0.1,
                "stage_infos": [
                    {
                        "backbone_factor": 0.8,
                        "backbone_offset": 0.5,
                        "skip_high_end_factor": 0.9
                    }
                ]
            }]
        }
    }
}

Here, since there is a single dict in the stage_infos array, freeu will only have an effect during the first stage of the unet.
If you want to modify only the second stage, prepend the "stage_infos" array with 1 empty dict {}.
If you want to modify only the third stage, prepend the "stage_infos" array with 2 empty dicts.

If "stop_ratio" or "start_ratio" is an integer, then it is a step number.
Otherwise, it is expected to be a float between 0.0 and 1.0 and it represents a ratio of the total sampling steps.