spacy-llm: Integrating LLMs into structured NLP pipelines

This package integrates Large Language Models (LLMs) into spaCy, featuring a modular system for fast prototyping and prompting, and turning unstructured responses into robust outputs for various NLP tasks, no training data required.

• Serializable llm component to integrate prompts into your pipeline
• Modular functions to define the task (prompting and parsing) and backend (model to use)
• Support for hosted APIs and self-hosted open-source models
• Integration with MiniChain and LangChain
• Access to OpenAI API, including GPT-4 and various GPT-3 models
• Built-in support for open-source Dolly models hosted on Hugging Face
• Usage examples for Named Entity Recognition and Text Classification
• Easy implementation of your own functions via spaCy's registry for custom prompting, parsing and model integrations

🧠 Motivation

Large Language Models (LLMs) feature powerful natural language understanding capabilities. With only a few (and sometimes no) examples, an LLM can be prompted to perform custom NLP tasks such as text categorization, named entity recognition, coreference resolution, information extraction and more.

spaCy is a well-established library for building systems that need to work with language in various ways. spaCy's built-in components are generally powered by supervised learning or rule-based approaches.

Supervised learning is much worse than LLM prompting for prototyping, but for many tasks it's much better for production. A transformer model that runs comfortably on a single GPU is extremely powerful, and it's likely to be a better choice for any task for which you have a well-defined output. You train the model with anything from a few hundred to a few thousand labelled examples, and it will learn to do exactly that. Efficiency, reliability and control are all better with supervised learning, and accuracy will generally be higher than LLM prompting as well.

spacy-llm lets you have the best of both worlds. You can quickly initialize a pipeline with components powered by LLM prompts, and freely mix in components powered by other approaches. As your project progresses, you can look at replacing some or all of the LLM-powered components as you require.

Of course, there can be components in your system for which the power of an LLM is fully justified. If you want a system that can synthesize information from multiple documents in subtle ways and generate a nuanced summary for you, bigger is better. However, even if your production system needs an LLM for some of the task, that doesn't mean you need an LLM for all of it. Maybe you want to use a cheap text classification model to help you find the texts to summarize, or maybe you want to add a rule-based system to sanity check the output of the summary. These before-and-after tasks are much easier with a mature and well-thought-out library, which is exactly what spaCy provides.

⏳ Install

spacy-llm will be installed automatically in future spaCy versions. For now, you can run the following in the same virtual environment where you already have spacy installed.

python -m pip install spacy-llm

⚠️ This package is still experimental and it is possible that changes made to the interface will be breaking in minor version updates.

🐍 Usage

The task and the backend have to be supplied to the llm pipeline component using spaCy's config system. This package provides various built-in functionality, as detailed in the API documentation.

Example 1: Add a text classifier using a GPT-3 model from OpenAI

Create a new API key from openai.com or fetch an existing one, and ensure the keys are set as environmental variables. For more background information, see the OpenAI section.

Create a config file config.cfg containing at least the following (or see the full example here):

[nlp]
lang = "en"
pipeline = ["llm"]

[components]

[components.llm]
factory = "llm"

[components.llm.task]
@llm_tasks = "spacy.TextCat.v2"
labels = ["COMPLIMENT", "INSULT"]

[components.llm.backend]
@llm_backends = "spacy.REST.v1"
api = "OpenAI"
config = {"model": "gpt-3.5-turbo", "temperature": 0.3}

Now run:

from spacy_llm.util import assemble

nlp = assemble("config.cfg")
doc = nlp("You look gorgeous!")
print(doc.cats)

Example 2: Add NER using an open-source model through Hugging Face

To run this example, ensure that you have a GPU enabled, and transformers, torch and CUDA installed. For more background information, see the DollyHF section.

Create a config file config.cfg containing at least the following (or see the full example here):

[nlp]
lang = "en"
pipeline = ["llm"]

[components]

[components.llm]
factory = "llm"

[components.llm.task]
@llm_tasks = "spacy.NER.v2"
labels = ["PERSON", "ORGANISATION", "LOCATION"]

[components.llm.backend]
@llm_backends = "spacy.Dolly_HF.v1"
# For better performance, use databricks/dolly-v2-12b instead
model = "databricks/dolly-v2-3b"

Now run:

from spacy_llm.util import assemble

nlp = assemble("config.cfg")
doc = nlp("Jack and Jill rode up the hill in Les Deux Alpes")
print([(ent.text, ent.label_) for ent in doc.ents])

Note that Hugging Face will download the "databricks/dolly-v2-3b" model the first time you use it. You can define the cached directory by setting the environmental variable HF_HOME. Also, you can upgrade the model to be "databricks/dolly-v2-12b" for better performance.

Example 3: Create the component directly in Python

The llm component behaves as any other spaCy component does, so adding it to an existing pipeline follows the same pattern:

import spacy

nlp = spacy.blank("en")
nlp.add_pipe(
    "llm",
    config={
        "task": {
            "@llm_tasks": "spacy.NER.v2",
            "labels": ["PERSON", "ORGANISATION", "LOCATION"]
        },
        "backend": {
            "@llm_backends": "spacy.REST.v1",
            "api": "OpenAI",
            "config": {"model": "gpt-3.5-turbo"},
        },
    },
)
nlp.initialize()
doc = nlp("Jack and Jill rode up the hill in Les Deux Alpes")
print([(ent.text, ent.label_) for ent in doc.ents])

Note that for efficient usage of resources, typically you would use nlp.pipe(docs) with a batch, instead of calling nlp(doc) with a single document.

Example 4: Implement your own custom task

To write a task, you need to implement two functions: generate_prompts that takes a list of spaCy Doc objects and transforms them into a list of prompts, and parse_responses that transforms the LLM outputs into annotations on the Doc, e.g. entity spans, text categories and more.

To register your custom task with spaCy, decorate a factory function using the spacy_llm.registry.llm_tasks decorator with a custom name that you can refer to in your config.

📖 For more details, see the usage example on writing your own task

from typing import Iterable, List
from spacy.tokens import Doc
from spacy_llm.registry import registry
from spacy_llm.util import split_labels


@registry.llm_tasks("my_namespace.MyTask.v1")
def make_my_task(labels: str, my_other_config_val: float) -> "MyTask":
    labels_list = split_labels(labels)
    return MyTask(labels=labels_list, my_other_config_val=my_other_config_val)


class MyTask:
    def __init__(self, labels: List[str], my_other_config_val: float):
        ...

    def generate_prompts(self, docs: Iterable[Doc]) -> Iterable[str]:
        ...

    def parse_responses(
        self, docs: Iterable[Doc], responses: Iterable[str]
    ) -> Iterable[Doc]:
        ...

# config.cfg (excerpt)
[components.llm.task]
@llm_tasks = "my_namespace.MyTask.v1"
labels = LABEL1,LABEL2,LABEL3
my_other_config_val = 0.3

Logging

spacy-llm has a built-in logger that can log the prompt sent to the LLM as well as its raw response. This logger uses the debug level and by default has a logging.NullHandler() configured.

In order to use this logger, you can setup a simple handler like this:

import logging
import spacy_llm


spacy_llm.logger.addHandler(logging.StreamHandler())
spacy_llm.logger.setLevel(logging.DEBUG)

NOTE: Any logging handler will work here so you probably want to use some sort of rotating FileHandler as the generated prompts can be quite long, especially for tasks with few-shot examples.

Then when using the pipeline you'll be able to view the prompt and response.

E.g. with the config and code from Example 1 above:

from spacy_llm.util import assemble


nlp = assemble("config.cfg")
doc = nlp("You look gorgeous!")
print(doc.cats)

You will see logging output similar to:

Generated prompt for doc: You look gorgeous!

You are an expert Text Classification system. Your task is to accept Text as input
and provide a category for the text based on the predefined labels.

Classify the text below to any of the following labels: COMPLIMENT, INSULT
The task is non-exclusive, so you can provide more than one label as long as
they're comma-delimited. For example: Label1, Label2, Label3.
Do not put any other text in your answer, only one or more of the provided labels with nothing before or after.
If the text cannot be classified into any of the provided labels, answer `==NONE==`.

Here is the text that needs classification


Text:
'''
You look gorgeous!
'''

Backend response for doc: You look gorgeous!
COMPLIMENT

print(doc.cats) to standard output should look like:

{'COMPLIMENT': 1.0, 'INSULT': 0.0}

📓 API

spacy-llm exposes a llm factory that accepts the following configuration options:

An llm component is defined by two main settings:

• A task, defining the prompt to send to the LLM as well as the functionality to parse the resulting response back into structured fields on spaCy's Doc objects.
• A backend defining the model to use and how to connect to it. Note that spacy-llm supports both access to external APIs (such as OpenAI) as well as access to self-hosted open-source LLMs (such as using Dolly through Hugging Face).

Moreover, spacy-llm exposes a customizable caching functionality to avoid running the same document through an LLM service (be it local or through a REST API) more than once.

Finally, you can choose to save a stringified version of LLM prompts/responses within the Doc.user_data["llm_io"] attribute by setting save_io to True. Doc.user_data["llm_io"] is a dictionary containing one entry for every LLM component within the spaCy pipeline. Each entry is itself a dictionary, with two keys: prompt and response.

A note on validate_types: by default, spacy-llm checks whether the signatures of the backend and task callables are consistent with each other and emits a warning if they don't. validate_types can be set to False if you want to disable this behavior.

Tasks

A task defines an NLP problem or question, that will be sent to the LLM via a prompt. Further, the task defines how to parse the LLM's responses back into structured information. All tasks are registered in spaCy's llm_tasks registry.

Practically speaking, a task should adhere to the Protocol LLMTask defined in ty.py. It needs to define a generate_prompts function and a parse_responses function.

Moreover, the task may define an optional scorer method. It should accept an iterable of Examples as input and return a score dictionary. If the scorer method is defined, spacy-llm will call it to evaluate the component.

function task.generate_prompts

Takes a collection of documents, and returns a collection of "prompts", which can be of type Any. Often, prompts are of type str - but this is not enforced to allow for maximum flexibility in the framework.

function task.parse_responses

Takes a collection of LLM responses and the original documents, parses the responses into structured information, and sets the annotations on the documents. The parse_responses function is free to set the annotations in any way, including Doc fields like ents, spans or cats, or using custom defined fields.

The responses are of type Iterable[Any], though they will often be str objects. This depends on the return type of the backend.

spacy.NER.v2

The built-in NER task supports both zero-shot and few-shot prompting. This version also supports explicitly defining the provided labels with custom descriptions.

[components.llm.task]
@llm_tasks = "spacy.NER.v2"
labels = ["PERSON", "ORGANISATION", "LOCATION"]
examples = null

The NER task implementation doesn't currently ask the LLM for specific offsets, but simply expects a list of strings that represent the enties in the document. This means that a form of string matching is required. This can be configured by the following parameters:

• The single_match parameter is typically set to False to allow for multiple matches. For instance, the response from the LLM might only mention the entity "Paris" once, but you'd still want to mark it every time it occurs in the document.
• The case-sensitive matching is typically set to False to be robust against case variances in the LLM's output.
• The alignment_mode argument is used to match entities as returned by the LLM to the tokens from the original Doc - specifically it's used as argument in the call to doc.char_span(). The "strict" mode will only keep spans that strictly adhere to the given token boundaries. "contract" will only keep those tokens that are fully within the given range, e.g. reducing "New Y" to "New". Finally, "expand" will expand the span to the next token boundaries, e.g. expanding "New Y" out to "New York".

To perform few-shot learning, you can write down a few examples in a separate file, and provide these to be injected into the prompt to the LLM. The default reader spacy.FewShotReader.v1 supports .yml, .yaml, .json and .jsonl.

- text: Jack and Jill went up the hill.
  entities:
    PERSON:
      - Jack
      - Jill
    LOCATION:
      - hill
- text: Jack fell down and broke his crown.
  entities:
    PERSON:
      - Jack

[components.llm.task]
@llm_tasks = "spacy.NER.v2"
labels = PERSON,ORGANISATION,LOCATION
[components.llm.task.examples]
@misc = "spacy.FewShotReader.v1"
path = "ner_examples.yml"

If you don't have specific examples to provide to the LLM, you can write definitions for each label and provide them via the label_definitions argument. This lets you tell the LLM exactly what you're looking for rather than relying on the LLM to interpret its task given just the label name. Label descriptions are freeform so you can write whatever you want here, but through some experiments a brief description along with some examples and counter examples seems to work quite well.

[components.llm.task]
@llm_tasks = "spacy.NER.v2"
labels = PERSON,SPORTS_TEAM
[components.llm.task.label_definitions]
PERSON = "Extract any named individual in the text."
SPORTS_TEAM = "Extract the names of any professional sports team. e.g. Golden State Warriors, LA Lakers, Man City, Real Madrid"

Label descriptions can also be used with explicit examples to give as much info to the LLM backend as possible.

spacy.NER.v1

The original version of the built-in NER task supports both zero-shot and few-shot prompting.

[components.llm.task]
@llm_tasks = "spacy.NER.v1"
labels = PERSON,ORGANISATION,LOCATION
examples = null

The NER task implementation doesn't currently ask the LLM for specific offsets, but simply expects a list of strings that represent the enties in the document. This means that a form of string matching is required. This can be configured by the following parameters:

• The single_match parameter is typically set to False to allow for multiple matches. For instance, the response from the LLM might only mention the entity "Paris" once, but you'd still want to mark it every time it occurs in the document.
• The case-sensitive matching is typically set to False to be robust against case variances in the LLM's output.
• The alignment_mode argument is used to match entities as returned by the LLM to the tokens from the original Doc - specifically it's used as argument in the call to doc.char_span(). The "strict" mode will only keep spans that strictly adhere to the given token boundaries. "contract" will only keep those tokens that are fully within the given range, e.g. reducing "New Y" to "New". Finally, "expand" will expand the span to the next token boundaries, e.g. expanding "New Y" out to "New York".

To perform few-shot learning, you can write down a few examples in a separate file, and provide these to be injected into the prompt to the LLM. The default reader spacy.FewShotReader.v1 supports .yml, .yaml, .json and .jsonl.

- text: Jack and Jill went up the hill.
  entities:
    PERSON:
      - Jack
      - Jill
    LOCATION:
      - hill
- text: Jack fell down and broke his crown.
  entities:
    PERSON:
      - Jack

[components.llm.task]
@llm_tasks = "spacy.NER.v1"
labels = PERSON,ORGANISATION,LOCATION
[components.llm.task.examples]
@misc = "spacy.FewShotReader.v1"
path = "ner_examples.yml"

spacy.SpanCat.v2

The built-in SpanCat task is a simple adaptation of the NER task to support overlapping entities and store its annotations in doc.spans.

[components.llm.task]
@llm_tasks = "spacy.SpanCat.v2"
labels = ["PERSON", "ORGANISATION", "LOCATION"]
examples = null

Except for the spans_key parameter, the SpanCat task reuses the configuration from the NER task. Refer to its documentation for more insight.

spacy.SpanCat.v1

The original version of the built-in SpanCat task is a simple adaptation of the v1 NER task to support overlapping entities and store its annotations in doc.spans.

[components.llm.task]
@llm_tasks = "spacy.SpanCat.v1"
labels = PERSON,ORGANISATION,LOCATION
examples = null

Except for the spans_key parameter, the SpanCat task reuses the configuration from the NER task. Refer to its documentation for more insight.

spacy.TextCat.v3

Version 3 (the most recent) of the built-in TextCat task supports both zero-shot and few-shot prompting. It allows setting definitions of labels. Those definitions are included in the prompt.

[components.llm.task]
@llm_tasks = "spacy.TextCat.v3"
labels = ["COMPLIMENT", "INSULT"]
label_definitions = {
    "COMPLIMENT": "a polite expression of praise or admiration.",
    "INSULT": "a disrespectful or scornfully abusive remark or act."
}
examples = null

To perform few-shot learning, you can write down a few examples in a separate file, and provide these to be injected into the prompt to the LLM. The default reader spacy.FewShotReader.v1 supports .yml, .yaml, .json and .jsonl.

[
  {
    "text": "You look great!",
    "answer": "Compliment"
  },
  {
    "text": "You are not very clever at all.",
    "answer": "Insult"
  }
]

[components.llm.task]
@llm_tasks = "spacy.TextCat.v3"
labels = ["COMPLIMENT", "INSULT"]
label_definitions = {
    "COMPLIMENT": "a polite expression of praise or admiration.",
    "INSULT": "a disrespectful or scornfully abusive remark or act."
}
[components.llm.task.examples]
@misc = "spacy.FewShotReader.v1"
path = "textcat_examples.json"

spacy.TextCat.v2

Version 2 of the built-in TextCat task supports both zero-shot and few-shot prompting and includes an improved prompt template.

[components.llm.task]
@llm_tasks = "spacy.TextCat.v2"
labels = ["COMPLIMENT", "INSULT"]
examples = null

To perform few-shot learning, you can write down a few examples in a separate file, and provide these to be injected into the prompt to the LLM. The default reader spacy.FewShotReader.v1 supports .yml, .yaml, .json and .jsonl.

[
  {
    "text": "You look great!",
    "answer": "Compliment"
  },
  {
    "text": "You are not very clever at all.",
    "answer": "Insult"
  }
]

[components.llm.task]
@llm_tasks = "spacy.TextCat.v2"
labels = ["COMPLIMENT", "INSULT"]
[components.llm.task.examples]
@misc = "spacy.FewShotReader.v1"
path = "textcat_examples.json"

spacy.TextCat.v1

Version 1 of the built-in TextCat task supports both zero-shot and few-shot prompting.

[components.llm.task]
@llm_tasks = "spacy.TextCat.v1"
labels = COMPLIMENT,INSULT
examples = null

To perform few-shot learning, you can write down a few examples in a separate file, and provide these to be injected into the prompt to the LLM. The default reader spacy.FewShotReader.v1 supports .yml, .yaml, .json and .jsonl.

[
  {
    "text": "You look great!",
    "answer": "Compliment"
  },
  {
    "text": "You are not very clever at all.",
    "answer": "Insult"
  }
]

[components.llm.task]
@llm_tasks = "spacy.TextCat.v2"
labels = COMPLIMENT,INSULT
[components.llm.task.examples]
@misc = "spacy.FewShotReader.v1"
path = "textcat_examples.json"

spacy.REL.v1

The built-in REL task supports both zero-shot and few-shot prompting. It relies on an upstream NER component for entities extraction.

[components.llm.task]
@llm_tasks = "spacy.REL.v1"
labels = ["LivesIn", "Visits"]

To perform few-shot learning, you can write down a few examples in a separate file, and provide these to be injected into the prompt to the LLM. The default reader spacy.FewShotReader.v1 supports .yml, .yaml, .json and .jsonl.

{"text": "Laura bought a house in Boston with her husband Mark.", "ents": [{"start_char": 0, "end_char": 5, "label": "PERSON"}, {"start_char": 24, "end_char": 30, "label": "GPE"}, {"start_char": 48, "end_char": 52, "label": "PERSON"}], "relations": [{"dep": 0, "dest": 1, "relation": "LivesIn"}, {"dep": 2, "dest": 1, "relation": "LivesIn"}]}
{"text": "Michael travelled through South America by bike.", "ents": [{"start_char": 0, "end_char": 7, "label": "PERSON"}, {"start_char": 26, "end_char": 39, "label": "LOC"}], "relations": [{"dep": 0, "dest": 1, "relation": "Visits"}]}

Note: the REL task relies on pre-extracted entities to make its prediction. Hence, you'll need to add a component that populates doc.ents with recognized spans to your spaCy pipeline and put it before the REL component.

[components.llm.task]
@llm_tasks = "spacy.REL.v1"
labels = ["LivesIn", "Visits"]
[components.llm.task.examples]
@misc = "spacy.FewShotReader.v1"
path = "rel_examples.jsonl"

spacy.Lemma.v1

The Lemma.v1 task lemmatizes the provided text and updates the lemma_ attribute in the doc's tokens accordingly.

[components.llm.task]
@llm_tasks = "spacy.Lemma.v1"
examples = null

Lemma.v1 prompts the LLM to lemmatize the passed text and return the lemmatized version as a list of tokens and their corresponding lemma. E. g. the text I'm buying ice cream for my friends should invoke the response

I: I
'm: be
buying: buy
ice: ice
cream: cream
for: for
my: my
friends: friend
.: .

If for any given text/doc instance the number of lemmas returned by the LLM doesn't match the number of tokens recognized by spaCy, no lemmas are stored in the corresponding doc's tokens. Otherwise the tokens .lemma_ property is updated with the lemma suggested by the LLM.

To perform few-shot learning, you can write down a few examples in a separate file, and provide these to be injected into the prompt to the LLM. The default reader spacy.FewShotReader.v1 supports .yml, .yaml, .json and .jsonl.

- text: I'm buying ice cream.
  lemmas:
    - "I": "I"
    - "'m": "be"
    - "buying": "buy"
    - "ice": "ice"
    - "cream": "cream"
    - ".": "."

- text: I've watered the plants.
  lemmas:
    - "I": "I"
    - "'ve": "have"
    - "watered": "water"
    - "the": "the"
    - "plants": "plant"
    - ".": "."

[components.llm.task]
@llm_tasks = "spacy.Lemma.v1"
[components.llm.task.examples]
@misc = "spacy.FewShotReader.v1"
path = "lemma_examples.yml"

spacy.NoOp.v1

This task is only useful for testing - it tells the LLM to do nothing, and does not set any fields on the docs.

[components.llm.task]
@llm_tasks = "spacy.NoOp.v1"

Backends

A backend defines which LLM model to query, and how to query it. It can be a simple function taking a collection of prompts (consistent with the output type of task.generate_prompts()) and returning a collection of responses (consistent with the expected input of parse_responses). Generally speaking, it's a function of type Callable[[Iterable[Any]], Iterable[Any]], but specific implementations can have other signatures, like Callable[[Iterable[str]], Iterable[str]].

All built-in backends are registered in llm_backends. If no backend is specified, the repo currently connects to the OpenAI API by default, using the built-in REST protocol, and accesses the "gpt-3.5-turbo" model.

❓ Why are there backends for third-party libraries in addition to a native REST backend and which should I choose?

Third-party libraries like langchain or minichain focus on prompt management, integration of many different LLM APIs, and other related features such as conversational memory or agents. spacy-llm on the other hand emphasizes features we consider useful in the context of NLP pipelines utilizing LLMs to process documents (mostly) independent from each other. It makes sense that the feature set of such third-party libraries and spacy-llm is not identical - and users might want to take advantage of features not available in spacy-llm.

The advantage of offering our own REST backend is that we can ensure a larger degree of stability of robustness, as we can guarantee backwards-compatibility and more smoothly integrated error handling.

Ultimately we recommend trying to implement your use case using the REST backend first (which is configured as the default backend). If however there are features or APIs not covered by spacy-llm, it's trivial to switch to the backend of a third-party library - and easy to customize the prompting mechanism, if so required.

OpenAI

When the backend uses OpenAI, you have to get an API key from openai.com, and ensure that the keys are set as environmental variables:

export OPENAI_API_KEY="sk-..."
export OPENAI_API_ORG="org-..."

spacy.REST.v1

This default backend uses requests and a simple retry mechanism to access an API.

[components.llm.backend]
@llm_backends = "spacy.REST.v1"
api = "OpenAI"
config = {"model": "gpt-3.5-turbo", "temperature": 0.3}

When api is set to OpenAI, the following settings can be defined in the config dictionary:

• model: one of the following list of supported models:
  • "gpt-4"
  • "gpt-4-0314"
  • "gpt-4-32k"
  • "gpt-4-32k-0314"
  • "gpt-3.5-turbo"
  • "gpt-3.5-turbo-0301"
  • "text-davinci-003"
  • "text-davinci-002"
  • "text-curie-001"
  • "text-babbage-001"
  • "text-ada-001"
  • "davinci"
  • "curie"
  • "babbage"
  • "ada"
• url: By default, this is https://api.openai.com/v1/completions. For models requiring the chat endpoint, use https://api.openai.com/v1/chat/completions.

spacy.MiniChain.v1

To use MiniChain for the API retrieval part, make sure you have installed it first:

python -m pip install "minichain>=0.3,<0.4"
# Or install with spacy-llm directly
python -m pip install "spacy-llm[minichain]"

Note that MiniChain currently only supports Python 3.8, 3.9 and 3.10.

Example config blocks:

[components.llm.backend]
@llm_backends = "spacy.MiniChain.v1"
api = "OpenAI"

[components.llm.backend.query]
@llm_queries = "spacy.RunMiniChain.v1"

The default query (spacy.RunMiniChain.v1) executes the prompts by running model(text).run() for each given textual prompt.

spacy.LangChain.v1

To use LangChain for the API retrieval part, make sure you have installed it first:

python -m pip install "langchain>=0.0.144,<0.1"
# Or install with spacy-llm directly
python -m pip install "spacy-llm[langchain]"

Note that LangChain currently only supports Python 3.9 and beyond.

Example config block:

[components.llm.backend]
@llm_backends = "spacy.LangChain.v1"
api = "OpenAI"
query = {"@llm_queries": "spacy.CallLangChain.v1"}
config = {"temperature": 0.3}

The default query (spacy.CallLangChain.v1) executes the prompts by running model(text) for each given textual prompt.

spacy.Dolly_HF.v1

To use this backend, ideally you have a GPU enabled and have installed transformers, torch and CUDA in your virtual environment. This allows you to have the setting device=cuda:0 in your config, which ensures that the model is loaded entirely on the GPU (and fails otherwise).

You can do so with

python -m pip install "spacy-llm[transformers]" "transformers[sentencepiece]"

If you don't have access to a GPU, you can install accelerate and setdevice_map=auto instead, but be aware that this may result in some layers getting distributed to the CPU or even the hard drive, which may ultimately result in extremely slow queries.

python -m pip install "accelerate>=0.16.0,<1.0"

Example config block:

[components.llm.backend]
@llm_backends = "spacy.Dolly_HF.v1"
model = "databricks/dolly-v2-3b"

Supported models (see the Databricks models page on Hugging Face for details):

• "databricks/dolly-v2-3b"
• "databricks/dolly-v2-7b"
• "databricks/dolly-v2-12b"

Note that Hugging Face will download this model the first time you use it - you can define the cached directory by setting the environmental variable HF_HOME.

spacy.StableLM_HF.v1

To use this backend, ideally you have a GPU enabled and have installed transformers, torch and CUDA in your virtual environment.

You can do so with

python -m pip install "spacy-llm[transformers]" "transformers[sentencepiece]"

If you don't have access to a GPU, you can install accelerate and setdevice_map=auto instead, but be aware that this may result in some layers getting distributed to the CPU or even the hard drive, which may ultimately result in extremely slow queries.

python -m pip install "accelerate>=0.16.0,<1.0"

Example config block:

[components.llm.backend]
@llm_backends = "spacy.StableLM_HF.v1"
model = "stabilityai/stablelm-tuned-alpha-7b"

Supported models (see the Stability AI StableLM GitHub repo for details):

• "stabilityai/stablelm-base-alpha-3b"
• "stabilityai/stablelm-base-alpha-7b"
• "stabilityai/stablelm-tuned-alpha-3b"
• "stabilityai/stablelm-tuned-alpha-7b"

Note that Hugging Face will download this model the first time you use it - you can define the cached directory by setting the environmental variable HF_HOME.

spacy.OpenLLaMaHF.v1

To use this backend, ideally you have a GPU enabled and have installed

• transformers[sentencepiece]
• torch
• CUDA in your virtual environment.

You can do so with

python -m pip install "spacy-llm[transformers]" "transformers[sentencepiece]"

If you don't have access to a GPU, you can install accelerate and setdevice_map=auto instead, but be aware that this may result in some layers getting distributed to the CPU or even the hard drive, which may ultimately result in extremely slow queries.

python -m pip install "accelerate>=0.16.0,<1.0"

Example config block:

[components.llm.backend]
@llm_backends = "spacy.OpenLLaMaHF.v1"
model = "openlm-research/open_llama_3b_350bt_preview"

Supported models (see the OpenLM Research OpenLLaMa GitHub repo for details):

• "openlm-research/open_llama_3b_350bt_preview"
• "openlm-research/open_llama_3b_600bt_preview"
• "openlm-research/open_llama_7b_400bt_preview"
• "openlm-research/open_llama_7b_700bt_preview"

Note that Hugging Face will download this model the first time you use it - you can define the cached directory by setting the environmental variable HF_HOME.

Cache

Interacting with LLMs, either through an external API or a local instance, is costly. Since developing an NLP pipeline generally means a lot of exploration and prototyping, spacy-llm implements a built-in cache to avoid reprocessing the same documents at each run that keeps batches of documents stored on disk.

Example config block:

[components.llm.cache]
@llm_misc = "spacy.BatchCache.v1"
path = "path/to/cache"
batch_size = 64
max_batches_in_mem = 4

When retrieving a document, the BatchCache will first figure out what batch the document belongs to. If the batch isn't in memory it will try to load the batch from disk and then move it into memory.

Note that since the cache is generated by a registered function, you can also provide your own registered function returning your own cache implementation. If you wish to do so, ensure that your cache object adheres to the Protocol defined in spacy_llm.ty.Cache.

Various functions

spacy.FewShotReader.v1

This function is registered in spaCy's misc registry, and reads in examples from a .yml, .yaml, .json or .jsonl file. It uses srsly to read in these files and parses them depending on the file extension.

[components.llm.task.examples]
@misc = "spacy.FewShotReader.v1"
path = "ner_examples.yml"

spacy.FileReader.v1

This function is registered in spaCy's misc registry, and reads a file provided to the path to return a str representation of its contents. This function is typically used to read Jinja files containing the prompt template.

[components.llm.task.template]
@misc = "spacy.FileReader.v1"
path = "ner_template.jinja2"

Normalizer functions

These functions provide simple normalizations for string comparisons, e.g. between a list of specified labels and a label given in the raw text of the LLM response. They are registered in spaCy's misc registry and have the signature Callable[[str], str].

• spacy.StripNormalizer.v1: only apply text.strip()
• spacy.LowercaseNormalizer.v1: applies text.strip().lower() to compare strings in a case-insensitive way.

🚀 Ongoing work

In the near future, we will

• Add more example tasks
• Support a broader range of models
• Provide more example use-cases and tutorials
• Make the built-in tasks easier to customize via Jinja templates to define the instructions & examples

PRs are always welcome!

📝️ Reporting issues

If you have questions regarding the usage of spacy-llm, or want to give us feedback after giving it a spin, please use the discussion board. Bug reports can be filed on the spaCy issue tracker. Thank you!