Semantic search and workflows powered by language models
txtai is an open-source platform for semantic search and workflows powered by language models.
Traditional search systems use keywords to find data. Semantic search has an understanding of natural language and identifies results that have the same meaning, not necessarily the same keywords.
txtai builds embeddings databases, which are a union of vector indexes and relational databases. This enables similarity search with SQL. Embeddings databases can stand on their own and/or serve as a powerful knowledge source for large language model (LLM) prompts.
Semantic workflows connect language models together to build intelligent applications.
Integrate vector search, conversational search, automatic summarization, transcription, translation and more.
Summary of txtai features:
π Similarity search with SQL, object storage, topic modeling, graph analysis, multiple vector index backends (Faiss, Annoy, Hnswlib) and support for external vector databasesπ Create embeddings for text, documents, audio, images and videoπ‘ Pipelines powered by language models that run question-answering, labeling, transcription, translation, summarization, LLM prompts and moreβͺοΈ οΈ Workflows to join pipelines together and aggregate business logic. txtai processes can be simple microservices or multi-model workflows.βοΈ Build with Python or YAML. API bindings available for JavaScript, Java, Rust and Go.βοΈ Cloud-native architecture that scales out with container orchestration systems (e.g. Kubernetes)
The following applications are powered by txtai.
| Application | Description |
|---|---|
| txtchat | Conversational search and workflows for all |
| paperai | Semantic search and workflows for medical/scientific papers |
| codequestion | Semantic search for developers |
| tldrstory | Semantic search for headlines and story text |
txtai is built with Python 3.8+, Hugging Face Transformers, Sentence Transformers and FastAPI
Why txtai?
In addition to traditional search systems, a growing number of language model backed solutions are available, so why txtai?
# Get started in a couple lines
from txtai.embeddings import Embeddings
embeddings = Embeddings({"path": "sentence-transformers/all-MiniLM-L6-v2"})
embeddings.index([(0, "Correct", None), (1, "Not what we hoped", None)])
embeddings.search("positive", 1)
#[(0, 0.2986203730106354)]- Built-in API makes it easy to develop applications using your programming language of choice
# app.yml
embeddings:
path: sentence-transformers/all-MiniLM-L6-v2CONFIG=app.yml uvicorn "txtai.api:app"
curl -X GET "http://localhost:8000/search?query=positive"- Run local - no need to ship data off to disparate remote services
- Work with micromodels all the way up to large language models (LLMs)
- Low footprint - install additional dependencies and scale up when needed
- Learn by example - notebooks cover all available functionality
Installation
The easiest way to install is via pip and PyPI
pip install txtai
Python 3.8+ is supported. Using a Python virtual environment is recommended.
See the detailed install instructions for more information covering optional dependencies, environment specific prerequisites, installing from source, conda support and how to run with containers.
Examples
The examples directory has a series of notebooks and applications giving an overview of txtai. See the sections below.
Semantic Search
Build semantic/similarity/vector/neural search applications.
| Notebook | Description | |
|---|---|---|
| Introducing txtai |
Overview of the functionality provided by txtai |  |
| Build an Embeddings index with Hugging Face Datasets | Index and search Hugging Face Datasets | |
| Build an Embeddings index from a data source | Index and search a data source with word embeddings | |
| Add semantic search to Elasticsearch | Add semantic search to existing search systems | |
| Similarity search with images | Embed images and text into the same space for search | |
| Distributed embeddings cluster | Distribute an embeddings index across multiple data nodes | |
| What's new in txtai 4.0 | Content storage, SQL, object storage, reindex and compressed indexes | |
| Anatomy of a txtai index | Deep dive into the file formats behind a txtai embeddings index | |
| Custom Embeddings SQL functions | Add user-defined functions to Embeddings SQL | |
| Model explainability | Explainability for semantic search | |
| Query translation | Domain-specific natural language queries with query translation | |
| Build a QA database | Question matching with semantic search | |
| Embeddings components | Composable search with vector, SQL and scoring components | |
| Semantic Graphs | Explore topics, data connectivity and run network analysis | |
| Topic Modeling with BM25 | Topic modeling backed by a BM25 index | |
| Prompt-driven search with LLMs | Embeddings-guided and Prompt-driven search with Large Language Models (LLMs) | |
| Embeddings in the Cloud | Load and use an embeddings index from the Hugging Face Hub | |
| Customize your own embeddings database | Ways to combine vector indexes with relational databases | |
Pipelines
Transform data with language model backed pipelines.
| Notebook | Description | |
|---|---|---|
| Extractive QA with txtai | Introduction to extractive question-answering with txtai | |
| Extractive QA with Elasticsearch | Run extractive question-answering queries with Elasticsearch | |
| Extractive QA to build structured data | Build structured datasets using extractive question-answering | |
| Apply labels with zero shot classification | Use zero shot learning for labeling, classification and topic modeling | |
| Building abstractive text summaries | Run abstractive text summarization | |
| Extract text from documents | Extract text from PDF, Office, HTML and more | |
| Text to speech generation | Generate speech from text | |
| Transcribe audio to text | Convert audio files to text | |
| Translate text between languages | Streamline machine translation and language detection | |
| Generate image captions and detect objects | Captions and object detection for images | |
| Near duplicate image detection | Identify duplicate and near-duplicate images | |
| API Gallery | Using txtai in JavaScript, Java, Rust and Go | |
Workflows
Efficiently process data at scale.
| Notebook | Description | |
|---|---|---|
| Run pipeline workflows |
Simple yet powerful constructs to efficiently process data | |
| Transform tabular data with composable workflows | Transform, index and search tabular data | |
| Tensor workflows | Performant processing of large tensor arrays | |
| Entity extraction workflows | Identify entity/label combinations | |
| Workflow Scheduling | Schedule workflows with cron expressions | |
| Push notifications with workflows | Generate and push notifications with workflows | |
| Pictures are a worth a thousand words | Generate webpage summary images with DALL-E mini | |
| Run txtai with native code | Execute workflows in native code with the Python C API | |
| Prompt templates and task chains | Build model prompts and connect tasks together with workflows | |
Model Training
Train NLP models.
| Notebook | Description | |
|---|---|---|
| Train a text labeler | Build text sequence classification models | |
| Train without labels | Use zero-shot classifiers to train new models | |
| Train a QA model | Build and fine-tune question-answering models | |
| Train a language model from scratch | Build new language models | |
| Export and run models with ONNX | Export models with ONNX, run natively in JavaScript, Java and Rust | |
| Export and run other machine learning models | Export and run models from scikit-learn, PyTorch and more | |
Applications
Series of example applications with txtai. Links to hosted versions on Hugging Face Spaces also provided.
| Application | Description | |
|---|---|---|
| Basic similarity search | Basic similarity search example. Data from the original txtai demo. | |
| Baseball stats | Match historical baseball player stats using vector search. | |
| Book search | Book similarity search application. Index book descriptions and query using natural language statements. | Local run only |
| Image search | Image similarity search application. Index a directory of images and run searches to identify images similar to the input query. | |
| Summarize an article | Summarize an article. Workflow that extracts text from a webpage and builds a summary. | |
| Wiki search | Wikipedia search application. Queries Wikipedia API and summarizes the top result. | |
| Workflow builder | Build and execute txtai workflows. Connect summarization, text extraction, transcription, translation and similarity search pipelines together to run unified workflows. |
Documentation
Full documentation on txtai including configuration settings for pipelines, workflows, indexing and the API.
Further Reading
- Introducing txtai, semantic search and workflows built on Transformers
- Tutorial series on Hashnode | dev.to
- What's new in txtai 5.0 | 4.0
- Getting started with semantic search | workflows
- Run workflows to transform data and build semantic search applications with txtai
- Semantic search on the cheap
- Serverless vector search with txtai
- Insights from the txtai console
- The big and small of txtai
Contributing
For those who would like to contribute to txtai, please see this guide.