• Stars
    star
    519
  • Rank 85,261 (Top 2 %)
  • Language
    Python
  • License
    Apache License 2.0
  • Created over 1 year ago
  • Updated 9 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A Python vector database you just need - no more, no less.

VectorDB from Jina AI logo

A Python vector database you just need - no more, no less.

PyPI PyPI - Downloads from official pypistats Github CD status

vectordb is a Pythonic vector database offers a comprehensive suite of CRUD (Create, Read, Update, Delete) operations and robust scalability options, including sharding and replication. It's readily deployable in a variety of environments, from local to on-premise and cloud. vectordb delivers exactly what you need - no more, no less. It's a testament to effective Pythonic design without over-engineering, making it a lean yet powerful solution for all your needs.

vectordb capitalizes on the powerful retrieval prowess of DocArray and the scalability, reliability, and serving capabilities of Jina. Here's the magic: DocArray serves as the engine driving vector search logic, while Jina guarantees efficient and scalable index serving. This synergy culminates in a robust, yet user-friendly vector database experience - that's vectordb for you.

Install

pip install vectordb
Use vectordb from Jina AI locally Use vectordb from Jina AI as a service Use vectordb from Jina AI on Jina AI Cloud

Getting started with vectordb locally

  1. Kick things off by defining a Document schema with the DocArray dataclass syntax:
from docarray import BaseDoc
from docarray.typing import NdArray

class ToyDoc(BaseDoc):
  text: str = ''
  embedding: NdArray[128]
  1. Opt for a pre-built database (like InMemoryExactNNVectorDB or HNSWVectorDB), and apply the schema:
from docarray import DocList
import numpy as np
from vectordb import InMemoryExactNNVectorDB, HNSWVectorDB

# Specify your workspace path
db = InMemoryExactNNVectorDB[ToyDoc](workspace='./workspace_path')

# Index a list of documents with random embeddings
doc_list = [ToyDoc(text=f'toy doc {i}', embedding=np.random.rand(128)) for i in range(1000)]
db.index(inputs=DocList[ToyDoc](doc_list))

# Perform a search query
query = ToyDoc(text='query', embedding=np.random.rand(128))
results = db.search(inputs=DocList[ToyDoc]([query]), limit=10)

# Print out the matches
for m in results[0].matches:
  print(m)

Since we issued a single query, results contains only one element. The nearest neighbour search results are conveniently stored in the .matches attribute.

Getting started with vectordb as a service

vectordb is designed to be easily served as a service, supporting gRPC, HTTP, and Websocket communication protocols.

Server Side

On the server side, you would start the service as follows:

with db.serve(protocol='grpc', port=12345, replicas=1, shards=1) as service:
   service.block()

This command starts vectordb as a service on port 12345, using the gRPC protocol with 1 replica and 1 shard.

Client Side

On the client side, you can access the service with the following commands:

from vectordb import Client

# Instantiate a client connected to the server. In practice, replace 0.0.0.0 to the server IP address.
client = Client[ToyDoc](address='grpc://0.0.0.0:12345')

# Perform a search query
results = client.search(inputs=DocList[ToyDoc]([query]), limit=10)

This allows you to perform a search query, receiving the results directly from the remote vectordb service.

Hosting vectordb on Jina AI Cloud

You can seamlessly deploy your vectordb instance to Jina AI Cloud, which ensures access to your database from any location.

Start by embedding your database instance or class into a Python file:

# example.py
from docarray import BaseDoc
from vectordb import InMemoryExactNNVectorDB

db = InMemoryExactNNVectorDB[ToyDoc](workspace='./vectordb') # notice how `db` is the instance that we want to serve

if __name__ == '__main__':
    # IMPORTANT: make sure to protect this part of the code using __main__ guard
    with db.serve() as service:
        service.block()

Next, follow these steps to deploy your instance:

  1. If you haven't already, sign up for a Jina AI Cloud account.

  2. Use the jc command line to login to your Jina AI Cloud account:

jc login
  1. Deploy your instance:
vectordb deploy --db example:db

Connect from the client

After deployment, use the vectordb Client to access the assigned endpoint:

from vectordb import Client

# replace the ID with the ID of your deployed DB as shown in the screenshot above
c = Client(address='grpcs://ID.wolf.jina.ai')

Manage your deployed instances using jcloud

You can then list, pause, resume or delete your deployed DBs with jc command:

jcloud list ID

jcloud pause ID or jcloud resume ID

jcloud remove ID

Advanced Topics

What is a vector database?

Vector databases serve as sophisticated repositories for embeddings, capturing the essence of semantic similarity among disparate objects. These databases facilitate similarity searches across a myriad of multimodal data types, paving the way for a new era of information retrieval. By providing contextual understanding and enriching generation results, vector databases greatly enhance the performance and utility of Language Learning Models (LLM). This underscores their pivotal role in the evolution of data science and machine learning applications.

CRUD support

Both the local library usage and the client-server interactions in vectordb share the same API. This provides index, search, update, and delete functionalities:

  • index: Accepts a DocList to index.
  • search: Takes a DocList of batched queries or a single BaseDoc as a single query. It returns either single or multiple results, each with matches and scores attributes sorted by relevance.
  • delete: Accepts a DocList of documents to remove from the index. Only the id attribute is necessary, so make sure to track the indexed IDs if you need to delete documents.
  • update: Accepts a DocList of documents to update in the index. The update operation will replace the indexed document with the same index with the attributes and payload from the input documents.

Service endpoint configuration

You can serve vectordb and access it from a client with the following parameters:

  • protocol: The serving protocol. It can be gRPC, HTTP, websocket or a combination of them, provided as a list. Default is gRPC.
  • port: The service access port. Can be a list of ports for each provided protocol. Default is 8081.
  • workspace: The path where the VectorDB persists required data. Default is '.' (current directory).

Scaling your DB

You can set two scaling parameters when serving or deploying your Vector Databases with vectordb:

  • Shards: The number of data shards. This improves latency, as vectordb ensures Documents are indexed in only one of the shards. Search requests are sent to all shards and results are merged.
  • Replicas: The number of DB replicas. vectordb uses the RAFT algorithm to sync the index between replicas of each shard. This increases service availability and search throughput, as multiple replicas can respond in parallel to more search requests while allowing CRUD operations. Note: In JCloud deployments, the number of replicas is set to 1. We're working on enabling replication in the cloud.

Vector search configuration

Here are the parameters for each VectorDB type:

InMemoryExactNNVectorDB

This database performs exhaustive search on embeddings and has limited configuration settings:

  • workspace: The folder where required data is persisted.
InMemoryExactNNVectorDB[MyDoc](workspace='./vectordb')
InMemoryExactNNVectorDB[MyDoc].serve(workspace='./vectordb')

HNSWVectorDB

This database employs the HNSW (Hierarchical Navigable Small World) algorithm from HNSWLib for Approximate Nearest Neighbor search. It provides several configuration options:

  • workspace: Specifies the directory where required data is stored and persisted.

Additionally, HNSWVectorDB offers a set of configurations that allow tuning the performance and accuracy of the Nearest Neighbor search algorithm. Detailed descriptions of these configurations can be found in the HNSWLib README:

  • space: Specifies the similarity metric used for the space (options are "l2", "ip", or "cosine"). The default is "l2".
  • max_elements: Sets the initial capacity of the index, which can be increased dynamically. The default is 1024.
  • ef_construction: This parameter controls the speed/accuracy trade-off during index construction. The default is 200.
  • ef: This parameter controls the query time/accuracy trade-off. The default is 10.
  • M: This parameter defines the maximum number of outgoing connections in the graph. The default is 16.
  • allow_replace_deleted: If set to True, this allows replacement of deleted elements with newly added ones. The default is False.
  • num_threads: This sets the default number of threads to be used during index and search operations. The default is 1.

Command line interface

vectordb includes a simple CLI for serving and deploying your database:

Description Command
Serve your DB locally vectordb serve --db example:db
Deploy your DB on Jina AI Cloud vectordb deploy --db example:db

Features

  • User-friendly Interface: With vectordb, simplicity is key. Its intuitive interface is designed to accommodate users across varying levels of expertise.

  • Minimalistic Design: vectordb packs all the essentials, with no unnecessary complexity. It ensures a seamless transition from local to server and cloud deployment.

  • Full CRUD Support: From indexing and searching to updating and deleting, vectordb covers the entire spectrum of CRUD operations.

  • DB as a Service: Harness the power of gRPC, HTTP, and Websocket protocols with vectordb. It enables you to serve your databases and conduct insertion or searching operations efficiently.

  • Scalability: Experience the raw power of vectordb's deployment capabilities, including robust scalability features like sharding and replication. Improve your service latency with sharding, while replication enhances availability and throughput.

  • Cloud Deployment: Deploying your service in the cloud is a breeze with Jina AI Cloud. More deployment options are coming soon!

  • Serverless Capability: vectordb can be deployed in a serverless mode in the cloud, ensuring optimal resource utilization and data availability as per your needs.

  • Multiple ANN Algorithms: vectordb offers diverse implementations of Approximate Nearest Neighbors (ANN) algorithms. Here are the current offerings, with more integrations on the horizon:

    • InMemoryExactNNVectorDB (Exact NN Search): Implements Simple Nearest Neighbor Algorithm.
    • HNSWVectorDB (based on HNSW): Utilizes HNSWLib

Roadmap

The future of Vector Database looks bright, and we have ambitious plans! Here's a sneak peek into the features we're currently developing:

  • More ANN Search Algorithms: Our goal is to support an even wider range of ANN search algorithms.
  • Enhanced Filtering Capabilities: We're working on enhancing our ANN Search solutions to support advanced filtering.
  • Customizability: We aim to make vectordb highly customizable, allowing Python developers to tailor its behavior to their specific needs with ease.
  • Expanding Serverless Capacity: We're striving to enhance the serverless capacity of vectordb in the cloud. While we currently support scaling between 0 and 1 replica, our goal is to extend this to 0 to N replicas.
  • Expanded Deployment Options: We're actively working on facilitating the deployment of vectordb across various cloud platforms, with a broad range of options.

Need help with vectordb? Interested in using it but require certain features to meet your unique needs? Don't hesitate to reach out to us. Join our Discord community to chat with us and other community members.

Contributing

The VectorDB project is backed by Jina AI and licensed under Apache-2.0. Contributions from the community are greatly appreciated! If you have an idea for a new feature or an improvement, we would love to hear from you. We're always looking for ways to make vectordb more user-friendly and effective.

More Repositories

1

jina

☁️ Build multimodal AI applications with cloud-native stack
Python
20,716
star
2

clip-as-service

🏄 Scalable embedding, reasoning, ranking for images and sentences with CLIP
Python
12,150
star
3

reader

Convert any URL to an LLM-friendly input with a simple prefix https://r.jina.ai/
TypeScript
6,640
star
4

dalle-flow

🌊 A Human-in-the-Loop workflow for creating HD images from text
Python
2,831
star
5

dev-gpt

Your Virtual Development Team
Python
1,756
star
6

langchain-serve

⚡ Langchain apps in production using Jina & FastAPI
Python
1,601
star
7

finetuner

🎯 Task-oriented embedding tuning for BERT, CLIP, etc.
Python
1,455
star
8

thinkgpt

Agent techniques to augment your LLM and push it beyong its limits
Python
1,402
star
9

auto-gpt-web

Set Your Goals, AI Achieves Them.
TypeScript
749
star
10

agentchain

Chain together LLMs for reasoning & orchestrate multiple large models for accomplishing complex tasks
Python
583
star
11

docarray

The data structure for unstructured data
Python
522
star
12

jcloud

Simplify deploying and managing Jina projects on Jina Cloud
Python
294
star
13

jina-video-chat

Python
266
star
14

jinabox.js

A lightweight, customizable omnibox in Javascript, for use with a Jina backend.
JavaScript
219
star
15

annlite

⚡ A fast embedded library for approximate nearest neighbor search
Python
216
star
16

rungpt

An open-source cloud-native of large multi-modal models (LMMs) serving framework.
Python
147
star
17

fastapi-serve

FastAPI to the Cloud, Batteries Included! ☁️🔋🚀
Python
139
star
18

jina-hub

An open-registry for hosting Jina executors via container images
Python
103
star
19

dashboard

Interactive UI for analyzing Jina logs, designing Flows and viewing Hub images
TypeScript
100
star
20

GoldRetriever

Create and host retrieval plugins for ChatGPT in one click
Python
63
star
21

jinaai-py

Python
48
star
22

example-multimodal-fashion-search

Input text or image, get back matching image fashion results, using Jina, DocArray, and CLIP
Python
45
star
23

streamlit-jina

Streamlit component for Jina neural search
Python
37
star
24

docs

Jina V1 Official Documentation. For the latest one, please check out https://docs.jina.ai
HTML
35
star
25

jinaai-js

TypeScript
28
star
26

executors

internal-only
Python
28
star
27

jerboa

LLM finetuning
Python
27
star
28

jina-ai.github.io

Homepage of Jina AI Limited
HTML
27
star
29

example-meme-search

Meme search engine built with Jina neural search framework. Search with captions or image files to find matching memes.
Python
23
star
30

example-app-store

App store search example, using Jina as backend and Streamlit as frontend
Python
21
star
31

docsQA-ui

Web UI for docsQA. Main branch: https://jina-docqa-ui.netlify.app/
TypeScript
20
star
32

example-speech-to-image

An example of building a speech to image generation pipeline with Jina, Whisper and StableDiffusion
Python
20
star
33

workshops

Jupyter Notebook
19
star
34

jina-hubble-sdk

Python API for authentication, resource management with Hubble
Python
19
star
35

product-recommendation-redis-docarray

Python
18
star
36

career

Find out job opportunities at Jina AI
17
star
37

executor-3d-encoder

An executor that wraps 3D mesh models and encodes 3D content documents to d-dimension vector.
Python
16
star
38

client-go

Golang Client for Jina (https://github.com/jina-ai/jina)
Go
16
star
39

benchmark

Benchmark environment and results of different versions of Jina.
Python
14
star
40

action-hub-builder

Simple interface for building & validating Jina Hub executors.
Python
12
star
41

inference-client

Python
12
star
42

executor-hnsw-postgres

A production-ready, scalable Indexer for the Jina neural search framework, based on HNSW and PSQL
Python
12
star
43

now

Python
11
star
44

cookiecutter-jina

Cookiecutter template for a Jina project
Python
10
star
45

simple-jina-examples

Python
9
star
46

executor-simpleindexer

Simple Indexer
Python
9
star
47

executor-clip-encoder

Encoder that embeds documents using either the CLIP vision encoder or the CLIP text encoder, depending on the content type of the document.
Python
9
star
48

cloud-ops

Python
8
star
49

good-first-issues

Issues that don't fit under Jina's other repos!
8
star
50

api

API schema of Jina command line interface exposed as JSON and YAML files.
HTML
8
star
51

inference-client-js

TypeScript
7
star
52

executor-text-transformers-dprreader-ranker

DPRReaderRanker
Python
7
star
53

executor-video-loader

Python
7
star
54

executor-image-clip-encoder

CLIPImageEncoder is an image encoder that wraps the image embedding functionality using the CLIP
Python
7
star
55

.github

This repository stores github actions templates as described https://docs.github.com/en/actions/learn-github-actions/sharing-workflows-with-your-organization
7
star
56

GSoC

Google Summer of Code
7
star
57

example-wikipedia-recommendation

An example of graph embeddings for wikipedia page recommendations
Jupyter Notebook
6
star
58

executor-U100KIndexer

An Indexer that works out-of-the-box when you have less than 100K stored Documents
Python
6
star
59

devrel-heartmaker

Heart mosaics of your GitHub contributors
Python
6
star
60

executor-text-transformers-torch-encoder

**TransformerTorchEncoder** wraps the torch-version of transformers from huggingface. It encodes text data into dense vectors.
Python
6
star
61

executor-cases

Summarize all Executor patterns for Hubble
Python
5
star
62

executor-normalizer

Jina executor package normalizer
Python
5
star
63

auth

deprecated, use `jina-hubble-sdk`
Python
5
star
64

jina-commons

A collection of shared function for Jina Executor
Python
5
star
65

tutorial-notebooks

Jupyter Notebook
5
star
66

jina-paddle-hackathon

极纳 x 百度飞桨 黑客马拉松
Python
5
star
67

executor-image-preprocessor

An executor that performs standard pre-processing and normalization on images.
Python
5
star
68

jina-hackathon

Support repo for Jina X Hackathon - Sep 2020
5
star
69

executor-featurehasher

FeatureHasher
Python
4
star
70

jina-sagemaker

Jina Embedding Models on AWS SageMaker
Jupyter Notebook
4
star
71

stress-test

A collection of stress tests of Jina infrastructure
Python
4
star
72

executor-image-clip-classifier

Python
4
star
73

executor-text-transformerqa

**TransformerQAExecutor* wraps a question-answering model from huggingface and return relevant answers given questions and contexts/paragraphs.
Python
4
star
74

executor-faissindexer

A similarity search indexer based on Faiss. https://hub.jina.ai/executor/8gsd0tts
Python
4
star
75

hub-integration

Integration test for hub
Python
4
star
76

example-audio-search

Python
3
star
77

example-video-qa

This is an example of building a video QA with jina
TypeScript
3
star
78

jinad

Management of Jina on remote
Python
3
star
79

executor-indexers

Indexer Executors for Jina
Python
3
star
80

executor-text-dpr-encoder

Encode text into embeddings using the DPR model.
Python
3
star
81

legacy-examples

Unmaintained examples for Jina
Python
3
star
82

executor-clip-image

Executor for the pre-trained clip model. https://openai.com/blog/clip/
Python
3
star
83

executor-weaviate-indexer

Python
3
star
84

executor-doc2query

Python
3
star
85

executor-image-paddle-encoder

Python
3
star
86

jupyter-notebooks

Jupyter Notebook
3
star
87

executor-evaluator-ranking

Python
3
star
88

executor-yolov5

Python
3
star
89

executor-lightgbm-ranker

Python
3
star
90

terraform-jina-jinad-aws

Module for deploying JinaD on AWS
HCL
3
star
91

encoder-image-torch

The ImageTorchEncoder encodes Document content from a ndarray to an d-dimensional vector.
Python
3
star
92

example-odqa

Roff
2
star
93

executor-text-clip-encoder

Encode text into embeddings using the CLIP model.
Python
2
star
94

jina-ui

Monorepo for JinaJS and frontend projects
TypeScript
2
star
95

executor-audio-clip-encoder

Wraps the AudioCLIP model for generating embeddings for audio data for the Jina framework
Python
2
star
96

executor-matchmerger

**MatchMerger** Merges the results of shards by appending all matches.
Python
2
star
97

executor-image-niireader

Python
2
star
98

executor-image-normalizer

Executor that reads, resizes, crops and normalizes images.
Python
2
star
99

executor-vgg-audio-encoder

Python
2
star
100

executor-image-hasher

An executor to encode images using comparable hashing techniques. Useful for duplicate detection
Python
2
star