redis-developer/redis-arXiv-search redis-developer/redis-arXiv-search

  • Stars
    star
    135
  • Rank 269,297 (Top 6 %)
  • Language
    TypeScript
  • License
    BSD 3-Clause "New...
  • Created about 2 years ago
  • Updated 3 months ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
redis-developer/redis-arXiv-search
github

redis-developer

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Vector search demo with the arXiv paper dataset, RedisVL, HuggingFace, OpenAI, Cohere, FastAPI, React, and Redis.


Hosted Demo    Code    Blog Post    Redis Vector Search Documentation   


🔎 Redis arXiv Search

This repository is the official codebase for the arxiv paper search app hosted at: https://docsearch.redisvl.com

Redis is a highly performant, production-ready vector database, which can be used for many types of applications. Here we showcase Redis vector search applied to a document retrieval use case. Read more about AI-powered search in the technical blog post published by our partners, Data Science Dojo.

Dataset

The arXiv papers dataset was sourced from the the following Kaggle link. arXiv is commonly used for scientific research in a variety of fields. Exposing a semantic search layer enables natural human language to be used to discover relevant papers.

Demo

Application

This app was built as a Single Page Application (SPA) with the following components:

Some inspiration was taken from this Cookiecutter project and turned into a SPA application instead of a separate front-end server approach.

Embedding Providers

Embeddings represent the semantic properies of the raw text and enable vector similarity search. This applications supports HuggingFace, OpenAI, and Cohere embeddings out of the box.

Provider Embedding Model Required?
HuggingFace sentence-transformers/all-mpnet-base-v2 Yes
OpenAI text-embedding-ada-002 Yes
Cohere embed-multilingual-v3.0 Yes

Interested in a different embedding provider? Feel free to open a PR and make a suggested addition.

Want to use a different model than the one listed? Set the following environment variables in your .env file (see below) to change:

  • SENTENCE_TRANSFORMER_MODEL
  • OPENAI_EMBEDDING_MODEL
  • COHERE_EMBEDDING_MODEL

🚀 Running the App

  1. Before running the app, install Docker Desktop.
  2. Clone (and optionally fork) this Github repo to your machine. 
    $ git clone https://github.com/RedisVentures/redis-arXiv-search.git
  3. Make a copy of the .env.template file: 
    $ cd redis-arXiv-search/
$ cp .env.template .env
  4. Decide which Redis you plan to use, choose one of the methods below
    • Redis Stack runs Redis as a local docker container.
    • Redis Cloud will manage a Redis database on your behalf in the cloud.

Redis Stack Docker (Local)

Using Redis Stack locally doesn't require any additional steps. However, it will consume more resources on your machine and have performance limitations.

Use the provided docker-compose file for running the application locally:

$ docker compose -f docker-local-redis.yml up

Redis Cloud

  1. Get a FREE Redis Cloud Database. Make sure to include the Search module.

  2. Add the REDIS_HOST, REDIS_PASSWORD, and REDIS_PORT environment variables to your .env file.

  3. Run the App:

    $ docker compose -f docker-cloud-redis.yml up

Customizing (optional)

  • You can use the provided Jupyter Notebook in the data/ directory to create paper embeddings and metadata. The output JSON files will end up stored in the data/ directory and used when creating your own container.
  • Use the ./build.sh script to build your own docker image based on the application source code and dataset changes.
  • If you want to use K8s instead of Docker Compose, we have some resources to help you get started.

React Dev Environment

It's typically easier to build front end in an interactive environment, testing changes in realtime.

  1. Deploy the app using steps above.
  2. Install packages (you may need to use npm to install yarn) 
    $ cd frontend/
$ yarn install --no-optional
  3. Use yarn to serve the application from your machine 
    $ yarn start
  4. Navigate to http://localhost:3000 in a browser.

All changes to your frontend code will be reflected in your display in semi realtime.

Troubleshooting

Every once and a while you need to clear out some Docker cached artifacts. Run docker system prune, restart Docker Desktop, and try again.

This project is maintained by Redis on a good faith basis. Please, open an issue here on GitHub and we will try to be responsive to these.

More Repositories

1

ArXivChatGuru

Use ArXiv ChatGuru to talk to research papers. This app uses LangChain, OpenAI, Streamlit, and Redis as a vector database/semantic cache.
Python
512
star
2

redis-product-search

Visual and semantic vector similarity with Redis Stack, FastAPI, PyTorch and Huggingface.
Python
153
star
3

redis-ai-resources

✨ A curated list of awesome community resources, integrations, and examples of Redis in the AI ecosystem.
Jupyter Notebook
144
star
4

LLM-Document-Chat

Using LlamaIndex, Redis, and OpenAI to chat with PDF documents. Supplementary material for blog post on Microsoft Developer Blog
Jupyter Notebook
106
star
5

basic-redis-chat-app-demo-nodejs

A basic Redis chat application demo written in Nodejs
JavaScript
84
star
6

redis-datasets

A Curated List of Sample Redis Datasets
Python
79
star
7

agentic-rag

Jupyter Notebook
69
star
8

fastapi-redis-tutorial

Using Redis with FastAPI
Python
68
star
9

redis-stack-stocks

This project demonstrates how you can use the multi-model capabilities of Redis to create a real-time stock watchlist application.
TypeScript
66
star
10

redis-nvidia-recsys

Three examples of recommendation system pipelines with NVIDIA Merlin and Redis
PureBasic
56
star
11

redis-om-python-flask-skeleton-app

A starter application for performing CRUD type operations with Redis OM Python and the Flask Microframework
Python
48
star
12

redis-developer.github.io

The Home of Redis Developers
TypeScript
47
star
13

basic-redis-chat-app-demo-python

Redis chat demo app written in Python
JavaScript
42
star
14

redis-microservices-ecommerce-solutions

TypeScript
39
star
15

basic-redis-chat-app-demo-dotnet

Basic Redis Chat Demo App dot net
JavaScript
32
star
16

gcp-redis-llm-stack

Reference architecture for LLM-based applications on Google Cloud Platform with Redis Enterprise as a high-performance data layer.
Jupyter Notebook
25
star
17

basic-redis-chat-demo-go

A Basic redis chat demo app written in Go language
JavaScript
23
star
18

testcontainers-redis

Testcontainers module for Redis
Java
23
star
19

basic-redis-shopping-chart-nodejs

Basic Redis Shopping cart demo using NodeJS
Vue
22
star
20

redis-om-fastapi

Examples of using Redis OM Python with FastAPI
Python
22
star
21

finding-bigfoot-with-semantic-search

Sample application showing how to use semantic search with Redis and JavaScript.
JavaScript
21
star
22

basic-caching-demo-nodejs

This is an example demo app that shows how Redis caching works using Nodejs
Vue
19
star
23

basic-redis-chat-app-demo-java

A sample redis chat demo app written in Java
JavaScript
18
star
24

express-redis-om-workshop

JavaScript
18
star
25

LLM-Recommender

Use OpenAI, Redis, and streamlit to recommend hotels using Large Language Models
Python
18
star
26

redis-aqi-monitor

A demonstration of Redis Stack Streams, Timeseries, JSON, and Search using a Raspberry Pico W and PMS5003 sensor
Python
16
star
27

redis-chat-challenge

A Hackathon challenge to build a real-time chat application powered by Redis.
JavaScript
16
star
28

redi2read

Java
15
star
29

nosql-data-modeling-patterns

Contains sample code for modeling data using Redis and NoSQL
Python
15
star
30

redis-hacker-news-demo

The demo is a Hacker News clone and uses RedisJSON for storing the data and RediSearch for searching.
JavaScript
15
star
31

redis-om-dotnet-skeleton-app

skeleton app for redis-om-dotnet
C#
14
star
32

beats-by-redis

A demonstration of Redis Stack with Vue, Express, and Node.js
JavaScript
14
star
33

brewdis

Retail demo
Java
12
star
34

introducing-the-geosearch-command

Demo application using a Bulma / Leaflet JS / JavaScript front end and Python / Flask backend to demonstrate the GEOSEARCH command added in Redis 6.2.
HTML
12
star
35

redis-om-spring-react-todomvc

Tutorial: Redis OM Spring + React TodoMVC
JavaScript
12
star
36

redis-bike-co

Redis Bike Company Example Application
Jupyter Notebook
12
star
37

basic-analytics-dashboard-redis-bitmaps-nodejs

This repo shows a basic analytics dashboard app that uses Redis Bitmap written in NodeJS (JavaScript)
Vue
10
star
38

basic-redis-leaderboard-demo-nodejs

Basic Redis Leaderboard demo written in Nodejs
JavaScript
10
star
39

redis-caching-api-responses

JavaScript
10
star
40

redis-om-spring-skeleton-app

Redis OM Spring Skeleton App
Java
10
star
41

basic-redis-leaderboard-demo-python

A Basic Redis leaderboard demo app written in Python
Python
10
star
42

nredi2read-preview

C#
9
star
43

aws-redis-cloud-demo

A Demo App using Redis Enterprise Cloud with AWS Services
Vue
9
star
44

basic-redis-leaderboard-demo-go

A basic Redis leaderboard demo app written in Go
Go
9
star
45

redis-om-python-search-demo

Demo using the Redis OM Python client and RediSearch
Python
8
star
46

financial-vss

Notebooks demonstrating vector search & RAG design patterns with Redis Python clients.
Jupyter Notebook
8
star
47

dot-net-redis-intro

C#
7
star
48

redis-om-node-tutorial

Code sample for tutorial on Redis OM for Node.js
JavaScript
7
star
49

redis-kaboom-rpg

Redis RPG game example with Kaboom.JS.
JavaScript
7
star
50

keyspace-notifications-node-redis

Keyspace Notifications demo with Node Redis 4
Python
7
star
51

hitc-dotnet-redisearch-demo

HITC video accompanying repository
C#
7
star
52

basic-redis-leaderboard-demo-dotnet

Basic leaderboard app written in C#.net
C#
7
star
53

basic-rate-limiting-demo-python

This repo shows a basic rate limiting demo in Python and Django
Python
7
star
54

basic-redis-rate-limiting-demo-nodejs

This is a basic Redis Rate Limiting demo written in NodeJS
JavaScript
7
star
55

redis-caching-slow-database-queries

Simple demonstration of caching database queries with Redis.
JavaScript
7
star
56

hackathon-docs

6
star
57

basic-redis-rate-limiting-demo-go-lang

A basic redis rate limiting demo written in Go language
Go
6
star
58

redis-streams-hotel-jobs

Redis Streams example with Python and Node.js
JavaScript
6
star
59

ebook-speed-mern-backend

TypeScript
6
star
60

haunted-places-data-service

Simple Node.js data service that exposes haunted places using Apollo GraphQL, Redi, and RediSearch.
JavaScript
6
star
61

introducing-redis-talk

Resources for Simon's Introducing Redis Talk.
JavaScript
6
star
62

video-qa-semantic-vector-caching

This repository is a Video Q&A site that uses LangChain, OpenAI, Google Gemini, and Redis to answer questions using videos for RAG.
TypeScript
6
star
63

redis-real-time-inventory-solutions

TypeScript
5
star
64

redismod-docker-compose

Contains a docker-compose.yml using redismod : a Docker image with selected Redis Labs modules
5
star
65

redis-om-autocomplete-demo

Redis OM Spring Autocomplete/Autosuggest Demo
Java
5
star
66

sliding-window-rate-limiter-aspnet

Sliding Window Rate Limiter Example for ASP.NET Core
C#
5
star
67

fixed-window-rate-limiter

Redis Fixed Window Rate Limiter with Spring Boot
Java
5
star
68

devcember

Redis Devcember: https://developer.redis.com/devcember/
JavaScript
5
star
69

basic-redis-rate-limiting-demo-csharp-dot-net

Basic redis rate limiting demo written in C# .net
C#
5
star
70

redis-om-node-skeleton-app

JavaScript
4
star
71

redisbloom-moby-dick

Experiments with RedisBloom and the text from Moby Dick
Python
4
star
72

basic-redisgraph-movie-demo-app-nodejs

Basic RedisGraph movie demo app written in NodeJS
JavaScript
4
star
73

acre-terraform

Templates to get started with the 'azurerm_redis_enterprise_cluster' resource.
HCL
4
star
74

demo-movie-app-redisearch-nodejs

A movie demo app that shows redisearch capabilities
Vue
4
star
75

banking-on-redis

An app demonstrating real time querying of JSON within a banking application
JavaScript
4
star
76

sql-cache-invalidation-debezium

This project shows how to use Debezium to update Redis from PostgreSQL transactions
Java
4
star
77

mlh-prize-potato

The inspiration repo for those who win (or don't!) MLH hackathons where Redis is a sponsor. This repo is aimed at getting you started with a Libre Potato mini computer and using Redis.
4
star
78

spring-ai-redis-demo

JavaScript
4
star
79

micropython-redis-boilerplate

A boilerplate MicroPython script for getting started with Redis on microcontrollers (tested on Raspberry Pi Pico W)
Python
3
star
80

fhir-search

Redis Document (JSON + Search) examples against FHIR data
JavaScript
3
star
81

polygon-demo

Polygon search in Redis
Python
3
star
82

ebook-speed-mern-frontend

TypeScript
3
star
83

doc-crawler

Web crawler with search indexing
JavaScript
3
star
84

nextjs-redis-netlify

This app is a minimal starter template for building a site with Next.js, TailwindCSS, Redis, and deploying to Netlify
JavaScript
3
star
85

redis-vector-nodejs-solutions

TypeScript
3
star
86

redis-polygon-search-trains-demo

Redis Stack Polygon search demo with train data, written in Node.js.
JavaScript
3
star
87

basic-caching-redis-demo-go-lang

Basic caching redis demo app written in GO language
Go
3
star
88

basic-caching-demo-python

This project shows a basic caching demo in Python
Python
2
star
89

the-redis-marketplace-app-frontend

The Redis Marketplace app
JavaScript
2
star
90

speechdis

A Tool that transcribe audio from Podcast using RedisAI and RediSearch
Jupyter Notebook
2
star
91

hentry-server

early prototype, please do not test your hacking skills here :)
TypeScript
2
star
92

RedisBirdTracker

A demonstration of JSON storage and query capabilities using bird sightings as a data set
Python
2
star
93

gitpod-flask-redis

A quick start to get up and running with Flask and Redis application development using GitPod
HTML
2
star
94

redis-developer-old

CSS
2
star
95

node-redis-json-demo

Node Redis 4 RedisJSON demo.
JavaScript
2
star
96

basic-redis-rate-limiting-demo-ruby

A basic Redis rate limiting demo written in Ruby
Ruby
2
star
97

basic-caching-demo-csharpdotnet

Basic caching demo written in C# .net
C#
2
star
98

gitpod-express-redis

A Node.js Express with Redis starter application that uses Gitpod
JavaScript
2
star
99

basic-rate-limiting-demo-java

This project show how to implement a basic rate limiting using Redis in Java
JavaScript
2
star
100

redis-streams-with-dotnet

Example applications use Redis Streams with various .NET libraries
C#
2
star