redis-developer/redis-arXiv-search redis-developer/redis-arXiv-search

  • Stars
    star
    135
  • Rank 267,796 (Top 6 %)
  • Language
    TypeScript
  • License
    BSD 3-Clause "New...
  • Created about 2 years ago
  • Updated about 2 months ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
redis-developer/redis-arXiv-search
github

redis-developer

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Vector search demo with the arXiv paper dataset, RedisVL, HuggingFace, OpenAI, Cohere, FastAPI, React, and Redis.


Hosted Demo    Code    Blog Post    Redis Vector Search Documentation   


🔎 Redis arXiv Search

This repository is the official codebase for the arxiv paper search app hosted at: https://docsearch.redisvl.com

Redis is a highly performant, production-ready vector database, which can be used for many types of applications. Here we showcase Redis vector search applied to a document retrieval use case. Read more about AI-powered search in the technical blog post published by our partners, Data Science Dojo.

Dataset

The arXiv papers dataset was sourced from the the following Kaggle link. arXiv is commonly used for scientific research in a variety of fields. Exposing a semantic search layer enables natural human language to be used to discover relevant papers.

Demo

Application

This app was built as a Single Page Application (SPA) with the following components:

Some inspiration was taken from this Cookiecutter project and turned into a SPA application instead of a separate front-end server approach.

Embedding Providers

Embeddings represent the semantic properies of the raw text and enable vector similarity search. This applications supports HuggingFace, OpenAI, and Cohere embeddings out of the box.

Provider Embedding Model Required?
HuggingFace sentence-transformers/all-mpnet-base-v2 Yes
OpenAI text-embedding-ada-002 Yes
Cohere embed-multilingual-v3.0 Yes

Interested in a different embedding provider? Feel free to open a PR and make a suggested addition.

Want to use a different model than the one listed? Set the following environment variables in your .env file (see below) to change:

  • SENTENCE_TRANSFORMER_MODEL
  • OPENAI_EMBEDDING_MODEL
  • COHERE_EMBEDDING_MODEL

🚀 Running the App

  1. Before running the app, install Docker Desktop.
  2. Clone (and optionally fork) this Github repo to your machine. 
    $ git clone https://github.com/RedisVentures/redis-arXiv-search.git
  3. Make a copy of the .env.template file: 
    $ cd redis-arXiv-search/
$ cp .env.template .env
  4. Decide which Redis you plan to use, choose one of the methods below
    • Redis Stack runs Redis as a local docker container.
    • Redis Cloud will manage a Redis database on your behalf in the cloud.

Redis Stack Docker (Local)

Using Redis Stack locally doesn't require any additional steps. However, it will consume more resources on your machine and have performance limitations.

Use the provided docker-compose file for running the application locally:

$ docker compose -f docker-local-redis.yml up

Redis Cloud

  1. Get a FREE Redis Cloud Database. Make sure to include the Search module.

  2. Add the REDIS_HOST, REDIS_PASSWORD, and REDIS_PORT environment variables to your .env file.

  3. Run the App:

    $ docker compose -f docker-cloud-redis.yml up

Customizing (optional)

  • You can use the provided Jupyter Notebook in the data/ directory to create paper embeddings and metadata. The output JSON files will end up stored in the data/ directory and used when creating your own container.
  • Use the ./build.sh script to build your own docker image based on the application source code and dataset changes.
  • If you want to use K8s instead of Docker Compose, we have some resources to help you get started.

React Dev Environment

It's typically easier to build front end in an interactive environment, testing changes in realtime.

  1. Deploy the app using steps above.
  2. Install packages (you may need to use npm to install yarn) 
    $ cd frontend/
$ yarn install --no-optional
  3. Use yarn to serve the application from your machine 
    $ yarn start
  4. Navigate to http://localhost:3000 in a browser.

All changes to your frontend code will be reflected in your display in semi realtime.

Troubleshooting

Every once and a while you need to clear out some Docker cached artifacts. Run docker system prune, restart Docker Desktop, and try again.

This project is maintained by Redis on a good faith basis. Please, open an issue here on GitHub and we will try to be responsive to these.

More Repositories

1

ArXivChatGuru

Use ArXiv ChatGuru to talk to research papers. This app uses LangChain, OpenAI, Streamlit, and Redis as a vector database/semantic cache.
Python
512
star
2

redis-product-search

Visual and semantic vector similarity with Redis Stack, FastAPI, PyTorch and Huggingface.
Python
153
star
3

redis-ai-resources

✨ A curated list of awesome community resources, integrations, and examples of Redis in the AI ecosystem.
Jupyter Notebook
144
star
4

LLM-Document-Chat

Using LlamaIndex, Redis, and OpenAI to chat with PDF documents. Supplementary material for blog post on Microsoft Developer Blog
Jupyter Notebook
106
star
5

basic-redis-chat-app-demo-nodejs

A basic Redis chat application demo written in Nodejs
JavaScript
84
star
6

redis-datasets

A Curated List of Sample Redis Datasets
Python
79
star
7

agentic-rag

Jupyter Notebook
69
star
8

fastapi-redis-tutorial

Using Redis with FastAPI
Python
68
star
9

redis-stack-stocks

This project demonstrates how you can use the multi-model capabilities of Redis to create a real-time stock watchlist application.
TypeScript
66
star
10

redis-microservices-demo

Microservice application with various Redis use-cases with RediSearch, RedisGraph and Streams. The data are synchronize between MySQL and Redis using Debezium as a CDC engine
Java
56
star
11

redis-nvidia-recsys

Three examples of recommendation system pipelines with NVIDIA Merlin and Redis
PureBasic
56
star
12

redis-om-python-flask-skeleton-app

A starter application for performing CRUD type operations with Redis OM Python and the Flask Microframework
Python
48
star
13

redis-developer.github.io

The Home of Redis Developers
TypeScript
47
star
14

basic-redis-chat-app-demo-python

Redis chat demo app written in Python
JavaScript
42
star
15

redis-microservices-ecommerce-solutions

TypeScript
39
star
16

basic-redis-chat-app-demo-dotnet

Basic Redis Chat Demo App dot net
JavaScript
32
star
17

gcp-redis-llm-stack

Reference architecture for LLM-based applications on Google Cloud Platform with Redis Enterprise as a high-performance data layer.
Jupyter Notebook
25
star
18

basic-redis-chat-demo-go

A Basic redis chat demo app written in Go language
JavaScript
23
star
19

testcontainers-redis

Testcontainers module for Redis
Java
23
star
20

redis-om-fastapi

Examples of using Redis OM Python with FastAPI
Python
22
star
21

basic-redis-shopping-chart-nodejs

Basic Redis Shopping cart demo using NodeJS
Vue
21
star
22

finding-bigfoot-with-semantic-search

Sample application showing how to use semantic search with Redis and JavaScript.
JavaScript
20
star
23

basic-caching-demo-nodejs

This is an example demo app that shows how Redis caching works using Nodejs
Vue
19
star
24

basic-redis-chat-app-demo-java

A sample redis chat demo app written in Java
JavaScript
18
star
25

express-redis-om-workshop

JavaScript
18
star
26

LLM-Recommender

Use OpenAI, Redis, and streamlit to recommend hotels using Large Language Models
Python
18
star
27

redis-aqi-monitor

A demonstration of Redis Stack Streams, Timeseries, JSON, and Search using a Raspberry Pico W and PMS5003 sensor
Python
16
star
28

redis-chat-challenge

A Hackathon challenge to build a real-time chat application powered by Redis.
JavaScript
16
star
29

redi2read

Java
15
star
30

nosql-data-modeling-patterns

Contains sample code for modeling data using Redis and NoSQL
Python
15
star
31

redis-hacker-news-demo

The demo is a Hacker News clone and uses RedisJSON for storing the data and RediSearch for searching.
JavaScript
15
star
32

redis-om-dotnet-skeleton-app

skeleton app for redis-om-dotnet
C#
14
star
33

beats-by-redis

A demonstration of Redis Stack with Vue, Express, and Node.js
JavaScript
14
star
34

brewdis

Retail demo
Java
12
star
35

introducing-the-geosearch-command

Demo application using a Bulma / Leaflet JS / JavaScript front end and Python / Flask backend to demonstrate the GEOSEARCH command added in Redis 6.2.
HTML
12
star
36

redis-om-spring-react-todomvc

Tutorial: Redis OM Spring + React TodoMVC
JavaScript
12
star
37

redis-bike-co

Redis Bike Company Example Application
Jupyter Notebook
12
star
38

basic-redis-leaderboard-demo-nodejs

Basic Redis Leaderboard demo written in Nodejs
JavaScript
10
star
39

redis-caching-api-responses

JavaScript
10
star
40

redis-om-spring-skeleton-app

Redis OM Spring Skeleton App
Java
10
star
41

basic-redis-leaderboard-demo-python

A Basic Redis leaderboard demo app written in Python
Python
10
star
42

basic-analytics-dashboard-redis-bitmaps-nodejs

This repo shows a basic analytics dashboard app that uses Redis Bitmap written in NodeJS (JavaScript)
Vue
9
star
43

nredi2read-preview

C#
9
star
44

aws-redis-cloud-demo

A Demo App using Redis Enterprise Cloud with AWS Services
Vue
9
star
45

basic-redis-leaderboard-demo-go

A basic Redis leaderboard demo app written in Go
Go
9
star
46

redis-om-python-search-demo

Demo using the Redis OM Python client and RediSearch
Python
8
star
47

financial-vss

Notebooks demonstrating vector search & RAG design patterns with Redis Python clients.
Jupyter Notebook
8
star
48

dot-net-redis-intro

C#
7
star
49

redis-om-node-tutorial

Code sample for tutorial on Redis OM for Node.js
JavaScript
7
star
50

redis-kaboom-rpg

Redis RPG game example with Kaboom.JS.
JavaScript
7
star
51

keyspace-notifications-node-redis

Keyspace Notifications demo with Node Redis 4
Python
7
star
52

hitc-dotnet-redisearch-demo

HITC video accompanying repository
C#
7
star
53

basic-redis-leaderboard-demo-dotnet

Basic leaderboard app written in C#.net
C#
7
star
54

basic-rate-limiting-demo-python

This repo shows a basic rate limiting demo in Python and Django
Python
7
star
55

basic-redis-rate-limiting-demo-nodejs

This is a basic Redis Rate Limiting demo written in NodeJS
JavaScript
7
star
56

redis-caching-slow-database-queries

Simple demonstration of caching database queries with Redis.
JavaScript
7
star
57

hackathon-docs

6
star
58

basic-redis-rate-limiting-demo-go-lang

A basic redis rate limiting demo written in Go language
Go
6
star
59

redis-streams-hotel-jobs

Redis Streams example with Python and Node.js
JavaScript
6
star
60

ebook-speed-mern-backend

TypeScript
6
star
61

haunted-places-data-service

Simple Node.js data service that exposes haunted places using Apollo GraphQL, Redi, and RediSearch.
JavaScript
6
star
62

introducing-redis-talk

Resources for Simon's Introducing Redis Talk.
JavaScript
6
star
63

video-qa-semantic-vector-caching

This repository is a Video Q&A site that uses LangChain, OpenAI, Google Gemini, and Redis to answer questions using videos for RAG.
TypeScript
6
star
64

redis-real-time-inventory-solutions

TypeScript
5
star
65

redismod-docker-compose

Contains a docker-compose.yml using redismod : a Docker image with selected Redis Labs modules
5
star
66

redis-om-autocomplete-demo

Redis OM Spring Autocomplete/Autosuggest Demo
Java
5
star
67

sliding-window-rate-limiter-aspnet

Sliding Window Rate Limiter Example for ASP.NET Core
C#
5
star
68

fixed-window-rate-limiter

Redis Fixed Window Rate Limiter with Spring Boot
Java
5
star
69

devcember

Redis Devcember: https://developer.redis.com/devcember/
JavaScript
5
star
70

basic-redis-rate-limiting-demo-csharp-dot-net

Basic redis rate limiting demo written in C# .net
C#
5
star
71

redis-om-node-skeleton-app

JavaScript
4
star
72

redisbloom-moby-dick

Experiments with RedisBloom and the text from Moby Dick
Python
4
star
73

basic-redisgraph-movie-demo-app-nodejs

Basic RedisGraph movie demo app written in NodeJS
JavaScript
4
star
74

acre-terraform

Templates to get started with the 'azurerm_redis_enterprise_cluster' resource.
HCL
4
star
75

demo-movie-app-redisearch-nodejs

A movie demo app that shows redisearch capabilities
Vue
4
star
76

banking-on-redis

An app demonstrating real time querying of JSON within a banking application
JavaScript
4
star
77

sql-cache-invalidation-debezium

This project shows how to use Debezium to update Redis from PostgreSQL transactions
Java
4
star
78

mlh-prize-potato

The inspiration repo for those who win (or don't!) MLH hackathons where Redis is a sponsor. This repo is aimed at getting you started with a Libre Potato mini computer and using Redis.
4
star
79

spring-ai-redis-demo

JavaScript
4
star
80

micropython-redis-boilerplate

A boilerplate MicroPython script for getting started with Redis on microcontrollers (tested on Raspberry Pi Pico W)
Python
3
star
81

fhir-search

Redis Document (JSON + Search) examples against FHIR data
JavaScript
3
star
82

polygon-demo

Polygon search in Redis
Python
3
star
83

basic-caching-redis-demo-go-lang

Basic caching redis demo app written in GO language
Go
3
star
84

ebook-speed-mern-frontend

TypeScript
3
star
85

doc-crawler

Web crawler with search indexing
JavaScript
3
star
86

nextjs-redis-netlify

This app is a minimal starter template for building a site with Next.js, TailwindCSS, Redis, and deploying to Netlify
JavaScript
3
star
87

redis-vector-nodejs-solutions

TypeScript
3
star
88

redis-polygon-search-trains-demo

Redis Stack Polygon search demo with train data, written in Node.js.
JavaScript
3
star
89

basic-caching-demo-python

This project shows a basic caching demo in Python
Python
2
star
90

the-redis-marketplace-app-frontend

The Redis Marketplace app
JavaScript
2
star
91

speechdis

A Tool that transcribe audio from Podcast using RedisAI and RediSearch
Jupyter Notebook
2
star
92

hentry-server

early prototype, please do not test your hacking skills here :)
TypeScript
2
star
93

RedisBirdTracker

A demonstration of JSON storage and query capabilities using bird sightings as a data set
Python
2
star
94

gitpod-flask-redis

A quick start to get up and running with Flask and Redis application development using GitPod
HTML
2
star
95

redis-developer-old

CSS
2
star
96

node-redis-json-demo

Node Redis 4 RedisJSON demo.
JavaScript
2
star
97

basic-redis-rate-limiting-demo-ruby

A basic Redis rate limiting demo written in Ruby
Ruby
2
star
98

basic-caching-demo-csharpdotnet

Basic caching demo written in C# .net
C#
2
star
99

gitpod-express-redis

A Node.js Express with Redis starter application that uses Gitpod
JavaScript
2
star
100

basic-rate-limiting-demo-java

This project show how to implement a basic rate limiting using Redis in Java
JavaScript
2
star