⚡ LangChain Apps on Production with Jina & FastAPI 🚀

Jina is an open-source framework for building scalable multi modal AI apps on Production. LangChain is another open-source framework for building applications powered by LLMs.

langchain-serve helps you deploy your LangChain apps on Jina AI Cloud in a matter of seconds. You can benefit from the scalability and serverless architecture of the cloud without sacrificing the ease and convenience of local development. And if you prefer, you can also deploy your LangChain apps on your own infrastructure to ensure data privacy. With langchain-serve, you can craft REST/Websocket APIs, spin up LLM-powered conversational Slack bots, or wrap your LangChain apps into FastAPI packages on cloud or on-premises.

Give us a ⭐ and tell us what more you'd like to see!

☁️ LLM Apps as-a-service

langchain-serve currently wraps following apps as a service to be deployed on Jina AI Cloud with one command.

🔮 AutoGPT-as-a-service

AutoGPT is an "AI agent" that given a goal in natural language, will attempt to achieve it by breaking it into sub-tasks and using the internet and other tools in an automatic loop.

🧠 Babyagi-as-a-service

Babyagi is a task-driven autonomous agent that uses LLMs to create, prioritize, and execute tasks. It is a general-purpose AI agent that can be used to automate a wide variety of tasks.

🐼 pandas-ai-as-a-service

pandas-ai integrates LLM capabilities into Pandas, to make dataframes conversational in Python code. Thanks to langchain-serve, we can now expose pandas-ai APIs on Jina AI Cloud in just a matter of seconds.

💬 Question Answer Bot on PDFs

pdfqna is a simple question answering bot that uses LLMs to answer questions on PDF documents, showcasing the how easy it is to integrate langchain apps on Jina AI Cloud.

💪 Features

🎉 LLM Apps on production

• 👉 Define your API using @serving decorator
• 👉 Build, deploy & distribute Slack bots using @slackbot decorator
• 👉 Bring your own FastAPI app

🔥 Secure, Scalable, Serverless, Streaming REST/Websocket APIs on Jina AI Cloud.

• 🌎 Globally available REST/Websocket APIs with automatic TLS certs.
• 🌊 Stream LLM interactions in real-time with Websockets.
• 👥 Enable human in the loop for your agents.
• 💬 Build, deploy & distribute Slack bots built with langchain.
• 🔑 Protect your APIs with API authorization using Bearer tokens.
• 📄 Swagger UI, and OpenAPI spec included with your APIs.
• ⚡️ Serverless, autoscaling apps that scales automatically with your traffic.
• 📁 Persistent storage (EFS) mounted on your app for your data.
• 📊 Builtin logging, monitoring, and traces for your APIs.
• 🤖 No need to change your code to manage APIs, or manage dockerfiles, or worry about infrastructure!

🏠 Self-host LLM Apps with Docker Compose or Kubernetes

• 🚀 Export your apps as Kubernetes or Docker Compose YAMLs with single command.
• 👉 lc-serve export app --kind <kubernetes/docker-compose> --path .
• 📦 Deploy your app on your own internal infrastructure with your own security policies.
• 📞 Talk to us if you need all the features of Jina AI Cloud on your own infrastructure.

🧰 Usage

Let's first install langchain-serve using pip.

pip install langchain-serve

🔄 REST APIs using @serving decorator

👉 Let's go through a step-by-step guide to build, deploy and use a REST API using @serving decorator.

🤖💬 Build, Deploy & Distribute Slack bots built with LangChain

langchain-serve exposes a @slackbot decorator to quickly build, deploy & distribute LLM-powered Slack bots without worrying about the infrastructure. It provides a simple interface to any langchain app on and makes them super accessible to users a platform they're already comfortable with.

✨ Ready to dive in?

• There's a step-by-step guide in the repository to help you build your own bot for helping with reasoning.
• Here's another step-by-step guide to help you chat over own internal HR-realted documents (like onboarding, policies etc.) with your employees right inside your Slack workspace.

🔐 Authorize your APIs

To add an extra layer of security, we can integrate any custom API authorization by adding a auth argument to the @serving decorator.

from lcserve import serving

def authorizer(token: str) -> Any:
    if not token == 'mysecrettoken':            # Change this to add your own authorization logic
        raise Exception('Unauthorized')         # Raise an exception if the request is not authorized

    return 'userid'                             # Return any user id or object

@serving(auth=authorizer)
def ask(question: str, **kwargs) -> str:
    auth_response = kwargs['auth_response']     # This will be 'userid'
    return ...

@serving(websocket=True, auth=authorizer)
async def talk(question: str, **kwargs) -> str:
    auth_response = kwargs['auth_response']     # This will be 'userid'
    return ...

🙋‍♂️ Enable streaming & human-in-the-loop (HITL) with WebSockets

HITL for LangChain agents on production can be challenging since the agents are typically running on servers where humans don't have direct access. langchain-serve bridges this gap by enabling websocket APIs that allow for real-time interaction and feedback between the agent and a human operator.

Check out this example to see how you can enable HITL for your agents.

📁 Persistent storage on Jina AI Cloud

Every app deployed on Jina AI Cloud gets a persistent storage (EFS) mounted locally which can be accessed via workspace kwarg in the @serving function.

from lcserve import serving

@serving
def store(text: str, **kwargs):
    workspace: str = kwargs.get('workspace')
    path = f'{workspace}/store.txt'
    print(f'Writing to {path}')
    with open(path, 'a') as f:
        f.writelines(text + '\n')
    return 'OK'


@serving(websocket=True)
async def stream(**kwargs):
    workspace: str = kwargs.get('workspace')
    websocket: WebSocket = kwargs.get('websocket')
    path = f'{workspace}/store.txt'
    print(f'Streaming {path}')
    async with aiofiles.open(path, 'r') as f:
        async for line in f:
            await websocket.send_text(line)
    return 'OK'

🚀 Bring your own FastAPI app

If you already have a FastAPI app with pre-defined endpoints, you can use lc-serve to deploy it on Jina AI Cloud.

lc-serve deploy jcloud --app filename:app 

.
└── endpoints.py

# endpoints.py
from typing import Union

from fastapi import FastAPI

app = FastAPI()


@app.get("/status")
def read_root():
    return {"Hello": "World"}


@app.get("/items/{item_id}")
def read_item(item_id: int, q: Union[str, None] = None):
    return {"item_id": item_id, "q": q}

lc-serve deploy jcloud --app endpoints:app

💻 lc-serve CLI

lc-serve is a simple CLI that helps you to deploy your agents on Jina AI Cloud (JCloud)

💡 JCloud Deployment

⚙️ Configurations

For JCloud deployment, you can configure your application infrastructure by providing a YAML configuration file using the --config option. The supported configurations are:

• Instance type (instance), as defined by Jina AI Cloud.
• Minimum number of replicas for your application (autoscale_min). Setting it 0 enables serverless.
• Disk size (disk_size), in GB. The default value is 1 GB.

For example:

instance: C4
autoscale_min: 0
disk_size: 1.5G

You can alternatively include a jcloud.yaml file in your application directory with the desired configurations. However, please note that if the --config option is explicitly used in the command line interface, the local jcloud.yaml file will be disregarded. The command line provided configuration file will take precedence.

If you don't provide a configuration file or a specific configuration isn't specified, the following default settings will be applied:

instance: C3
autoscale_min: 1
disk_size: 1G

💰 Pricing

Applications hosted on JCloud are priced in two categories:

Base credits

• Base credits are charged to ensure high availability for your application by maintaining at least one instance running continuously, ready to handle incoming requests. If you wish to stop the serving application, you can either remove the app completely or put it on pause, the latter allows you to resume the app serving based on persisted configurations (refer to lc-serve CLI section for more information). Both options will halt the consumption of credits.
• Actual credits charged for base credits are calculated based on the instance type as defined by Jina AI Cloud.
• By default, instance type C3 is used with a minimum of 1 instance and Amazon EFS disk of size 1G, which means that if your application is served on JCloud, you will be charged ~10 credits per hour.
• You can change the instance type and the minimum number of instances by providing a YAML configuration file using the --config option. For example, if you want to use instance type C4 with a minimum of 0 replicas, and 2G EFS disk, you can provide the following configuration file:

  instance: C4
  autoscale_min: 0
  disk_size: 2G

Serving credits

• Serving credits are charged when your application is actively serving incoming requests.
• Actual credits charged for serving credits are calculated based on the credits for the instance type multiplied by the duration for which your application serves requests.
• You are charged for each second your application is serving requests.

Total credits charged = Base credits + Serving credits. (Jina AI Cloud defines each credit as €0.005)

Examples

instance: C4
autoscale_min: 0
disk_size: 2G

C4 instance has an hourly credit rate of 20.
EFS has hourly credit rate of 0.104 per GB.
Base credits = 0 + 2 * 0.104 = 0.208 (since `autoscale_min` is 0)
Serving credits = 20 * 10/60 = 3.33
Total credits per hour = 0.208 + 3.33 = 3.538

instance: C3
autoscale_min: 1
disk_size: 1G

C3 instance has an hourly credit rate of 10.
EFS has hourly credit rate of 0.104 per GB.
Base credits = 10 + 1 * 0.104 = 10.104 (since `autoscale_min` is 1)
Serving credits = 10 * 20/60 = 3.33
Total credits per hour = 10.104 + 3.33 = 13.434

❓ Frequently Asked Questions

• lc-serve command not found
• My client that connects to the JCloud hosted App gets timed-out, what should I do?
• How to pass environment variables to the app?
• JCloud deployment failed at pushing image to Jina Hubble, what should I do?
• Debug babyagi playground request/response for external integration

lc-serve command not found

My client that connects to the JCloud hosted App gets timed-out, what should I do?

How to pass environment variables to the app?

JCloud deployment failed at pushing image to Jina Hubble, what should I do?

Debug babyagi playground request/response for external integration

```bash
textual console -x EVENT -x SYSTEM -x DEBUG
```

📣 Reach out to us

Want to deploy your LLM apps on your own infrastructure with all capabilities of Jina AI Cloud?

• Serverless
• Autoscaling
• TLS certs
• Persistent storage
• End to end LLM observability
• and more on auto-pilot!

Join us on Discord and we'd be happy to hear more about your use case.