• Stars
    star
    206
  • Rank 190,504 (Top 4 %)
  • Language
    Rust
  • License
    Apache License 2.0
  • Created about 1 year ago
  • Updated 10 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

The easiest & fastest way to run customized and fine-tuned LLMs locally or on the edge

LlamaEdge

The LlamaEdge project makes it easy for you to run LLM inference apps and create OpenAI-compatible API services for the Llama2 series of LLMs locally.

Quick start

Run a single command in your command line terminal.

bash <(curl -sSfL 'https://code.flows.network/webhook/iwYN1SdN3AmPgR5ao5Gt/run-llm.sh')

Follow the on-screen instructions to install the WasmEdge Runtime and download your favorite open-source LLM. Then, choose whether you want to chat with the model via the CLI or via a web UI.

See it in action | Docs

How it works?

The Rust source code for the inference applications are all open source and you can modify and use them freely for your own purposes.

  • The folder simple contains the source code project to generate text from a prompt using run llama2 models.
  • The folder chat contains the source code project to "chat" with a llama2 model on the command line.
  • The folder api-server contains the source code project for a web server. It provides an OpenAI-compatible API service, as well as an optional web UI, for llama2 models.

The tech stack

The Rust+Wasm stack provides a strong alternative to Python in AI inference.

  • Lightweight. The total runtime size is 30MB.
  • Fast. Full native speed on GPUs.
  • Portable. Single cross-platform binary on different CPUs, GPUs, and OSes.
  • Secure. Sandboxed and isolated execution on untrusted devices.
  • Container-ready. Supported in Docker, containerd, Podman, and Kubernetes.

For more information, please check out Fast and Portable Llama2 Inference on the Heterogeneous Edge.

Models

The LlamaEdge project supports all Large Language Models (LLMs) based on the llama2 framework. The model files must be in the GGUF format. We are committed to continuously testing and validating new open-source models that emerge every day.

Click here to see the supported model list with a download link and startup commands for each model. If you have success with other LLMs, don't hesitate to contribute by creating a Pull Request (PR) to help extend this list.

Platforms

The compiled Wasm file is cross platfrom. You can use the same Wasm file to run the LLM across OSes (e.g., MacOS, Linux, Windows SL), CPUs (e.g., x86, ARM, Apple, RISC-V), and GPUs (e.g., NVIDIA, Apple).

The installer from WasmEdge 0.13.5 will detect NVIDIA CUDA drivers automatically. If CUDA is detected, the installer will always attempt to install a CUDA-enabled version of the plugin. The CUDA support is tested on the following platforms in our automated CI.

  • Nvidia Jetson AGX Orin 64GB developer kit
  • Intel i7-10700 + Nvidia GTX 1080 8G GPU
  • AWS EC2 g5.xlarge + Nvidia A10G 24G GPU + Amazon deep learning base Ubuntu 20.04

If you're using CPU only machine, the installer will install the OpenBLAS version of the plugin instead. You may need to install libopenblas-dev by apt update && apt install -y libopenblas-dev.

Troubleshooting

Q: Even though my machine has a large RAM, after asking several questions, I received an error message returns 'Error: Backend Error: WASI-NN'. What should I do?

A: To enable machines with smaller RAM, like 8 GB, to run a 7b model, we've set the context size limit to 512. If your machine has more capacity, you can increase both the context size and batch size up to 4096 using the CLI options available here. Use these commands to adjust the settings:

-c, --ctx-size <CTX_SIZE>
-b, --batch-size <BATCH_SIZE>

Q: After running apt update && apt install -y libopenblas-dev, you may encounter the following error:

...
E: Could not open lock file /var/lib/dpkg/lock-frontend - open (13: Permission denied)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), are you root?

A: This indicates that you are not logged in as root. Please try installing again using the sudo command:

sudo apt update && sudo apt install -y libopenblas-dev

Q: After running the wasmedge command, you may receive the following error:

[2023-10-02 14:30:31.227] [error] loading failed: invalid path, Code: 0x20
[2023-10-02 14:30:31.227] [error]     load library failed:libblas.so.3: cannot open shared object file: No such file or directory
[2023-10-02 14:30:31.227] [error] loading failed: invalid path, Code: 0x20
[2023-10-02 14:30:31.227] [error]     load library failed:libblas.so.3: cannot open shared object file: No such file or directory
unknown option: nn-preload

A: This suggests that your plugin installation was not successful. To resolve this issue, please attempt to install your desired plugin again.

Q: After executing the wasmedge command, you might encounter the error message: [WASI-NN] GGML backend: Error: unable to init model.

A: This error signifies that the model setup was not successful. To resolve this issue, please verify the following:

  1. Check if your model file and the WASM application are located in the same directory. The WasmEdge runtime requires them to be in the same location to locate the model file correctly.
  2. Ensure that the model has been downloaded successfully. You can use the command shasum -a 256 <gguf-filename> to verify the model's sha256sum. Compare your result with the correct sha256sum available on the Hugging Face page for the model.
image

Credits

The WASI-NN ggml plugin embedded llama.cpp as its backend.

More Repositories

1

wasmedge-quickjs

A high-performance, secure, extensible, and OCI-complaint JavaScript runtime for WasmEdge.
JavaScript
487
star
2

wasm-learning

Building Rust functions for Node.js to take advantage of Rust's performance, WebAssembly's security and portability, and JavaScript's ease-of-use. Demo code and recipes.
Rust
430
star
3

SOLL

SOLL is a new compiler for generate Ewasm from solidity and yul. See a demo here: https://asciinema.org/a/ezJqNLicn5fya02zwu4VXIo8a
C++
376
star
4

buidl

A browser-based IDE for creating, deploying, and sharing blockchain apps (DApps, or decentralized apps). Publish your first blockchain DApps in 5 minutes! Here is how: https://docs.secondstate.io/buidl-developer-tool/getting-started
Vue
364
star
5

microservice-rust-mysql

A template project for building a database-driven microservice in Rust and run it in the WasmEdge sandbox.
Rust
346
star
6

dapr-wasm

A template project to demonstrate how to run WebAssembly functions as sidecar microservices in dapr
Rust
275
star
7

smart-contract-search-engine

Takes a link to a smart contract's raw ABI file and an RPC URL and then indexes all instances of that smart contract
JavaScript
267
star
8

WasmEdge-WASINN-examples

Rust
237
star
9

wasmedge-nodejs-starter

A template project to run Rust functions in Node.js through the Second State WebAssembly engine.
JavaScript
158
star
10

rustwasmc

Tool for building Rust functions for Node.js. Combine the performance of Rust, safety and portability of WebAssembly, and ease of use of JavaScript.
Rust
129
star
11

how_to_deploy_uniswap

Detailed instructions on how to deploy Uniswap on an Ethereum compatible blockchain
JavaScript
124
star
12

chat-with-chatgpt

Chat with ChatGPT via GitHub issue comments.
111
star
13

WasmEdge-go

The GO language SDK and API for WasmEdge
Go
107
star
14

vercel-wasm-runtime

A template project for building high-performance, portable, and safe serverless functions in Vercel.
JavaScript
81
star
15

wasmedge-containers-examples

Shell
71
star
16

wasmedge_wasi_socket

A Rust lib for socket on WasmEdge.
Rust
70
star
17

meetups

70
star
18

rust-by-example-ext

Rust by Example -- Extended Edition
Dockerfile
64
star
19

lity

A rule-based contract-oriented high-level language.
C++
56
star
20

wasm-joey

Serverless Wasm - A lightweight Node.js application for deploying and executing WebAssembly(Wasm) binary-code via HTTP
JavaScript
54
star
21

ssvm-deno-starter

A template project to run Rust functions in Deno through the Second State WebAssembly engine.
Dockerfile
53
star
22

SewUp

A library to help you sew up your Ethereum project with Rust and just like develop in a common backend
Rust
52
star
23

rust-wasm-ai-demo

Rust functions for Tensorflow inference in Node.js. Rust's performance, WebAssembly's security and portability, and Javascript's ease-of-use.
Dockerfile
49
star
24

WasmEdge-go-examples

Go
46
star
25

recrypt-as-a-service

An open key management service to support privacy-first and scalable file sharing.
Rust
46
star
26

learn-rust-with-github-actions

Getting started with the Rust programming language using this VSCode and GitHub Actions template.
Dockerfile
46
star
27

aws-lambda-wasm-runtime

A template project for building high-performance, portable, and safe serverless functions in AWS Lambda.
JavaScript
44
star
28

wasmedge-seL4

Integrate WasmEdge with seL4
Shell
42
star
29

dapr-sdk-wasmedge

An experimental Dapr SDK in Rust. It is designed to run in WasmEdge applications.
Rust
41
star
30

MEGA

Make ETLs Great Again!
Rust
40
star
31

tencent-tensorflow-scf

A template project for serverless functions for Tensorflow inference on Tencent Cloud.
HTML
38
star
32

simple-staking-smart-contract

A smart contract which allows users to stake and un-stake (after a set period of time) a specified ERC20 token.
Solidity
38
star
33

wasm32-wasi-benchmark

C++
35
star
34

WebAssembly-landscape

The current development of WebAssembly
34
star
35

rust-examples

Simple Rust applications that run in WasmEdge
Rust
32
star
36

wasmedge-rustsdk-examples

Rust
31
star
37

tencent-scf-wasm-runtime

基于 WebAssembly 容器镜像的高性能腾讯云函数开发模版。A template project for building high-performance, portable, and safe serverless functions in Tencent Serverless Cloud Functions.
JavaScript
29
star
38

crunw

Add WasmEdge support to crun so that Docker and k8s tools can manage WasmEdge runtimes side by side with Docker-like application containers.
C
29
star
39

wasmedge-bindgen

Let WebAssembly's exported function support more data types for its parameters and return values.
Rust
29
star
40

wasmedge_tensorflow_interface

Rust
20
star
41

kubecon-eu-2023

20
star
42

substrate-ewasm

A substrate runtime module library (SRML) for Etherem flavored WebAssembly (ewasm)
18
star
43

runw

C++
18
star
44

simple-timelock-smart-contract

A simple timelock smart contract which locks ERC20 tokens for a specific time period and then allows users to unlock when that time period has elapsed
Solidity
17
star
45

netlify-wasm-runtime

A template project for building high-performance, portable, and safe serverless functions in Netlify.
JavaScript
16
star
46

rust-ssvm

Use EVMC binding SSVM and host written in Rust
Rust
15
star
47

serverless-reactor-starter

Rust
15
star
48

hugo-website

A no-code, no-software and no-cost solution to publishing sophisticated web sites managed by non-technical people.
HTML
15
star
49

witc

wasm-interface-types supplement & compiler of wasmedge
Haskell
13
star
50

ssvm-napi

SSVM Node.js Addon
C++
12
star
51

WasmEdge-tensorflow

C++
12
star
52

AI-as-a-Service

Rust
12
star
53

OCR-tesseract-on-Centos7

How to install and use tesseract OCR on Centos7 - without root access
11
star
54

interest-earner-smart-contract

A Solidity smart contract which allows users to stake, earn and un-stake.
Solidity
10
star
55

linear-timelock-smart-contract

Solidity smart contract which disburses ERC20 tokens linearly, over a specific period of time (all values are dynamic and are set by the contract owner)
Solidity
10
star
56

substrate-wasmedge

Substrate on WasmEdge
Rust
10
star
57

ssvm-tencent-starter

Shell
9
star
58

nodejs-helper

Rust
9
star
59

WasmEdge-tensorflow-tools

C++
9
star
60

llm-web-api-server

Rust
8
star
61

wasm-llm

Python
6
star
62

WasmEdge-evmc

Second State WebAssembly VM for EVMC Extension
C++
6
star
63

wasmedge-core

WasmEdge Node.js Addon
C++
6
star
64

wasmedge_plugin_rust_sdk

Rust
5
star
65

tencent-meme-scf

HTML
5
star
66

chatbot-ui

TypeScript
5
star
67

WasmEdge-image

C
5
star
68

qdrant-rest-client

A lightweight Qdrant client library for Rust
Rust
4
star
69

llm_todo

Python
4
star
70

wasmedge-mysql

A repository which provides a MySQL persistent storage solution for the WasmEdge runtime
JavaScript
4
star
71

SSVMRPC

A Remote Procedure Call (RPC) implementation which facilitates both code-deployment and code-execution interactions with SecondState's stateless Virtual Machine (SSVM)
Rust
4
star
72

opendapps

HTML
4
star
73

ssvm-napi-extensions

C++
4
star
74

micro-slots

Storing and accessing multiple integer values in Ethereum, using only one single uint256 slot
Python
4
star
75

libeni

Official C++ implementation of libENI, which is part of the Lity project.
C++
4
star
76

serialize_deserialize_u8_i32

A Rust library that safely converts, back and forward, between u8 and i32
Rust
3
star
77

tencent-ocr-scf

Serverless functions for OCR on Tencent Cloud.
HTML
3
star
78

ssvm-tencent-tensorflow

HTML
3
star
79

wasmedge_hostfunctionexample_interface

This is an example to create a host function interface for calling the WasmEdge internal host functions.
Rust
3
star
80

wasmmark

JavaScript
3
star
81

wasmedge-extensions

WasmEdge for Node.js Addon with extensions
C++
3
star
82

server-side-wasm-video-editing

An example of how to build a third-party application that uses SecondState's Wasm VM to edit video on the server side. This prototype shows how video can undergo frame-by-frame, pixel-by-pixel processing using native Wasm data only i.e. pixel as i32
HTML
3
star
83

docai-hf-rs

Document AI for WasmEdge
2
star
84

linkerd-wasm

A template project to run WebAssembly functions as microservices in Linkerd
2
star
85

wasmedge_process_interface

Rust
2
star
86

rfq-marketplace

Rust
2
star
87

WASI-NN-GGML-PLUGIN-REGISTRY

This is a registry project for storing all released WasmEdge WASI-NN GGML plugin with different llama.cpp versions.
2
star
88

linear-timelock-user-interface

Interface for LinearTimelock.sol smart contract
JavaScript
2
star
89

rust-faas-showcases

HTML
2
star
90

rust_native_storage_library

A library which compiles to `.so` and `.dylib` and facilitates the native storage (and retrieval) of key:value pairs on the host system
Rust
2
star
91

WasmEdge-storage

Second State WebAssembly VM for Rust Storage Extension
C++
2
star
92

simple-timelock-user-interface

A user interface for the simple timelock smart contract
JavaScript
2
star
93

WasmEdge-paratime

An Oasis Paratime using WasmEdge
Rust
2
star
94

SSVMContainer

A Rust application that sits between incoming requests from the network and the SSVM. This application handles the deployment of Wasm applications and manages the execution of services (callable functions inside the Wasm application). The actual execution takes place inside the SSVM. The SSVM is a stack-based Virtual Machine which is stateless. This SSVMContainer application is responsible for fielding incoming deployments and service calls as well as storing application state.
HTML
2
star
95

wasmedge_rustls_api

Rust
1
star
96

libsql_wasi_nn

Rust
1
star
97

haiku-connector

Haiku Connector Runner
Rust
1
star
98

wasmedge_hyper_rustls

A crate similar to rustls-hyper for wasi.
Rust
1
star
99

wasmedge_asyncify

Rust
1
star
100

interest-earner-user-interface

A user interface for the interest earner smart contract
JavaScript
1
star