• Stars
    star
    166
  • Rank 220,433 (Top 5 %)
  • Language
    Go
  • License
    MIT License
  • Created 10 months ago
  • Updated 7 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

LLaMA-2 in native Go

llama2.go

Go Report Card codecov Go Reference

This is a native Go inference of LLaMA-2, as of 2023-08-19 state-of-the-art open source large language model from Meta. It is ported from github.com/karpathy/llama2.c@bd18228 on 2023-08-19. Additional features may be added.

How to run?

  1. get tokenizer.bin from llama2.c
  2. get weights wget https://huggingface.co/karpathy/tinyllamas/resolve/main/stories110M.bin
  3. go install github.com/nikolaydubina/llama2.go@latest
  4. llama2.go -checkpoint=stories110M.bin -prompt="good morning said sun to trees"
$ llama2.go -checkpoint=stories110M.bin -prompt="good morning said sun to trees"
2023/07/29 09:30:22 config: llama2.Config{Dim:768, HiddenDim:2048, NumLayers:12, NumHeads:12, NumKVHeads:12, VocabSize:32000, SeqLen:1024}
<s>
good morning said sun to trees: "Let's organize an operation!"
The trees clapped their branches and asked "What will we do?"
Badger smiled and replied "We will build a treehouse together!"
The trees got blocks of wood and started to build. Badger put nails in the tiny pieces of wood, while the trees put the blocks together to make a
 solid base. 
When they finished their treehouse, Goodger and the trees sat inside. Badger said, "Look how fancy we made it!"
The trees smiled and nodded. They said, "It's very fancy! Thank you for helping us organize this operation." 
Then they lived happily in their fancy treehouse together!
<s>
Once upon a time, there was a boy named Timmy. Timmy was very hungry and wanted to eat his meal. He asked his mom, "What are we having for dinner
?" His mom said, "We are having chicken and rice." Timmy said, "Yum! I love chicken and rice."
While they were eating, Timmy's dad came in and said, "Hey Timmy, do you want to watch a movie after
2023/07/29 09:30:58 achieved tok/s: 28.619646

Performance

system model llama2.c llama.cpp llama2.go1 llama2.go2
Apple M1 Max 10CPU 64GB stories110M 101.84 tok/s 10.47 tok/s 39.28 tok/s
Apple M1 Max 10CPU 64GB llama2_7b 1.83 tok/s 20.36 tok/s 0.87 tok/s
Apple M1 Max 10CPU 64GB llama2_13b (segfault) 11.71 tok/s 0.38 tok/s

Optimizations

  • transformer steps parallelism
  • loop unrolling
  • in-matrix parallelism
  • (todo) SIMD
  • (todo) quantization

All optimizations are Fuzz-tested against basic algorithm, which is itself tested. To disable optimizations update llama2/transformer.go import to package without optimizations and rebuild.

Related Work and References

Footnotes

  1. No linear algebra optimizations

  2. All linear algebra optimizations

More Repositories

1

go-recipes

🦩 Tools for Go projects
Go
3,831
star
2

go-binsize-treemap

🔍 Go binary size SVG treemap
Go
442
star
3

calendarheatmap

📅 Calendar heatmap inspired by GitHub contribution activity
Go
388
star
4

go-cover-treemap

🎄 Go code coverage to SVG treemap
Go
280
star
5

go-instrument

⚡️ Automatically add Trace Spans to Go methods and functions
Go
141
star
6

treemap

🍬 Pretty Treemaps
Go
137
star
7

go-featureprocessing

🔥 Fast, simple sklearn-like feature processing for Go
Go
115
star
8

go-cover-treemap-web

Go
78
star
9

go-graph-layout

🔮 Graph Layout Algorithms in Go
Go
78
star
10

jsonl-graph

🏝 JSONL Graph Tools
Go
71
star
11

import-graph

Collect data about your dependencies
Go
39
star
12

watchhttp

🌺 Run command periodically and expose latest STDOUT as HTTP endpoint
Go
30
star
13

go-ml-benchmarks

⏱ Benchmarks of machine learning inference for Go
Go
30
star
14

fpdecimal

🛫 Fixed-Point Decimals
Go
29
star
15

twitter-remover

Remove twitter likes, posts, retweets, replies, followers
25
star
16

fpmoney

🧧 Fixed-Point Decimal Money
Go
24
star
17

validate

🥬 validate. simply.
Go
20
star
18

hq

🐁 happy little queue
Go
15
star
19

neuroscience-landscape

🌌 Resources on Neuroscience
12
star
20

vertfn

Go linter for Vertical Function Ordering
Go
12
star
21

smrcptr

detect mixing pointer and value method receivers
Go
11
star
22

go-enum-encoding

Generate Go enum encoding
Go
8
star
23

htmljson

🫐 Rich rendering of JSON as HTML in Go
Go
7
star
24

go-enum-example

Go Enum: benchmarks, examples, analysis
Go
6
star
25

openapi-inline-examples

🌏 Inline OpenAPI JSON examples from filenames
Go
6
star
26

multiline-jsonl

Read and write multiline JSONL in Go
Go
5
star
27

rchan

Go channel through Redis List
Go
4
star
28

go-commentage

How far Go comments drifting behind?
Go
3
star
29

htmlyaml

🐹 render YAML as HTML in Go
Go
3
star
30

go-bench-errors

Benchmarking Go errors
Go
3
star
31

go-callsite-stats

analyse function callsites
Go
2
star
32

svgpan

Pan and Zoom of SVG in your Go front-end app in browser.
Go
2
star
33

mini-awesome-cv

📝 LaTeX Awesome-CV under 200LOC
TeX
2
star
34

go-bench-stream

🌊 Go Benchmarks for Stream Processing
Go
2
star
35

consistentimports

Detect inconsistent import aliases
Go
1
star
36

go-instrument-example

Go
1
star
37

mdpage

🍙 CLI tool to generate one-page Markdown lists based on YAML
Go
1
star
38

go-bitset-example

Go Bitset: benchmarks, examples, analysis
Go
1
star
39

aws-s3-reader

Efficient Go Reader for large AWS S3 Objects
Go
1
star
40

read-seek-peeker

Go Reader that can Seek() and Peek()
Go
1
star
41

go-bench-receiver

Which is more efficient value or pointer method receivers in Go?
Go
1
star
42

totp

TOTP (RFC-6238) and HOTP (RFC-4226)
Go
1
star
43

dotfiles

macOS, Linux
Vim Script
1
star
44

presentations

1
star
45

PINTOS

C
1
star
46

ARIA

C++
1
star
47

mini-blog

JavaScript
1
star
48

graph-tools-gallery

Inspiration gallery of graph tools
1
star