• Stars
    star
    187
  • Rank 206,464 (Top 5 %)
  • Language
    Go
  • License
    MIT License
  • Created over 1 year ago
  • Updated 6 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

LLaMA-2 in native Go

llama2.go

Go Report Card codecov Go Reference

This is a native Go inference of LLaMA-2, as of 2023-08-19 state-of-the-art open source large language model from Meta. It is ported from github.com/karpathy/llama2.c@bd18228 on 2023-08-19. Additional features may be added.

How to run?

  1. get tokenizer.bin from llama2.c
  2. get weights wget https://huggingface.co/karpathy/tinyllamas/resolve/main/stories110M.bin
  3. go install github.com/nikolaydubina/llama2.go@latest
  4. llama2.go -checkpoint=stories110M.bin -prompt="good morning said sun to trees"
$ llama2.go -checkpoint=stories110M.bin -prompt="good morning said sun to trees"
2023/07/29 09:30:22 config: llama2.Config{Dim:768, HiddenDim:2048, NumLayers:12, NumHeads:12, NumKVHeads:12, VocabSize:32000, SeqLen:1024}
<s>
good morning said sun to trees: "Let's organize an operation!"
The trees clapped their branches and asked "What will we do?"
Badger smiled and replied "We will build a treehouse together!"
The trees got blocks of wood and started to build. Badger put nails in the tiny pieces of wood, while the trees put the blocks together to make a
 solid base. 
When they finished their treehouse, Goodger and the trees sat inside. Badger said, "Look how fancy we made it!"
The trees smiled and nodded. They said, "It's very fancy! Thank you for helping us organize this operation." 
Then they lived happily in their fancy treehouse together!
<s>
Once upon a time, there was a boy named Timmy. Timmy was very hungry and wanted to eat his meal. He asked his mom, "What are we having for dinner
?" His mom said, "We are having chicken and rice." Timmy said, "Yum! I love chicken and rice."
While they were eating, Timmy's dad came in and said, "Hey Timmy, do you want to watch a movie after
2023/07/29 09:30:58 achieved tok/s: 28.619646

Performance

system model llama2.c llama.cpp llama2.go1 llama2.go2
Apple M1 Max 10CPU 64GB stories110M 101.84 tok/s 10.47 tok/s 39.28 tok/s
Apple M1 Max 10CPU 64GB llama2_7b 1.83 tok/s 20.36 tok/s 0.87 tok/s
Apple M1 Max 10CPU 64GB llama2_13b (segfault) 11.71 tok/s 0.38 tok/s

Optimizations

  • transformer steps parallelism
  • loop unrolling
  • in-matrix parallelism
  • (todo) SIMD
  • (todo) quantization

All optimizations are Fuzz-tested against basic algorithm, which is itself tested. To disable optimizations update llama2/transformer.go import to package without optimizations and rebuild.

Related Work and References

Footnotes

  1. No linear algebra optimizations

  2. All linear algebra optimizations

More Repositories

1

go-recipes

🦩 Tools for Go projects
Go
4,036
star
2

go-cover-treemap

🎄 Go code coverage to SVG treemap
Go
484
star
3

go-binsize-treemap

🔍 Go binary size SVG treemap
Go
444
star
4

calendarheatmap

📅 Calendar heatmap inspired by GitHub contribution activity
Go
393
star
5

go-instrument

⚡️ Automatically add Trace Spans to Go methods and functions
Go
163
star
6

treemap

🍬 Pretty Treemaps
Go
149
star
7

go-featureprocessing

🔥 Fast, simple sklearn-like feature processing for Go
Go
116
star
8

go-hackers-delight

"Hacker's Delight" in Go
Go
85
star
9

go-graph-layout

🔮 Graph Layout Algorithms in Go
Go
85
star
10

go-cover-treemap-web

Go
78
star
11

jsonl-graph

🏝 JSONL Graph Tools
Go
72
star
12

import-graph

Collect data about your dependencies
Go
40
star
13

twitter-remover

Remove twitter likes, posts, retweets, replies, followers
38
star
14

watchhttp

🌺 Run command periodically and expose latest STDOUT as HTTP endpoint
Go
32
star
15

go-ml-benchmarks

⏱ Benchmarks of machine learning inference for Go
Go
30
star
16

fpdecimal

🛫 Fixed-Point Decimals
Go
29
star
17

fpmoney

🧧 Fixed-Point Decimal Money
Go
25
star
18

validate

🥬 validate. simply.
Go
19
star
19

hq

🐁 happy little queue
Go
16
star
20

smrcptr

detect mixing pointer and value method receivers
Go
13
star
21

neuroscience-landscape

🌌 Resources on Neuroscience
12
star
22

vertfn

Go linter for Vertical Function Ordering
Go
12
star
23

go-enum-encoding

Generate Go enum encoding
Go
12
star
24

go-enum-example

Go Enum: benchmarks, examples, analysis
Go
8
star
25

htmljson

🫐 Rich rendering of JSON as HTML in Go
Go
7
star
26

multiline-jsonl

Read and write multiline JSONL in Go
Go
6
star
27

openapi-inline-examples

🌏 Inline OpenAPI JSON examples from filenames
Go
6
star
28

htmlyaml

🐹 render YAML as HTML in Go
Go
4
star
29

go-commentage

How far Go comments drifting behind?
Go
4
star
30

rchan

Go channel through Redis List
Go
4
star
31

mini-awesome-cv

📝 LaTeX Awesome-CV under 200LOC
TeX
4
star
32

go-instrument-example

Go
3
star
33

svgpan

Pan and Zoom of SVG in your Go front-end app in browser.
Go
3
star
34

go-bench-errors

Benchmarking Go errors
Go
3
star
35

mdpage

one-pager Markdown list from YAML
Go
3
star
36

consistentimports

Detect inconsistent import aliases
Go
2
star
37

go-callsite-stats

analyse function callsites
Go
2
star
38

read-seek-peeker

Go Reader that can Seek() and Peek()
Go
2
star
39

go-bench-stream

🌊 Go Benchmarks for Stream Processing
Go
2
star
40

go-bitset-example

Go Bitset: benchmarks, examples, analysis
Go
1
star
41

aws-s3-reader

Efficient Go Reader for large AWS S3 Objects
Go
1
star
42

go-bench-receiver

Which is more efficient value or pointer method receivers in Go?
Go
1
star
43

totp

TOTP (RFC-6238) and HOTP (RFC-4226)
Go
1
star
44

dotfiles

macOS, Linux
Vim Script
1
star
45

presentations

1
star
46

PINTOS

C
1
star
47

ARIA

C++
1
star
48

mini-blog

JavaScript
1
star
49

graph-tools-gallery

Inspiration gallery of graph tools
1
star