• Stars
    star
    196
  • Rank 198,553 (Top 4 %)
  • Language
    Jupyter Notebook
  • License
    Other
  • Created over 2 years ago
  • Updated 7 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Multilingual Generative Pretrained Model

mGPT

Multilingual Generative Pretrained Transformer

Apache license

PWC

PWC

PWC

PWC

PWC

[Paper] [Habr] [Model Card]

  • 1.3 billion parameter model
  • Trained on 60 languages
  • HuggingFace compatible model card

Web Demo

Integrated into Huggingface Spaces 🤗 using Gradio. Try out the Web Demo for generation: Hugging Face Spaces

Setting up environment

pip install -r requirements.txt

Checkpoint backup

Download checkpoints to load model from disk:

!wget https://files.sberdisk.ru/s/NzeBqYE84TAQDiS/download -O model.zip
!unzip model.zip -d mgptxl
model_name = "./mgptxl" 

Transformers usage 🤗

from transformers import GPT2LMHeadModel, GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained("sberbank-ai/mGPT")
model = GPT2LMHeadModel.from_pretrained("sberbank-ai/mGPT")

text = "Александр Сергеевич Пушкин родился в "
input_ids = tokenizer.encode(text, return_tensors="pt").cuda(device)
out = model.generate(
        input_ids, 
        min_length=100, 
        max_length=100, 
        eos_token_id=5, 
        pad_token=1,
        top_k=10,
        top_p=0.0,
        no_repeat_ngram_size=5
)
generated_text = list(map(tokenizer.decode, out))[0]
print(generated_text)
Александр Сергеевич Пушкин родился в  г. Санкт-Петербурге.

Choosing best parameters:

In general:

eos_token_id=5, 
pad_token=1,
do_sample=True,
top_k=0,
top_p=0.8,
no_repeat_ngram_size=4

English Generation: top_p=0.95, top_k=0

Examples

mGPT Generation Examples

Open In Colab

mGPT Fine-tuning example

Open In Colab

Languages supported

  • Languages: Afrikaans, Azerbaijani, Belarusian, Bengali, Chuvash, German, English, Basque, Finnish, Hebrew (modern), Hungarian, Indonesian, Japanese, Kazakh, Kirghiz, Kyrgyz, Latvian, Mongolian, Malay, Dutch, Polish, Romanian, Moldavan, Yakut, Swahili, Telugu, Thai, Turkish, Tuvinian, Urdu, Vietnamese, Yoruba, Arabic, Bashkir, Bulgarian, Buriat, Danish, Greek, Modern, Spanish; Castilian, Persian, French, Hindi, Armenian, Italian, Georgian, Korean, Lithuanian, Malayalam, Marathi, Burmese, Ossetian, Ossetic, Portuguese, Russian, Swedish, Tamil, Tajik, Turkmen, Tatar, Ukrainian, Uzbek, Kalmyk, Chinese
  • ISO codes: az, sw, af, ar, ba, be, bxr, bg, bn, cv, hy, da, de, el, es, eu, fa, fi, fr, he, hi, hu, kk, id, it, ja, ka, ky, ko, lt, lv, mn, ml, os, mr, ms, my, nl, ro, pl, pt, sah, ru, tg, sv, ta, te, tk, th, tr, tl, tt, tyv, uk, en, ur, vi, uz, yo, zh, xal

Cite Us

mGPT: Few-Shot Learners Go Multilingual

Abstract PDF

@misc{https://doi.org/10.48550/arxiv.2204.07580,
 doi = {10.48550/ARXIV.2204.07580},
 
 url = {https://arxiv.org/abs/2204.07580},
 
 author = {Shliazhko, Oleh and Fenogenova, Alena and Tikhonova, Maria and Mikhailov, Vladislav and Kozlova, Anastasia and Shavrina, Tatiana},
 
 keywords = {Computation and Language (cs.CL), Artificial Intelligence (cs.AI), FOS: Computer and information sciences, FOS: Computer and information sciences, I.2; I.2.7, 68-06, 68-04, 68T50, 68T01},
 
 title = {mGPT: Few-Shot Learners Go Multilingual},
 
 publisher = {arXiv},
 
 year = {2022},
 
 copyright = {Creative Commons Attribution 4.0 International}
}

Contributing

We welcome community contributions to the model, and celebrate both its inference and training technique enhancements

More Repositories

1

Kandinsky-2

Kandinsky 2 — multilingual text2image latent diffusion model
Jupyter Notebook
2,745
star
2

ru-gpts

Russian GPT3 models.
Python
2,077
star
3

ru-dalle

Generate images from texts. In Russian
Jupyter Notebook
1,640
star
4

ghost

A new one shot face swap approach for image and video domains
Python
1,190
star
5

ner-bert

BERT-NER (nert-bert) with google bert https://github.com/google-research.
Jupyter Notebook
405
star
6

ru-dolph

RUDOLPH: One Hyper-Tasking Transformer can be creative as DALL-E and GPT-3 and smart as CLIP
Jupyter Notebook
242
star
7

Real-ESRGAN

PyTorch implementation of Real-ESRGAN model
Python
201
star
8

KandinskyVideo

KandinskyVideo — multilingual end-to-end text2video latent diffusion model
Python
164
star
9

ru-clip

CLIP implementation for Russian language
Jupyter Notebook
134
star
10

sage

SAGE: Spelling correction, corruption and evaluation for multiple languages
Jupyter Notebook
129
star
11

ruGPT3_demos

121
star
12

deforum-kandinsky

Kandinsky x Deforum — generating short animations
Python
102
star
13

digital_peter_aij2020

Materials of the AI Journey 2020 competition dedicated to the recognition of Peter the Great's manuscripts, https://ai-journey.ru/contest/task01
Jupyter Notebook
66
star
14

music-composer

Python
62
star
15

ru-prompts

Python
57
star
16

gigachat

Библиотека для доступа к GigaChat
Python
57
star
17

MERA

MERA (Multimodal Evaluation for Russian-language Architectures) is a new open benchmark for the Russian language for evaluating fundamental models.
Jupyter Notebook
55
star
18

fusion_brain_aij2021

Creating multimodal multitask models
Jupyter Notebook
50
star
19

augmentex

Augmentex — a library for augmenting texts with errors
Python
48
star
20

model-zoo

NLP model zoo for Russian
45
star
21

OCR-model

An easy-to-run OCR model pipeline based on CRNN and CTC loss
Python
43
star
22

StackMix-OCR

Jupyter Notebook
40
star
23

MoVQGAN

MoVQGAN - model for the image encoding and reconstruction
Jupyter Notebook
35
star
24

tuned-vq-gan

Jupyter Notebook
28
star
25

ReadingPipeline

Text reading pipeline that combines segmentation and OCR-models.
Python
26
star
26

DataProcessingFramework

Framework for processing and filtering datasets
Python
25
star
27

htr_datasets

Repository containing our datasets for HTR (handwritten text recognition) task.
Jupyter Notebook
23
star
28

CerberusDet

CerberusDet: Unified Multi-Task Object Detection
Python
23
star
29

fbc3_aij2023

Jupyter Notebook
21
star
30

mineral-recognition

Python
20
star
31

DigiTeller

18
star
32

fbc2_aij2022

FusionBrain Challenge 2.0: creating multimodal multitask model
Python
16
star
33

combined_solution_aij2019

AI Journey 2019: Combined Solution
Python
15
star
34

SEGM-model

An easy-to-run semantic segmentation model based on Unet
Python
13
star
35

railway_infrastructure_detection_aij2021

AI Journey Contest 2021: AITrain
Python
13
star
36

no_fire_with_ai_aij2021

AI Journey Contest 2021: NoFireWithAI
Jupyter Notebook
13
star
37

ControlledNST

An implementation of Neural Style Transfer in PyTorch.
Jupyter Notebook
8
star
38

kandinsky3-diffusers

Python
6
star
39

mchs-wildfire

Соревнование по классификации лесных пожаров
Jupyter Notebook
4
star
40

no_flood_with_ai_aij2020

Материалы соревнования AI Journey 2020, посвященного прогнозированию паводков на реке Амур, https://ai-journey.ru/contest/task02
Jupyter Notebook
4
star
41

Zoom_In_Video_Kandinsky

Framework for creating Zoom in / Zoom out video based on inpainting Kandinsky
Jupyter Notebook
2
star
42

langchain-gigachat

Python
1
star
43

paper_persi_chat

PaperPersiChat: Scientific Paper Discussion Chatbot using Transformers and Discourse Flow Management
Jupyter Notebook
1
star