lonePatient/albert_pytorch lonePatient/albert_pytorch

  • Stars
    star
    708
  • Rank 63,504 (Top 2 %)
  • Language
    Python
  • License
    Apache License 2.0
  • Created almost 5 years ago
  • Updated over 4 years ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
lonePatient/albert_pytorch
github

lonePatient

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A Lite Bert For Self-Supervised Learning Language Representations

English Version | 中文版说明

albert_pytorch

This repository contains a PyTorch implementation of the albert model from the paper

A Lite Bert For Self-Supervised Learning Language Representations

by Zhenzhong Lan. Mingda Chen....

Dependencies

  • pytorch=1.10
  • cuda=9.0
  • cudnn=7.5
  • scikit-learn
  • sentencepiece

Download Pre-trained Models of English

Official download links: google albert

Adapt to this version，download pytorch model (google drive):

v1

v2

Fine-tuning

１. Place config.json and 30k-clean.model into the prev_trained_model/albert_base_v2 directory. example:

├── prev_trained_model
|  └── albert_base_v2
|  |  └── pytorch_model.bin
|  |  └── config.json
|  |  └── 30k-clean.model

2．convert albert tf checkpoint to pytorch

python convert_albert_tf_checkpoint_to_pytorch.py \
    --tf_checkpoint_path=./prev_trained_model/albert_base_tf_v2 \
    --bert_config_file=./prev_trained_model/albert_base_v2/config.json \
    --pytorch_dump_path=./prev_trained_model/albert_base_v2/pytorch_model.bin

The General Language Understanding Evaluation (GLUE) benchmark is a collection of nine sentence- or sentence-pair language understanding tasks for evaluating and analyzing natural language understanding systems.

Before running anyone of these GLUE tasks you should download the GLUE data by running this script and unpack it to some directory $DATA_DIR.

3．run sh scripts/run_classifier_sst2.shto fine tuning albert model

Result

Performance of ALBERT on GLUE benchmark results using a single-model setup on dev:

Cola Sst-2 Mnli Sts-b
metric matthews_corrcoef accuracy accuracy pearson
model Cola Sst-2 Mnli Sts-b
albert_base_v2 0.5756 0.926 0.8418 0.9091
albert_large_v2 0.5851 0.9507 0.9151
albert_xlarge_v2 0.6023 0.9221

More Repositories

1

awesome-pretrained-chinese-nlp-models

Awesome Pretrained Chinese NLP Models，高质量中文预训练模型&大模型&多模态模型&大语言模型集合
Python
4,712
star
2

BERT-NER-Pytorch

Chinese NER(Named Entity Recognition) using BERT(Softmax, CRF, Span)
Python
2,062
star
3

Bert-Multi-Label-Text-Classification

This repo contains a PyTorch implementation of a pretrained BERT model for multi-label text classification.
Python
860
star
4

NeZha_Chinese_PyTorch

NEZHA: Neural Contextualized Representation for Chinese Language Understanding
Python
261
star
5

lookahead_pytorch

pytorch implement of Lookahead Optimizer
Python
188
star
6

TorchBlocks

A PyTorch-based toolkit for natural language processing
Python
151
star
7

daguan_2019_rank9

datagrand 2019 information extraction competition rank9
Python
130
star
8

BiLSTM-CRF-NER-PyTorch

This repo contains a PyTorch implementation of a BiLSTM-CRF model for named entity recognition task.
Python
120
star
9

Deep_Learning_For_Computer_Vision_With_Python

Deep Learning For Computer Vision With Python
Python
118
star
10

BERT-chinese-text-classification-pytorch

This repo contains a PyTorch implementation of a pretrained BERT model for text classification.
Python
99
star
11

electra_pytorch

ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Python
91
star
12

CLUE_pytorch

CLUE baseline pytorch CLUE的pytorch版本基线
Python
73
star
13

MobileBert_PyTorch

MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
Python
61
star
14

BERT-Attribute-Value-Extract

A Pytorch implementation of "Scaling Up Open Tagging from Tens to Thousands: Comprehension Empowered Attribute Value Extraction from Product Title" (ACL 2019).
Python
59
star
15

multi-sample_dropout_pytorch

a simple pytorch implement of Multi-Sample Dropout
Python
56
star
16

BERT-SDA

A PyTorch implementation of "Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillation"
Python
56
star
17

ERNIE-text-classification-pytorch

This repo contains a PyTorch implementation of a pretrained ERNIE model for text classification.
Python
54
star
18

chinese-word2vec-pytorch

word2vec implementation for skip-gram in pytorch
Python
53
star
19

bert-sentence-similarity-pytorch

This repo contains a PyTorch implementation of a pretrained BERT model for sentence similarity task.
Python
49
star
20

label_smoothing_pytorch

pytorch implement of Label Smoothing
Python
32
star
21

EvoNorms_PyTorch

Evolving Normalization-Activation Layers
Python
19
star
22

NovoGrad-pytorch

pytorch implement of NovoGrad Optimizer
Python
18
star
23

cw2vec-pytorch

cw2vec implementation in pytorch
Python
17
star
24

train-bert-pytorch

Python
15
star
25

knowledge-driven-dialogue-lic2019-rank5

2019语言与智能技术竞赛第5名方案
Python
14
star
26

2021-GAIIC-Track1-idea

全球人工智能技术创新大赛【赛道一】
10
star
27

pytorch_fashionMNIST_practice

使用pytorch进行图像训练的模板
Python
9
star
28

keras_learning

Jupyter Notebook
9
star
29

Contextual-Chinese-Strokes-Embeddings

Implementation of the language model for Contextual chinese strokes Embeddings with PyTorch
Python
8
star
30

lonePatient.github.io

HTML
6
star
31

kaggle-camera-model-identification

IEEE's Signal Processing Society - Camera Model Identification
Python
6
star
32

tensorflow-eager-examples

Examples of Eager Execution in tensorflow
Python
6
star
33

char-cnn-text-classification

This repo contains a PyTorch implementation of a char-level CNN model for text classification.
Python
3
star