• Stars
    star
    236
  • Rank 170,480 (Top 4 %)
  • Language
    Python
  • License
    Apache License 2.0
  • Created over 5 years ago
  • Updated almost 2 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Kaldi model converter to ONNX

Kaldi-ONNX Converter

License Build Status

English | 中文

Kaldi-ONNX is a tool for porting Kaldi Speech Recognition Toolkit neural network models to ONNX models for inference. With the converted ONNX model, you can use MACE to speedup the inference on Android, iOS, Linux or Windows devices with highly optimized NEON kernels (more heterogeneous devices will be supported in the future).

This tool supports converting both Nnet2 and Nnet3 models. Almost all components in Nnet2 and Nnet3 models are supported, and the available components are listed in converter/common.py.

Besides DNN, many speech recognition models are using RNN or TDNN networks. To make them more portable and efficient, this tool will convert these RNN and TDNN networks to DAG-like networks.

Usage

1. Dependencies

pip install -r requirements.txt

2. Prepare models

This tool only supports Kaldi's text model as an input.

If you have a binary model, Kaldi's nnet-am-copy or nnet3-copy tool can help you get a text one:

Nnet2 Model

path/to/kaldi/src/nnet2bin/nnet-am-copy --binary=false          \
                                        final.raw text.mdl

Nnet3 Model

path/to/kaldi/src/nnet3bin/nnet3-copy --binary=false          \
                                      --raw=true              \
                                      --prepare-for-test=true \
                                      final.raw text.mdl

Don't forget to use the --prepare-for-test=true and --raw=trueoption to optimize the model.

Before converting, you need to use nnet-am-info or nnet3-am-info to get left_context\right_context\modulus, which is required for converting.

More details about kaldi's tools are in Kaldi's Documentation.

3. Convert

One command to convert:

python converter/convert.py --input=models/kaldi/model_name.mdl  \
                            --output=models/onnx/model_name.onnx \
                            --transition-model=path/to/save/transition_model.trans \
                            --conf=path/to/save/configuration.conf \
                            --chunk-size=20 \
                            --left-context=left_context \
                            --right-context=right_context \
                            --modulus=modulus \
                            --subsample-factor=subsample_factor \
                            --nnet-type=3

4. Graph review

After converting, there is a graphic tool for you to review the onnx model: ONNX Model Viewer.

5. Deployment

To run the ONNX models, an inference framework with Kaldi specific operators support is needed. Currently, MACE has Kaldi-ONNX support which is primarily optimized for mobile phones and IoT devices.

For more details about deployment, please refer to MACE documents.

6. Validation

Since MACE already supports most frequently used components in Kaldi. We can use it to validate converted Kaldi models. The validation process is giving the same inputs to Kaldi and MACE's computation and check if they will output the same results.

Generate random input

We provide a tool to generate the same random input data for Kaldi and MACE's computation.

python tools/generate_inputs.py --input-dim=40 \
                                --chunk-size=20 \
                                --left-context=left_context \
                                --right-context=right_contex \
                                --kaldi-input_file=path/to/kaldi/input/test_input.ark \
                                --mace-input_file=path/to/mace/input/test_input.npy

'input_dim' and 'chunk_size' are model's input dim and chunk, can be specified by case. 'kaldi_input_file' and 'mace_input_file' are paths for saving generated data files.

Compute in Kaldi

Kaldi has command line tools for computing the model's propogation.

Nnet2:

path/to/kaldi/src/nnet2bin/nnet-am-compute  --chunk-size=chunk_size \
                                            path/to/kaldi/model/file/final.mdl \
                                            ark,t:/path/to/kaldi/input/file/test_input.ark \
                                            ark,t:path/to/save/output/data/test_output.ark

Nnet3:

path/to/kaldi/src/nnet3-compute  --frames-per-chunk=chunk_size \
                                 path/to/kaldi/model/file/final.mdl \
                                 ark,t:/path/to/kaldi/input/file/test_input.ark \
                                 ark,t:path/to/save/output/data/test_output.ark

Convert output data file

After running nnet-compute, we'll get Kaldi's output data in text format . Because MACE supports numpy data file as input or output, we need to convert Kaldi's output data file to numpy format.

The script tools/kaldi_to_mace.py will help you doing this convertion.

python tools/kaldi_to_mace.py  --input=path/to/kaldi/data/file.ark \
                               --output=path/to/save/numpy/data/file.npy

Prepare config file for MACE

Here is an example yaml config file for fisher english model.

# fisher_english.yml

library_name: fisher_english_8
target_abis: [armeabi-v7a, arm64-v8a]
model_graph_format: file
model_data_format: file
models:
  fisher_english_8:
    platform: onnx
    model_file_path: https://cnbj1.fds.api.xiaomi.com/mace/miai-models/onnx/kaldi/nnet2/fisher_english_8_nnet_a.onnx
    model_sha256_checksum: e27d8147995b0a68e1367d060dc4f41c0f434043992a52548ff961e4e1e87e6c
    subgraphs:
      - input_tensors:
          - 0
        input_shapes:
          - 1,20,140
        output_tensors:
          - 17
        output_shapes:
          - 1,20,7880
        backend: kaldi
        input_data_formats: NONE
        output_data_formats: NONE
        validation_inputs_data:
            - https://cnbj1.fds.api.xiaomi.com/mace/miai-models/onnx/kaldi/data/kaldi_input_20_140.npy
        validation_outputs_data:
            - https://cnbj1.fds.api.xiaomi.com/mace/miai-models/onnx/kaldi/data/test_fisher_english_8_20_140_out.npy
    backend: kaldi
    runtime: cpu
    limit_opencl_kernel_time: 0
    nnlib_graph_mode: 0
    obfuscate: 0

Validate

Convert and validate the model in MACE:

cd path/to/mace
python tools/converter.py convert --config=path/to/fisher_english.yml
python tools/converter.py run --config=path/to/fisher_english.yml --validate

The command will give you the similarity between MACE's and Kaldi's output results.

More details about how to use MACE, please step to MACE documents.

5. Examples

We have converted numbers of Kaldi's Nnet2 and Nnet3 models, and put them in MACE Model Zoo.

Communication

  • GitHub issues: bug reports, usage issues, feature requests

License

Apache License 2.0.

Contributing

Python code should conform to PEP8 Style Guide for Python Code.

You can use pycodestyle to check the style.

pycodestyle $(find . -name "*.py")

Any kind of contribution is welcome. For bug reports, feature requests, please just open an issue without any hesitation. For code contributions, it's strongly suggested to open an issue for discussion first.

Acknowledgement

kaldi-onnx learned a lot from the following projects during the development:

More Repositories

1

soar

SQL Optimizer And Rewriter
Go
8,659
star
2

mace

MACE is a deep learning inference framework optimized for mobile heterogeneous computing platforms.
C++
4,922
star
3

open-falcon

A Distributed and High-Performance Monitoring System
3,025
star
4

Gaea

Gaea is a mysql proxy, it's developed by xiaomi b2c-dev team.
Go
2,621
star
5

naftis

An awesome dashboard for Istio built with love.
Go
1,891
star
6

mone

No description, website, or topics provided
Java
1,112
star
7

MiNLP

XiaoMi Natural Language Processing Toolkits
Scala
781
star
8

hiui

HIUI is a solution that is adequate for the fomulation and implementation of interaction and UI design standard for front, middle and backend.
TypeScript
738
star
9

android_tv_metro

android tv metro framework and server API
Java
653
star
10

minos

Minos is beyond a hadoop deployment system.
Python
522
star
11

rose

Rose is not only a framework.
Java
498
star
12

shepher

Java
493
star
13

MiLM-6B

427
star
14

chronos

Network service to provide globally strictly monotone increasing timestamp
Java
399
star
15

LuckyMoneyTool

Java
376
star
16

mace-models

Mobile AI Compute Engine Model Zoo
Python
368
star
17

mobile-ai-bench

Benchmarking Neural Network Inference on Mobile Devices
C++
355
star
18

linden

Java
233
star
19

themis

Themis provides cross-row/cross-table transaction on HBase based on google's percolator.
Java
226
star
20

rdsn

Has been migrated to https://github.com/apache/incubator-pegasus/tree/master/rdsn
C++
144
star
21

StableDiffusionOnDevice

本项目是一个通过文字生成图片的项目,基于开源模型Stable Diffusion V1.5生成可以在手机的CPU和NPU上运行的模型,包括其配套的模型运行框架。
C++
91
star
22

thain

Thain is a distributed flow schedule platform.
TypeScript
81
star
23

ozhera

Application Observable Platform in the Cloud Native Era
Java
72
star
24

misound

MiSound is a Android application making XiaoMi's SoundBar more powerful. EQ, control, player all in one.
Java
64
star
25

galaxy-sdk-java

Java SDK for Xiaomi Structured Datastore Service
Java
63
star
26

C3KG

Python
63
star
27

nnlib

Fork of https://source.codeaurora.org/quic/hexagon_nn/nnlib
C
53
star
28

subllm

This repository is the official implementation of the ECAI 2024 conference paper SUBLLM: A Novel Efficient Architecture with Token Sequence Subsampling for LLM
Python
53
star
29

galaxy-fds-sdk-python

Python SDK for Xiaomi File Data Storage.
Python
51
star
30

jack

Jack is a cluster manager built on top of Zookeeper and thrift.
50
star
31

dasheng

Official PyTorch code for Deep Audio-Signal Holistic Embeddings
Python
46
star
32

cmath

CMATH: Can your language model pass Chinese elementary school math test?
Python
38
star
33

pegasus-rocksdb

Has been migrated to https://github.com/pegasus-kv/rocksdb
C++
34
star
34

cloud-ml-sdk

Python
32
star
35

talos-sdk-golang

Go SDK for Xiaomi Streaming Message Queue
Go
32
star
36

pegasus-java-client

Has been migrated to https://github.com/apache/incubator-pegasus/tree/master/java-client
Java
31
star
37

ECFileCache

Java
30
star
38

mace-kit

C++
27
star
39

pegasus-go-client

Has been migrated to https://github.com/apache/incubator-pegasus/tree/master/go-client
Go
24
star
40

emma

Python
22
star
41

galaxy-fds-sdk-java

Java SDK for Xiaomi File Data Storage.
Java
22
star
42

xiaomi.github.com

JavaScript
21
star
43

CGNet

The official implementation of the ECCV 2024 paper: Continuity Preserving Online CenterLine Graph Learning
Python
20
star
44

galaxy-fds-sdk-android

Android SDK for Xiaomi File Data Storage.
Java
18
star
45

go-fds

Next-generation fds golang sdk
Go
17
star
46

galaxy-fds-sdk-php

PHP SDK for Xiaomi File Data Storage.
PHP
16
star
47

galaxy-sdk-python

Python SDK for Xiaomi Structured Datastore Service
Python
16
star
48

galaxy-sdk-go

Go SDK for Xiaomi Structured Datastore Service
Go
15
star
49

galaxy-hadoop

Hadoop interface for Xiaomi Open Storage
Java
13
star
50

galaxy-thrift-api

Thrift API for Xiaomi Structured Datastore Service
Thrift
12
star
51

galaxy-fds-sdk-cpp

C++ SDK for Xiaomi File Data Storage
C++
11
star
52

galaxy-fds-sdk-javascript

JavaScript
9
star
53

pegasus-python-client

Has been migrated to https://github.com/apache/incubator-pegasus/tree/master/python-client
Python
8
star
54

DetermLR

Open source code for paper
Python
8
star
55

galaxy-sdk-php

PHP SDK for Xiaomi Structured Datastore Service
PHP
8
star
56

pegasus-datax

Provide pegasus plugin in alibaba/DataX, please refer to 'pegasuswriter/doc/pegasuswriter.md'.
Java
8
star
57

galaxy-fds-migration-tool

A MapReduce tool to migrate objects or files parallely between different object storage systems
Java
7
star
58

galaxy-sdk-nodejs

Node.js SDK for Xiaomi Structured Datastore Service
JavaScript
6
star
59

pegasus-YCSB

Provide pegasus plugin in YCSB, please refer to 'Test Pegasus' section in README.
Java
6
star
60

pegasus-nodejs-client

Has been migrated to https://github.com/apache/incubator-pegasus/tree/master/nodejs-client
JavaScript
6
star
61

pegasus-scala-client

Has been migrated to https://github.com/apache/incubator-pegasus/tree/master/scala-client
Scala
6
star
62

PowerTestDemo

Java
5
star
63

galaxy-fds-sdk-ios

ios sdk for galaxy-fds
Objective-C
5
star
64

SiMuST-C

Python
5
star
65

galaxy-sdk-cpp

C++ SDK for Xiaomi Structured Datastore Service
C++
5
star
66

nlpcc-2023-shared-task-9

https://mp.weixin.qq.com/s/pBDvTmr_oOHUPzBhjXG-aw
Python
5
star
67

TED-MMST

1
star
68

PowerTestDemoGlobal

The demo script of Power Consumption Test.
Java
1
star
69

galaxy-sdk-javascript

Javascript SDK for Xiaomi Structured Datastore Service
JavaScript
1
star