NIID-Bench

This is the code of paper Federated Learning on Non-IID Data Silos: An Experimental Study.

This code runs a benchmark for federated learning algorithms under non-IID data distribution scenarios. Specifically, we implement 4 federated learning algorithms (FedAvg, FedProx, SCAFFOLD & FedNova), 3 types of non-IID settings (label distribution skew, feature distribution skew & quantity skew) and 9 datasets (MNIST, Cifar-10, Fashion-MNIST, SVHN, Generated 3D dataset, FEMNIST, adult, rcv1, covtype).

Updates on NIID-Bench

Implement partition.py to divide tabular datasets (csv format) into multiple files using our non-IID partitioning strategies. Column Class in the header is recognized as label. See an running example in partition_to_file.sh. The example dataset is Credit Card Fraud Detection.

To adapt to your own tabular dataset in ​partition.py, you need the following steps:

1. Load your own dataset in arrays. Replace Line 117-126.
2. The whole tabular dataset is stored in ​dataset​​. The label column ID is stored in class_id​​. Change Line 130 to your own label identifier.

If your dataset is image dataset, partition.py​​ is no longer applicable. You can refer to our function partition_data​​ in utils.py. You need to design your own dataloader like Line 183-198. For example, in load_mnist_data (Line 40), you need to write a dataloader to return your dataset as tuple (X_train, y_train, X_test, y_test). In terms of the dataloader format, you can refer to class MNIST_truncated​​ (Line 60 in dataset.py). After you get (X_train, y_train, X_test, y_test), the partition_data function will return the ​net_dataidx_map​.

To support more settings and faciliate future researches, we now integrate MOON. We add CIFAR-100 and Tiny-ImageNet.

Tiny-ImageNet

You can download Tiny-ImageNet here. Then, you can follow the instructions to reformat the validation folder.

Non-IID Settings

Label Distribution Skew

• Quantity-based label imbalance: each party owns data samples of a fixed number of labels.
• Distribution-based label imbalance: each party is allocated a proportion of the samples of each label according to Dirichlet distribution.

Feature Distribution Skew

• Noise-based feature imbalance: We first divide the whole dataset into multiple parties randomly and equally. For each party, we add different levels of Gaussian noises.
• Synthetic feature imbalance: For generated 3D data set, we allocate two parts which are symmetric of(0,0,0) to a subset for each party.
• Real-world feature imbalance: For FEMNIST, we divide and assign the writers (and their characters) into each party randomly and equally.

Quantity Skew

• While the data distribution may still be consistent amongthe parties, the size of local dataset varies according to Dirichlet distribution.

Usage

Here is one example to run this code:

python experiments.py --model=simple-cnn \
    --dataset=cifar10 \
    --alg=fedprox \
    --lr=0.01 \
    --batch-size=64 \
    --epochs=10 \
    --n_parties=10 \
    --mu=0.01 \
    --rho=0.9 \
    --comm_round=50 \
    --partition=noniid-labeldir \
    --beta=0.5\
    --device='cuda:0'\
    --datadir='./data/' \
    --logdir='./logs/' \
    --noise=0 \
    --sample=1 \
    --init_seed=0

Data Partition Map

You can call function get_partition_dict() in experiments.py to access net_dataidx_map. net_dataidx_map is a dictionary. Its keys are party ID, and the value of each key is a list containing index of data assigned to this party. For our experiments, we usually set init_seed=0. When we repeat experiments of some setting, we change init_seed to 1 or 2. The default value of noise is 0 unless stated. We list the way to get our data partition as follow.

• Quantity-based label imbalance: partition=noniid-#label1, noniid-#label2 or noniid-#label3
• Distribution-based label imbalance: partition=noniid-labeldir, beta=0.5 or 0.1
• Noise-based feature imbalance: partition=homo, noise=0.1 (actually noise does not affect net_dataidx_map)
• Synthetic feature imbalance & Real-world feature imbalance: partition=real
• Quantity Skew: partition=iid-diff-quantity, beta=0.5 or 0.1
• IID Setting: partition=homo
• Mixed skew: partition = mixed for mixture of distribution-based label imbalance and quantity skew; partition = noniid-labeldir and noise = 0.1 for mixture of distribution-based label imbalance and noise-based feature imbalance.

Here is explanation of parameter for function get_partition_dict().

Leader Board

Note that the accuracy shows the average of three experiments, while the training curve is based on only one experiment. Thus, there may be some difference. We show the training curve to compare convergence rate of different algorithms.

Quantity-based label imbalance

• Cifar-10, 10 parties, sample rate = 1, batch size = 64, learning rate = 0.01

• Cifar-10, 100 parties, sample rate = 0.1, batch size = 64, learning rate = 0.01

Distribution-based label imbalance

• Cifar-10, 10 parties, sample rate = 1, batch size = 64, learning rate = 0.01

• Cifar-10, 100 parties, sample rate = 0.1, batch size = 64, learning rate = 0.01

Noise-based feature imbalance

• Cifar-10, 10 parties, sample rate = 1, batch size = 64, learning rate = 0.01

Quantity Skew

• Cifar-10, 10 parties, sample rate = 1, batch size = 64, learning rate = 0.01

IID Setting

• Cifar-10, 100 parties, sample rate = 0.1, batch size = 64, learning rate = 0.01

Citation

If you find this repository useful, please cite our paper:

@inproceedings{li2022federated,
      title={Federated Learning on Non-IID Data Silos: An Experimental Study},
      author={Li, Qinbin and Diao, Yiqun and Chen, Quan and He, Bingsheng},
      booktitle={IEEE International Conference on Data Engineering},
      year={2022}
}