• Stars
    star
    192
  • Rank 202,019 (Top 4 %)
  • Language
    C++
  • License
    Apache License 2.0
  • Created about 3 years ago
  • Updated 12 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

You can refer to Readme-CN for introduction in Chinese.

What is PFS

The PolarDB File System (hereafter simplified as PFS or PolarFS) is a high-performance Distributed File System in User Space that is developed by Alibaba Cloud and used in PolarDB. PFS follows the standards of Portable Operating System Interface (POSIX).

Quick Start

PFS for PostgreSQL uses the background process pfsdaemon to provide services. PFS for PostgreSQL is developed and tested based on AliOS and CentOS 7.5. Theoretically, PFS for PostgreSQL can also be built based on other Linux versions.

Install Dependencies

In the following example, CentOS 7.5 is selected. Before you build PFS for PostgreSQL, install the following software:

  • CMake: The CMake version must be 2.8 or later.
  • GCC or G++: The GNU Compiler Collection (GCC) version or the GNU C++ Compiler (G++) version must be 4.8.5 or later.
  • zlog: The zlog version must be 1.2.12 or later.
  • libaio-devel

We recommend that you use yum or apt-get command to install CMake, GCC or G++, and libaio-devel. To install zlog, you must download the source code and run make && sudo make install command. zlog is installed in the /usr/local/lib directory. If the dynamic libraries cannot be located when pfsdaemon is running, you can run ldconfig command to add /usr/local/lib directory into the settings of dynamic libraries.

We also provide you installation by rpm package. If you use rpm package installation, skip the two steps of "Compile" and "Install pfsdaemon".

Compile

After the dependencies are installed, go to the root directory of PFS source code and run the autobuild.sh script to compile PFS.

./autobuild.sh

Install pfsdaemon

To install or uninstall pfsdaemon, you must be granted the root permissions. After you compile PFS, run the install.sh script to automatically install pfsdaemon.

sudo ./install.sh

Run pfsdaemon

1. Format the storage devices of PFS.

First, run the following command to find the existing block devices:

lsblk

โ€‹ Then, select the block device that you want to format, such as nvme1n1, and run the following command to format the device:

sudo pfs -C disk mkfs nvme1n1
2. Run the following command to start pfsdaemon:
sudo /usr/local/polarstore/pfsd/bin/start_pfsd.sh -p nvme1n1

โ€‹ -p nvme1n1 is a parameter that specifies the device name, and it is required. โ€‹ The following parameters are optional in the command:

-f (not daemon mode)
-w #nworkers
-c log_config_file
-b (if bind cpuset)
-e db ins id
-a shm directory
-i #inode_list_size
3. Run the following command to stop pfsdaemon:
sudo /usr/local/polarstore/pfsd/bin/stop_pfsd.sh nvme1n1

nvme1n1 specifies the device name.

4. Clear the files that are generated when pfsdaemon is running.

After stopping pfsdaemon, run the following command to clear the temporary files, logs and shared memory files that are generated when pfsdaemon is running:

sudo /usr/local/polarstore/pfsd/bin/clean_pfsd.sh nvme1n1

nvme1n1 specifies the device name.

5. Use the PFS tool to check whether PFS is running as expected.

Perform common operations on files to verify that PFS is running as expected. For more information, see Instruction to the PFS tool . For example, you can run the following commands to view the new file hello.txt:

sudo pfs -C disk touch /nvme1n1/hello.txt
sudo pfs -C disk ls /nvme1n1/

โ€‹ nvme1n1 specifies the device name.

Uninstall pfsdaemon

To uninstall pfsdaemon, you must be granted the root permissions.

1. Run the following command to stop pfsdaemon:
sudo /usr/local/polarstore/pfsd/bin/stop_pfsd.sh nvme1n1

nvme1n1 specifies the device name.

2. Run the uninstall.sh script to uninstall pfsdaemon.
sudo ./uninstall.sh

Documentation

The doc folder includes the following file:

Software License

PFS is developed based on the open source software license Apache License 2.0.

Publications

  • PolarFS: An Ultra-low Latency and Failure Resilient Distributed File System for Shared Storage Cloud Database in VLDB 2018
  • POLARDB Meets Computational Storage: Efficiently Support Analytical Workloads in Cloud-Native Relational Database in FAST 2020

Contact us

  • For more information about the ApsaraDB PolarDB PostgreSQL-compatible edition, see PolarDB Official Site.
  • Use the DingTalk application to scan the following QR code and join the DingTalk group.

โ€‹

More Repositories

1

PolarDB-for-PostgreSQL

A cloud-native database based on PostgreSQL developed by Alibaba Cloud.
C
2,830
star
2

galaxysql

PolarDB-X is a cloud native distributed SQL Database designed for high concurrency, massive storage, complex querying scenarios.
Java
943
star
3

galaxyengine

GalaxyEngine is a MySQL branch originated from Alibaba Group, especially supports large-scale distributed database system.
C++
170
star
4

galaxycdc

GalaxyCDC is a core component of PolarDB-X which is responsible for global binary log generation, publication and subscription.
Java
45
star
5

galaxykube

PolarDB-X Operator is a Kubernetes extension that aims to create and manage PolarDB-X cluster on Kubernetes.
Go
44
star
6

PolarDB-X

PolarDB-X is a cloud native distributed SQL Database designed for high concurrency, massive storage, complex querying scenarios.
Makefile
37
star
7

galaxyglue

GalaxyGlue is an extension to MySQL Connector/J 8.0.
Java
27
star
8

PolarDB-Stack-Operator

PolarDB Stack is a DBaaS implementation for PolarDB-for-Postgres, as an operator creates and manages PolarDB/PostgreSQL clusters running in Kubernetes. It provides re-construct, failover swtich-over, scale up/out, high-available capabilities for each clusters.
Go
24
star
9

PolarDB-NodeAgent

PolarDB-NodeAgent is a light-weight and flexible agent for data collection, which supports performance data collection from hosts and instances. It is a plug-in process running on physical machines or virtual machines, collecting performance data every second and real-time logs of all the instances (both containerized instances and non-containerized instances) on the machine.
Go
15
star
10

PolarDB-Stack-Workflow

Go
13
star
11

PolarDB-Stack-Storage

Go
11
star
12

PolarDB-ClusterManager

PolarDB Cluster Manager is the cluster management component of PolarDB for PostgreSQL, responsible for topology management, high availability, configuration management, and plugin extensions.
Go
11
star
13

PolarDB-Stack-Daemon

PolarStack-Daemon is a daemon process in DBaaS PolarStack. It runs on all hosts and is responsible for port status collection, db log clear, db engine images availability collection, host network status collection. It provides the basic host information and status for db cluster creating/migrating/state recognition and running.
Go
10
star
14

PolarDB-Stack-Common

Go
9
star
15

PolarDB-Hands-On

C
6
star
16

polardbx-backup

polardbx-backup is a hot backup tool for PolarDB-X
C++
3
star
17

polardb-pg-docker-images

Shell
3
star
18

PolarDB-Hackathon-2023

3
star
19

PolarDB-ImageBuilder

Python
3
star
20

galaxysql-tools

Tools for PolarDB-X, such as data migration, CSV files import, benchmark, etc.
Java
3
star
21

learn-some-polardb-x

Java
1
star
22

PolarDB-BackupAgent

A distributed, high performance and high avaliable backup agent for polardb pg which has rich features and is easy to extend storage plugins.
C
1
star