• Stars
    star
    226
  • Rank 176,514 (Top 4 %)
  • Language
    C
  • License
    Other
  • Created over 7 years ago
  • Updated over 4 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

new heap with in-place update, powered by undo, for eventual integration into PostgreSQL

The purpose of this document is to let users know how they can use zheap (a new storage format for PostgreSQL) and the work that is still pending. This new storage format provides a better control over bloat, reduces the tuple size and reduces the write amplification. The detail design of zheap is present in zheap design document (src/backend/access/zheap/README).

How do I use zheap?

We have provided a storage engine option which you can set when creating a table. For example:

create table t_zheap(c1 int, c2 varchar) USING zheap;

Index creation for zheap tables doesn't need any special syntax.

You can also set the GUC parameter default_table_access_method. The default value is “heap", but you can set it to “zheap”. If you do, all subsequently-created tables will use zheap.

These interfaces will probably change once the storage format API work is integrated into PostgreSQL. We’ll adjust this code to use whatever interfaces are agreed by the PostgreSQL community.

Each zheap page has fixed set of transaction slots each of which contains the transaction information (transaction id and epoch) and the latest undo record pointer for that transaction. By default, we have four transaction slots per page, but this can be changed by setting --with-trans_slots_per_zheap_page=value while configuring zheap.

What doesn’t work yet?

  • Logical decoding
  • Snapshot too old - We might want to implement this after first version is committed as this will work differently for zheap.
  • Alter Table <table_name> Set Tablesapce <tbs_name> - For this feature to work correctly in zheap, while copying pages, we need to ensure that pending aborts gets applied before copying the page.

Tools

  • pg_undo_dump similar to pg_wal_dump: We would like to develop this utility as it can be used to view undo record contents and can help us debug problems related to undo chains.
  • We also want to develop tools like pgstattuple, pgrowlocks that allow us to inspect the contents of database pages at a low level.
  • wal consistency checker: This will be used to check for bugs in the WAL redo routines. Currently, it is quite similar to what we have in current heap, but we want to extend it to check the consistency of undo pages similar to how it checks for data and index pages.

Open Issues

  • Currently, the TPD pages are not added to FSM even if they can be completely reused.
  • Single user mode: This needs some investigation as to what exactly is required. I think we need to ensure that undo gets applied without the need to invoke undo worker.

The other pending code related items are tracked on zheap wiki page: https://wiki.postgresql.org/wiki/Zheap

Test run before pushing code

  • make check-world
  • make installcheck and isolation test (with default_table_access_method = 'zheap')
  • make installcheck with hot standby (with wal_consistency_checking=all)

You can find overall design of zheap in the README: src/backend/access/zheap/README

More Repositories

1

repmgr

A lightweight replication manager for PostgreSQL (Postgres)
C
1,556
star
2

mysql_fdw

PostgreSQL foreign data wrapper for MySQL
C
477
star
3

pg_nosql_benchmark

A tool which can be used to benchmark PostgreSQL (JSONB) and MongoDB (BSON)
Shell
217
star
4

hdfs_fdw

PostgreSQL foreign data wrapper for HDFS
C
133
star
5

pg_catcheck

Postgres Catalog Checker
C
77
star
6

pg_failover_slots

PG Failover Slots extension
C
76
star
7

pldebugger

Procedural Language Debugger Plugin for PostgreSQL and EDB Postgres Advanced Server
C
70
star
8

system_stats

A Postgres extension for exposing system metrics such as CPU, memory and disk information
C
70
star
9

postgres-deployment

EDB Postgres Deployment
Python
62
star
10

edb-ansible

Ansible code for deploying EDB Postgres database clusters and related products.
Python
59
star
11

tpa

Trusted Postgres Architect
Python
57
star
12

cnp-bench

Benchmarking guidelines for Cloud Native PostgreSQL
Smarty
25
star
13

docs

EDB Docs
MDX
17
star
14

docker-postgresql

Repo for PostgreSQL containers.
Shell
17
star
15

edb-ref-archs

EDB Reference Architectures
15
star
16

benchmark-framework

Ansible based configurable framework for running consistent pgbench, TPROC-C and TPROC-H benchmarks
Shell
15
star
17

benchmarking-postgres

Scripts to benchmark PostgreSQL database server using pgbench and HammerDB.
Shell
12
star
18

terraform-provider-biganimal

A terraform provider that uses the BigAnimal.com API in order to manage postgres clusters in the cloud
Go
12
star
19

cnp-sandbox

Sandbox for Cloud Native PostgreSQL in Kubernetes environments
11
star
20

edb-installers

PostgreSQL installers packaged by EDB
C
9
star
21

edb-postgres-for-kubernetes-charts

Helm charts to deploy Cloud Native PostgreSQL
Smarty
8
star
22

sslutils

A Postgres extension for managing SSL certificates through SQL.
C
7
star
23

edb-cnpg-gke-autopilot-deployer

Deployer image for the CNP GKE Autopilot integration
Shell
5
star
24

pgldapsync

A tool for syncing LDAP users to Postgres Roles
Python
5
star
25

cloud-utilities

Shell
5
star
26

kubectl-cnp

Plugin for kubectl to manage a Cloud Native PostgreSQL cluster in Kubernetes
Shell
4
star
27

pgworkload

A workload generator for Postgres.
C++
4
star
28

k8s-api-docgen

Reads Go source files to extract metadata about Kubernetes API custom resources defined by the developer in Godoc and produces different kinds of output - such as JSON or Markdown.
Go
4
star
29

edb-terraform

Terraform templates aimed to provide easy to use YAML configuration file describing the target cloud infrastructure.
HCL
3
star
30

barman-tutorial

Shell
3
star
31

pg-backup-api

A server that provides an HTTP API to interact with Postgres backups
Python
3
star
32

docker-pgbouncer

Repo for PgBouncer image containers
Dockerfile
2
star
33

bdr-benchmark-kit

BDR Benchmark Kit based on Terraform and Ansible
Python
2
star
34

cto-tproc-results

CTO Team TPROC benchmark results
2
star
35

efm-api-node-state

HTTP service and REST API exposing the state of the current EFM node
Rust
2
star
36

ciclops

Project moved to https://github.com/cloudnative-pg/ciclops
2
star
37

edb-partner-docs

JavaScript
1
star
38

edb-benchmarks

Benchmark execution scripts
Shell
1
star
39

cto-tproc-scripts

CTO Team TPROC benchmark scripts
1
star