Multi-Node
how to install pigsty on multiple nodes
There is a configuration Tutorial for expanding Pigsty from one node to multiple nodes. While the easiest way is always pre-planing everything and provisioning them in one-pass.
1-node Setup
We already illustrated the 1-node installation in the Quick Start section, which may be the simplest setup.
It is not advised to put all the eggs in one basket, but even this one-node setup can be used for production, as long as an external MinIO / S3 / NFS... is configured for PG as remote backup repo.
There are lots of 1-node config templates for your reference.
2-node Setup
The semi-HA setup
A two-node setup enables database replication and semi-HA capabilities:
While more robust than a single node setup, HA has limitations:
- No automatic failover if
node-1fails - manual promotion ofnode-2required - Automatic failover works if
node-2fails -node-1gets promoted automatically
This "semi-HA" setup can only auto-recover from specific node failures.
You can use the dual.yml config template, and provision required VM with:
Vagrant dual.rb to provion this environment.
3-node Setup
The true HA
A true HA setup that can automatically recover from any single node failure:
| ID | IP Address | NODE | PGSQL | INFRA | ETCD |
|---|---|---|---|---|---|
| 1 | 10.10.10.10 | node-1 | pg-meta-1 | infra-1 | etcd-1 |
| 2 | 10.10.10.11 | node-1 | pg-meta-1 | infra-2 | etcd-2 |
| 3 | 10.10.10.12 | node-2 | pg-meta-1 | infra-3 | etcd-3 |
You can use the trio.yml config template, and provision required VM with:
Vagrant trio.rb to provion this environment.
4-node Sandbox
This is a sandbox demo environment used in Pigsty, which has one infra node and three extra data nodes:
| ID | IP Address | NODE | PGSQL | INFRA | ETCD | MINIO |
|---|---|---|---|---|---|---|
| 1 | 10.10.10.10 | meta | pg-meta-1 | infra-1 | etcd-1 | minio-1 |
| 2 | 10.10.10.11 | node-1 | pg-test-1 | |||
| 3 | 10.10.10.12 | node-2 | pg-test-1 | |||
| 4 | 10.10.10.13 | node-3 | pg-test-1 |
You can use the full.yml config template, and provision required VM with:
Vagrant full.rb or Terraform full.tf to provion this environment.
5-node Building
This pro.yml is a 5-node building enviroinment which contains supported Linux distros.
| ID | IP Address | NODE | PGSQL | INFRA | ETCD |
|---|---|---|---|---|---|
| 1 | 10.10.10.8 | el8 | el8-1 | infra-1 | |
| 2 | 10.10.10.9 | el9 | el9-1 | infra-2 | etcd-2 |
| 3 | 10.10.10.12 | u12 | u12-1 | infra-3 | |
| 4 | 10.10.10.22 | u22 | u22-1 | infra-4 | |
| 5 | 10.10.10.24 | u24 | u24-1 | infra-5 |
You can use the Vagrant pro.rb or Terraform pro.tf to provion this environment.
36-node Simulation
A production simulation environment (simu.yml) with 36 nodes covering all Pigsty components
| IP Address | SPEC | NODE | PGSQL | INFRA | ETCD | MINIO | REDIS |
|---|---|---|---|---|---|---|---|
10.10.10.10 | 8C32G | meta1 | pg-meta-1 | infra-1 | |||
10.10.10.11 | 8C32G | meta2 | pg-meta-2 | infra-2 | |||
10.10.10.12 | 2C4G | pg12 | pg-v12-1 | ||||
10.10.10.13 | 2C4G | pg13 | pg-v13-1 | ||||
10.10.10.14 | 2C4G | pg14 | pg-v14-1 | ||||
10.10.10.15 | 2C4G | pg15 | pg-v15-1 | ||||
10.10.10.16 | 2C4G | pg16 | pg-v16-1 | ||||
10.10.10.17 | 2C4G | pg17 | pg-v17-1 | ||||
10.10.10.18 | 2C4G | proxy1 | |||||
10.10.10.19 | 2C4G | proxy2 | |||||
10.10.10.21 | 2C4G | minio1 | etcd-1 | minio-1 | redis-meta-1 | ||
10.10.10.22 | 2C4G | minio2 | etcd-2 | minio-2 | redis-meta-2 | ||
10.10.10.23 | 2C4G | minio3 | etcd-3 | minio-3 | redis-meta-3 | ||
10.10.10.24 | 2C4G | minio4 | etcd-4 | minio-4 | redis-meta-4 | ||
10.10.10.25 | 2C4G | minio5 | etcd-5 | minio-5 | redis-meta-5 | ||
10.10.10.40 | 1C2G | node40 | pg-pitr-1 | ||||
10.10.10.41 | 1C2G | node41 | pg-test-1 | redis-test-1 | |||
10.10.10.42 | 1C2G | node42 | pg-test-2 | redis-test-2 | |||
10.10.10.43 | 1C2G | node43 | pg-test-3 | redis-test-3 | |||
10.10.10.44 | 1C2G | node44 | pg-test-4 | redis-test-4 | |||
10.10.10.45 | 1C2G | node45 | pg-src-1 | redis-src-1 | |||
10.10.10.46 | 1C2G | node46 | pg-src-2 | redis-src-2 | |||
10.10.10.47 | 1C2G | node47 | pg-src-3 | redis-src-3 | |||
10.10.10.48 | 1C2G | node48 | pg-dst-1 | redis-dst-1 | |||
10.10.10.49 | 1C2G | node49 | pg-dst-2 | redis-dst-2 | |||
10.10.10.50 | 1C2G | node50 | pg-citus0-1 | ||||
10.10.10.51 | 1C2G | node51 | pg-citus0-2 | ||||
10.10.10.52 | 1C2G | node52 | pg-citus1-1 | ||||
10.10.10.53 | 1C2G | node53 | pg-citus1-2 | ||||
10.10.10.54 | 1C2G | node54 | pg-citus2-1 | ||||
10.10.10.55 | 1C2G | node55 | pg-citus2-2 | ||||
10.10.10.56 | 1C2G | node56 | pg-citus3-1 | ||||
10.10.10.57 | 1C2G | node57 | pg-citus3-2 | ||||
10.10.10.58 | 1C2G | node58 | pg-citus4-1 | ||||
10.10.10.59 | 1C2G | node59 | pg-citus4-2 | ||||
10.10.10.88 | 4C8G | test |
You can use the Vagrant simu.rb to provion this environment.
You can run the entre simulation on a real server (72C / 256G) with libvirt as VM provider with vagrant.
- 2 infra nodes, monitoring each other
- 2 dedicate proxy nodes that run haproxy
- 5-node etcd cluster which tolerates 2 node failures, and 5-node redis sentinel cluster
- 5-node minio cluster with 4 disks on each node
- 10 postgres clusters,
pg13-pg15,pg-src,pg-dst,pg-pitr,pg-test - 10-node citus cluster with 5 shareds
- redis standalone cluster
redis-srcandredis-dst, and native clusterredis-test