Administration

Here are some administration SOP for etcd:

• Create Cluster
• Remove Cluster
• CLI Environment
• Reload Config
• Management Scripts
• Append Member
• Remove Member

Check ETCD: FAQ for more questions.

Create Cluster

To create an etcd cluster, define the etcd cluster in inventory first:

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }
    10.10.10.11: { etcd_seq: 2 }
    10.10.10.12: { etcd_seq: 3 }
  vars: { etcd_cluster: etcd   }

Then run the etcd.yml playbook.

./etcd.yml   # init etcd module on group 'etcd'

If there's an existing etcd cluster, this playbook will update the config and restart all the etcd instances

Pigsty has a safeguard mechanism to prevent accidental purge. By default, etcd_clean is true, and etcd_safeguard is false, which means the playbook will purge etcd cluster even if there are running etcd instances. In this case, etcd.yml is truly idempotent, which is useful for development, testing, and emergency rebuild of etcd cluster in production.

For provisioned etcd cluster in prod env, you can enable safeguard to prevent accidental clean.

Remove Cluster

To remove an existing etcd cluster, you can use the dedicated etcd-rm.yml:

./etcd-rm.yml   # remove the default etcd cluster

bin/etcd-rm     # remove the default etcd cluster

If the etcd_safeguard is set to true, the playbook will abort.

CLI Environment

Pigsty use etcd v3 API by default. (v2 support is dropped since v3.6.0)

Here’s an example of client environment config.

alias e="etcdctl"
alias em="etcdctl member"
export ETCDCTL_ENDPOINTS=https://10.10.10.10:2379
export ETCDCTL_CACERT=/etc/pki/ca.crt
export ETCDCTL_CERT=/etc/etcd/server.crt
export ETCDCTL_KEY=/etc/etcd/server.key

You can do CRUD with the following commands after setting up the envs:

e put a 10 ; e get a; e del a ; # V3 API

Reload Config

In case of permanent etcd cluster membership changes, You’ll have to refresh the 4 etcd endpoints references:

• config file of existing etcd members and client env var
• patroni dcs endpoint config
• vip-manager dcs endpoint config

To refresh etcd config file /etc/etcd/etcd.conf on existing members and client env vars:

./etcd.yml -t etcd_conf         # refresh /etc/etcd/etcd.conf with latest status
./etcd.yml -t etcd_launch -f 1  # restart etcd instances one by one

Update patroni reference to etcd endpoints:

./pgsql.yml -t pg_conf                                  # re-gen patroni config
./pgsql.yml -t patroni_reload -e patroni_reload=true    # reload patroni config

Update vip-manager reference to etcd endpoints (if you are using PGSQL L2 VIP):

./pgsql.yml -t pg_vip # reload vip-manager config

Append Member

ETCD Reference: Add a member

Pigsty can perform etcd cluster expansion with bin/etcd-add script or the etcd.yml playbook.

bin/etcd-add <ip>

You can add new members to existing etcd cluster in 5 steps:

1. issue etcdctl member add command to tell existing cluster that a new member is coming (use learner mode)
2. update inventory group etcd with new instance
3. init the new member with etcd_init=existing, to join the existing cluster rather than create a new one (VERY IMPORTANT)
4. promote the new member from leaner to follower
5. update etcd endpoints reference with reload-config

Manual Approach

etcdctl member add <etcd-?> --learner=true --peer-urls=https://<new_ins_ip>:2380
./etcd.yml -l <new_ins_ip> -e etcd_init=existing
etcdctl member promote <new_ins_server_id>

Automated Approach (Recommended)

Use the bin/etcd-add script to simplify the process:

# Add new members to inventory first, then:
bin/etcd-add <ip1> <ip2> ...  # append specific members to existing cluster

The etcd-add script will:

• Validate IP addresses
• Execute the etcd.yml playbook with appropriate parameters
• Provide safety warnings and countdown timers
• Guide you through post-operation configuration updates

Remove Member

To remove a member from existing etcd cluster, you have two approaches:

Automated Approach (Recommended)

Use the bin/etcd-rm script for simplified removal:

bin/etcd-rm <ip> ...    # remove specific members

Or use the dedicated removal playbook:

./etcd-rm.yml -l <ip>         # remove specific member

Manual Approach

For manual removal, it usually takes 3 steps:

1. remove/uncomment it from inventory and reload config
2. remove it with etcdctl member remove <server_id> command and kick it out of the cluster
3. use the etcd-rm.yml playbook to clean up the instance

Removal Parameters

The etcd_remove role supports several configuration options:

• etcd_safeguard=true: Prevents accidental removal
• etcd_rm_data=true: Removes etcd data directories (default: true)
• etcd_rm_pkg=false: Uninstalls etcd packages (default: false)

Example with custom parameters:

./etcd-rm.yml -l <ip> -e etcd_rm_pkg=true  # also remove packages