# Высокодоступный кластер  ## Установка etcd ЗАМЕЧАНИЕ!!! Далее по тексту вместо переменных - ${HOST1_IP} нужно вставить ip vs01 - ${HOST2_IP} нужно вставить ip vs02 - ${HOST3_IP} нужно вставить ip vs03 !!! 1. Устанавливаем требуемые пакеты ```bash vs01$ yum install etcd patroni patroni-etcd -y ``` ```bash vs02$ yum install patroni patroni-etcd -y ``` ```bash vs03$ yum install patroni patroni-etcd -y ``` 2. Проверяем установку ```bash vs01$ yum list installed etcd patroni patroni-etcd Loaded plugins: fastestmirror, protectbase Installed Packages etcd.x86_64 3.3.11-2.el7.centos @extras patroni.x86_64 2.1.1-1.rhel7 @pgdg-common patroni-etcd.x86_64 2.1.1-1.rhel7 @pgdg-common ``` ```bash vs02$ yum list installed patroni patroni-etcd Loaded plugins: fastestmirror, protectbase Installed Packages patroni.x86_64 2.1.1-1.rhel7 @pgdg-common patroni-etcd.x86_64 2.1.1-1.rhel7 @pgdg-common ``` ```bash vs03$ yum list installed patroni patroni-etcd Loaded plugins: fastestmirror, protectbase Installed Packages patroni.x86_64 2.1.1-1.rhel7 @pgdg-common patroni-etcd.x86_64 2.1.1-1.rhel7 @pgdg-common ``` 3. Кроме etcd patroni поддерживает и другие распределенные хранилища. ```bash vs01$ yum search patroni Loaded plugins: fastestmirror, protectbase Loading mirror speeds from cached hostfile * epel: fedora-epel.koyanet.lv 0 packages excluded due to repository protections ============================= N/S matched: patroni ============================= patroni-aws.x86_64 : Related components to use patroni on AWS patroni-consul.x86_64 : Related components to use patroni with Consul patroni-etcd.x86_64 : Related components to use patroni with etcd patroni-zookeeper.x86_64 : Related components to use patroni with Zookeeper patroni.x86_64 : A Template for PostgreSQL HA with ZooKeeper, etcd or Consul Name and summary matches only, use "search all" for everything. ``` 4. Настраиваем etcd. Заменяем все упоминания localhost на IP сервера. А в параметре ETCD_LISTEN_CLIENT_URLS ставим слушать локальный адрес и IP адрес ```bash vs01$ sed -i 's/localhost/${HOST1_IP}/' /etc/etcd/etcd.conf vs01$ sed -i 's/ETCD_LISTEN_CLIENT_URLS=\"http:\/\/${HOST1_IP}:2379\"/ETCD_LISTEN_CLIENT_URLS=\"http:\/\/${HOST1_IP}:2379,http:\/\/localhost:2379\"/' /etc/etcd/etcd.conf ``` ЗАМЕЧАНИЕ!!! Вместо переменной ${HOST1_IP} нужно поставить ip адреса сервера vs01!!! 5. Проверяем, что заменили правильно ```bash vs01$ cat /etc/etcd/etcd.conf | grep ETCD_LISTEN_CLIENT_URLS ETCD_LISTEN_CLIENT_URLS="http://172.25.27.10:2379,http://localhost:2379" ``` 6. Запускаем службу etcd ```bash vs01$ service etcd start Redirecting to /bin/systemctl start etcd.service ``` 7. Ждем 10 секунд, когда запустится сервис и проверяем его статус ```bash vs01$ sleep 10 vs01$ service etcd status Redirecting to /bin/systemctl status etcd.service ● etcd.service - Etcd Server Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled) Active: active (running) since Thu 2021-09-23 17:47:27 UTC; 16s ago Main PID: 2210 (etcd) CGroup: /system.slice/etcd.service └─2210 /usr/bin/etcd --name=default --data-dir=/var/lib/etcd/default.etcd --listen-client-urls=http://172.25.27.10:2379,http://localhost:2379 Sep 23 17:47:27 vs01.s013162.slurm.io etcd[2210]: 8e9e05c52164694d became leader at term 2 Sep 23 17:47:27 vs01.s013162.slurm.io etcd[2210]: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 2 Sep 23 17:47:27 vs01.s013162.slurm.io etcd[2210]: setting up the initial cluster version to 3.3 Sep 23 17:47:27 vs01.s013162.slurm.io etcd[2210]: set the initial cluster version to 3.3 Sep 23 17:47:27 vs01.s013162.slurm.io etcd[2210]: enabled capabilities for version 3.3 Sep 23 17:47:27 vs01.s013162.slurm.io etcd[2210]: published {Name:default ClientURLs:[http://172.25.27.10:2379]} to cluster cdf818194e3a8c32 Sep 23 17:47:27 vs01.s013162.slurm.io etcd[2210]: ready to serve client requests Sep 23 17:47:27 vs01.s013162.slurm.io etcd[2210]: serving insecure client requests on 172.25.27.10:2379, this is strongly discouraged! Sep 23 17:47:27 vs01.s013162.slurm.io etcd[2210]: ready to serve client requests Sep 23 17:47:27 vs01.s013162.slurm.io etcd[2210]: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged! ``` ## Установка patroni на vs01 8. Настраиваем конфигурацию patroni. Создаем папку /etc/patroni и файл /etc/patroni/patroni.yml ```bash vs01$ rm -rf /etc/patroni vs01$ mkdir /etc/patroni vs01$ touch /etc/patroni/patroni.yml ``` 9. Вносим в конфигурационный файл /etc/patroni/patroni.yml следующие данные ```bash vs01$ echo 'scope: cluster #namespace: /service/ name: postgresql0 restapi: listen: ${HOST1_IP}:8008 connect_address: ${HOST1_IP}:8008 etcd: #Provide host to do the initial discovery of the cluster topology: host: ${HOST1_IP}:2379 #Or use "hosts" to provide multiple endpoints #Could be a comma separated string: #hosts: host1:port1,host2:port2 #or an actual yaml list: #hosts: #- host1:port1 #- host2:port2 #Once discovery is complete Patroni will use the list of advertised clientURLs #It is possible to change this behavior through by setting: #use_proxies: true bootstrap: # this section will be written into Etcd:///config after initializing new cluster # and all other cluster members will use it as a ''global configuration'' dcs: ttl: 30 loop_wait: 10 retry_timeout: 10 maximum_lag_on_failover: 1048576 # master_start_timeout: 300 # synchronous_mode: false #standby_cluster: #host: 127.0.0.1 #port: 1111 #primary_slot_name: patroni postgresql: use_pg_rewind: true # use_slots: true parameters: # wal_level: hot_standby # hot_standby: "on" # max_connections: 100 # max_worker_processes: 8 # wal_keep_segments: 8 # max_wal_senders: 10 # max_replication_slots: 10 # max_prepared_transactions: 0 # max_locks_per_transaction: 64 # wal_log_hints: "on" # track_commit_timestamp: "off" # archive_mode: "on" # archive_timeout: 1800s # archive_command: mkdir -p ../wal_archive && test ! -f ../wal_archive/%f && cp %p ../wal_archive/%f # recovery_conf: # restore_command: cp ../wal_archive/%f %p # some desired options for ''initdb'' initdb: # Note: It needs to be a list (some options need values, others are switches) - encoding: UTF8 - data-checksums pg_hba: # Add following lines to pg_hba.conf after running ''initdb'' # For kerberos gss based connectivity #- host replication replicator 127.0.0.1/32 gss include_realm=0 #- host all all 0.0.0.0/0 gss include_realm=0 - host replication replicator 127.0.0.1/32 md5 - host replication replicator 0.0.0.0/0 md5 - host all all 0.0.0.0/0 md5 # - hostssl all all 0.0.0.0/0 md5 # Additional script to be launched after initial cluster creation (will be passed the connection URL as parameter) # post_init: /usr/local/bin/setup_cluster.sh # Some additional users users which needs to be created after initializing new cluster users: admin: password: admin options: - createrole - createdb postgresql: listen: ${HOST1_IP}:5432 connect_address: ${HOST1_IP}:5432 data_dir: /var/lib/pgsql/12/main bin_dir: /usr/pgsql-12/bin/ # config_dir: pgpass: /tmp/pgpass0 authentication: replication: username: replicator password: replicator superuser: username: postgres password: postgres rewind: # Has no effect on postgres 10 and lower username: postgres password: postgres # Server side kerberos spn # krbsrvname: postgres parameters: # Fully qualified kerberos ticket file for the running user # same as KRB5CCNAME used by the GSS # krb_server_keyfile: /var/spool/keytabs/postgres unix_socket_directories: ''.'' # Additional fencing script executed after acquiring the leader lock but before promoting the replica #pre_promote: /path/to/pre_promote.sh #watchdog: # mode: automatic # Allowed values: off, automatic, required # device: /dev/watchdog # safety_margin: 5 tags: nofailover: false noloadbalance: false clonefrom: false nosync: false' > /etc/patroni/patroni.yml ``` ЗАМЕЧАНИЕ!!! Вместо переменной ${HOST1_IP} нужно поставить ip адреса сервера vs01!!! 10. Проверяем, что ошибок при создании и внесении информации в файл /etc/patroni/patroni.yml нет ```bash vs01$ ls -la /etc/patroni/patroni.yml -rw-r--r-- 1 root root 3653 Sep 23 17:50 /etc/patroni/patroni.yml ``` 11. Запускаем сервер patroni ```bash vs01$ service patroni start Redirecting to /bin/systemctl start patroni.service ``` 12. Ждем 20 секунд и проверяем статус сервера patroni ```bash vs01$ sleep 20 vs01$ service patroni status Redirecting to /bin/systemctl status patroni.service ● patroni.service - Runners to orchestrate a high-availability PostgreSQL Loaded: loaded (/usr/lib/systemd/system/patroni.service; disabled; vendor preset: disabled) Active: active (running) since Thu 2021-09-23 17:50:58 UTC; 26s ago Main PID: 2351 (patroni) CGroup: /system.slice/patroni.service ├─2351 /usr/bin/python3 /usr/bin/patroni /etc/patroni/patroni.yml ├─2386 /usr/pgsql-12/bin/postgres -D /var/lib/pgsql/12/main --config-file=/var/lib/pgsql/12/main/postgresql.conf --listen_addresses=172.25.27.10 --port=5432 --cluster_name=cluster --wal_level=replica --hot_standby=on --max_connections=100 --max_wal_senders=10 --max_prepared_transactions=0 --max_locks_per_transaction=64 --track_commit_timestamp=off --max_replication_slots=10 --max_worker_processes=8 --wal_log_hints=on ├─2388 postgres: cluster: logger ├─2393 postgres: cluster: checkpointer ├─2394 postgres: cluster: background writer ├─2395 postgres: cluster: walwriter ├─2396 postgres: cluster: autovacuum launcher ├─2397 postgres: cluster: stats collector ├─2398 postgres: cluster: logical replication launcher └─2401 postgres: cluster: postgres postgres 172.25.27.10(50054) idle Sep 23 17:51:02 vs01.s013162.slurm.io patroni[2351]: 172.25.27.10:5432 - rejecting connections Sep 23 17:51:02 vs01.s013162.slurm.io patroni[2351]: 172.25.27.10:5432 - rejecting connections Sep 23 17:51:03 vs01.s013162.slurm.io patroni[2351]: 172.25.27.10:5432 - accepting connections Sep 23 17:51:03 vs01.s013162.slurm.io patroni[2351]: 2021-09-23 17:51:03,939 INFO: establishing a new patroni connection to the postgres cluster Sep 23 17:51:03 vs01.s013162.slurm.io patroni[2351]: 2021-09-23 17:51:03,952 INFO: running post_bootstrap Sep 23 17:51:03 vs01.s013162.slurm.io patroni[2351]: 2021-09-23 17:51:03,964 WARNING: Could not activate Linux watchdog device: "Can't open watchdog device: [Errno 2] No such file or directory: '/dev/watchdog'" Sep 23 17:51:03 vs01.s013162.slurm.io patroni[2351]: 2021-09-23 17:51:03,977 INFO: initialized a new cluster Sep 23 17:51:13 vs01.s013162.slurm.io patroni[2351]: 2021-09-23 17:51:13,977 INFO: no action. I am (postgresql0) the leader with the lock Sep 23 17:51:14 vs01.s013162.slurm.io patroni[2351]: 2021-09-23 17:51:14,018 INFO: no action. I am (postgresql0) the leader with the lock Sep 23 17:51:24 vs01.s013162.slurm.io patroni[2351]: 2021-09-23 17:51:24,019 INFO: no action. I am (postgresql0) the leader with the lock ``` 13. Запросим версию PostgreSQL и проверим его работоспособность ```bash vs01$ sudo -u postgres -i PGPASSWORD=postgres psql -h ${HOST1_IP} -c "select version();" version --------------------------------------------------------------------------------------------------------- PostgreSQL 12.5 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39), 64-bit (1 row) ``` ЗАМЕЧАНИЕ!!! Вместо переменной ${HOST1_IP} нужно поставить ip адреса сервера vs01!!! ## Установка patroni на vs02 14. Создаем директорию /etc/patroni на втором сервере ```bash vs02$ mkdir /etc/patroni ``` 15. Копируем конфигурационный файл patroni.yml с первого сервера ```bash vs02$ scp ${HOST1_IP}:/etc/patroni/patroni.yml /etc/patroni/patroni.yml ``` ЗАМЕЧАНИЕ!!! Вместо переменной ${HOST1_IP} нужно поставить ip адреса сервера vs01!!! 16. Заменяем имя сервера с postgresql0 на postgresql1 ```bash vs02$ sed -i 's/name: postgresql0/name: postgresql1/' /etc/patroni/patroni.yml vs02$ sed -i 's/${HOST1_IP}:5432/${HOST2_IP}:5432/' /etc/patroni/patroni.yml vs02$ sed -i 's/listen: ${HOST1_IP}:8008/listen: ${HOST2_IP}:8008/' /etc/patroni/patroni.yml vs02$ sed -i 's/connect_address: ${HOST1_IP}:8008/connect_address: ${HOST2_IP}:8008/' /etc/patroni/patroni.yml ``` ЗАМЕЧАНИЕ!!! Вместо переменной ${HOST1_IP} нужно поставить ip адреса сервера vs01. Вместо переменной ${HOST2_IP} нужно поставить ip адреса сервера vs02!!! 17. Проверяем, что замена сработала ```bash vs02$ cat /etc/patroni/patroni.yml | grep 'name: postgresql' vs02$ cat /etc/patroni/patroni.yml | grep '${HOST2_IP}:5432' name: postgresql1 listen: 172.25.27.11:5432 connect_address: 172.25.27.11:5432 ``` ЗАМЕЧАНИЕ!!! Вместо переменной ${HOST2_IP} нужно поставить ip адреса сервера vs02!!! 18. Запускаем сервер patroni и проверяем статус ```bash vs02$ service patroni start vs02$ sleep 20 vs02$ service patroni status Redirecting to /bin/systemctl start patroni.service Redirecting to /bin/systemctl status patroni.service ● patroni.service - Runners to orchestrate a high-availability PostgreSQL Loaded: loaded (/usr/lib/systemd/system/patroni.service; disabled; vendor preset: disabled) Active: active (running) since Thu 2021-09-23 17:59:15 UTC; 20s ago Main PID: 2274 (patroni) CGroup: /system.slice/patroni.service ├─2274 /usr/bin/python3 /usr/bin/patroni /etc/patroni/patroni.yml ├─2292 /usr/pgsql-12/bin/postgres -D /var/lib/pgsql/12/main --config-file=/var/lib/pgsql/12/main/postgresql.conf --listen_addresses=172.25.27.11 --port=5432 --cluster_name=cluster --wal_level=replica --hot_standby=on --max_connections=100 --max_wal_senders=10 --max_prepared_transactions=0 --max_locks_per_transaction=64 --track_commit_timestamp=off --max_replication_slots=10 --max_worker_processes=8 --wal_log_hints=on ├─2294 postgres: cluster: logger ├─2295 postgres: cluster: startup recovering 000000010000000000000003 ├─2299 postgres: cluster: checkpointer ├─2300 postgres: cluster: background writer ├─2301 postgres: cluster: stats collector ├─2307 postgres: cluster: postgres postgres 172.25.27.11(41180) idle └─2309 postgres: cluster: walreceiver streaming 0/3000060 Sep 23 17:59:19 vs02.s013162.slurm.io patroni[2274]: 2021-09-23 17:59:19.831 UTC [2292] LOG: redirecting log output to logging collector process Sep 23 17:59:19 vs02.s013162.slurm.io patroni[2274]: 2021-09-23 17:59:19.831 UTC [2292] HINT: Future log output will appear in directory "log". Sep 23 17:59:19 vs02.s013162.slurm.io patroni[2274]: 172.25.27.11:5432 - rejecting connections Sep 23 17:59:19 vs02.s013162.slurm.io patroni[2274]: 172.25.27.11:5432 - rejecting connections Sep 23 17:59:20 vs02.s013162.slurm.io patroni[2274]: 172.25.27.11:5432 - accepting connections Sep 23 17:59:20 vs02.s013162.slurm.io patroni[2274]: 2021-09-23 17:59:20,877 INFO: Lock owner: postgresql0; I am postgresql1 Sep 23 17:59:20 vs02.s013162.slurm.io patroni[2274]: 2021-09-23 17:59:20,877 INFO: establishing a new patroni connection to the postgres cluster Sep 23 17:59:20 vs02.s013162.slurm.io patroni[2274]: 2021-09-23 17:59:20,902 INFO: no action. I am a secondary (postgresql1) and following a leader (postgresql0) Sep 23 17:59:24 vs02.s013162.slurm.io patroni[2274]: 2021-09-23 17:59:24,027 INFO: no action. I am a secondary (postgresql1) and following a leader (postgresql0) Sep 23 17:59:34 vs02.s013162.slurm.io patroni[2274]: 2021-09-23 17:59:34,027 INFO: no action. I am a secondary (postgresql1) and following a leader (postgresql0) ``` 19. Проверяем, что PostgreSQL подцепился репликой ```bash vs02$ sudo -u postgres -i PGPASSWORD=postgres psql -h ${HOST2_IP} -c "select version();" pid | status | receive_start_lsn | receive_start_tli | received_lsn | received_tli | last_msg_send_time | last_msg_receipt_time | latest_end_lsn | latest_end_time | slot_name | sender_host | sender_port | conninfo ------+-----------+-------------------+-------------------+--------------+--------------+-------------------------------+-------------------------------+----------------+-------------------------------+-------------+--------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2309 | streaming | 0/3000000 | 1 | 0/3000060 | 1 | 2021-09-23 17:59:24.939901+00 | 2021-09-23 17:59:24.936167+00 | 0/3000060 | 2021-09-23 17:59:24.939901+00 | postgresql1 | 172.25.27.10 | 5432 | user=replicator passfile=/tmp/pgpass0 dbname=replication host=172.25.27.10 port=5432 application_name=postgresql1 fallback_application_name=cluster sslmode=prefer sslcompression=0 gssencmode=prefer krbsrvname=postgres target_session_attrs=any (1 row) ``` ЗАМЕЧАНИЕ!!! Вместо переменной ${HOST2_IP} нужно поставить ip адреса сервера vs02!!! ## Установка patroni на vs03 20. Аналогичные действия для patroni проводим для 3 сервера ```bash vs03$ mkdir /etc/patroni vs03$ scp ${HOST1_IP}:/etc/patroni/patroni.yml /etc/patroni/patroni.yml vs03$ sed -i 's/name: postgresql0/name: postgresql2/' /etc/patroni/patroni.yml vs03$ sed -i 's/${HOST1_IP}:5432/${HOST3_IP}:5432/' /etc/patroni/patroni.yml vs03$ sed -i 's/listen: ${HOST1_IP}:8008/listen: ${HOST3_IP}:8008/' /etc/patroni/patroni.yml vs03$ sed -i 's/connect_address: ${HOST1_IP}:8008/connect_address: ${HOST3_IP}:8008/' /etc/patroni/patroni.yml vs03$ cat /etc/patroni/patroni.yml | grep 'name: postgresql' vs03$ cat /etc/patroni/patroni.yml | grep '${HOST3_IP}:5432' vs03$ service patroni start vs03$ sleep 20 vs03$ service patroni status name: postgresql2 listen: 172.25.27.12:5432 connect_address: 172.25.27.12:5432 Redirecting to /bin/systemctl start patroni.service Redirecting to /bin/systemctl status patroni.service ● patroni.service - Runners to orchestrate a high-availability PostgreSQL Loaded: loaded (/usr/lib/systemd/system/patroni.service; disabled; vendor preset: disabled) Active: active (running) since Thu 2021-09-23 17:59:57 UTC; 20s ago Main PID: 2269 (patroni) CGroup: /system.slice/patroni.service ├─2269 /usr/bin/python3 /usr/bin/patroni /etc/patroni/patroni.yml ├─2300 /usr/pgsql-12/bin/postgres -D /var/lib/pgsql/12/main --config-file=/var/lib/pgsql/12/main/postgresql.conf --listen_addresses=172.25.27.12 --port=5432 --cluster_name=cluster --wal_level=replica --hot_standby=on --max_connections=100 --max_wal_senders=10 --max_prepared_transactions=0 --max_locks_per_transaction=64 --track_commit_timestamp=off --max_replication_slots=10 --max_worker_processes=8 --wal_log_hints=on ├─2302 postgres: cluster: logger ├─2303 postgres: cluster: startup recovering 000000010000000000000005 ├─2304 postgres: cluster: checkpointer ├─2305 postgres: cluster: background writer ├─2306 postgres: cluster: stats collector ├─2314 postgres: cluster: postgres postgres 172.25.27.12(48890) idle └─2316 postgres: cluster: walreceiver streaming 0/5000060 Sep 23 18:00:01 vs03.s013162.slurm.io patroni[2269]: 2021-09-23 18:00:01.711 UTC [2300] LOG: listening on Unix socket "./.s.PGSQL.5432" Sep 23 18:00:01 vs03.s013162.slurm.io patroni[2269]: 2021-09-23 18:00:01.732 UTC [2300] LOG: redirecting log output to logging collector process Sep 23 18:00:01 vs03.s013162.slurm.io patroni[2269]: 2021-09-23 18:00:01.732 UTC [2300] HINT: Future log output will appear in directory "log". Sep 23 18:00:02 vs03.s013162.slurm.io patroni[2269]: 172.25.27.12:5432 - accepting connections Sep 23 18:00:02 vs03.s013162.slurm.io patroni[2269]: 172.25.27.12:5432 - accepting connections Sep 23 18:00:02 vs03.s013162.slurm.io patroni[2269]: 2021-09-23 18:00:02,690 INFO: Lock owner: postgresql0; I am postgresql2 Sep 23 18:00:02 vs03.s013162.slurm.io patroni[2269]: 2021-09-23 18:00:02,690 INFO: establishing a new patroni connection to the postgres cluster Sep 23 18:00:02 vs03.s013162.slurm.io patroni[2269]: 2021-09-23 18:00:02,711 INFO: no action. I am a secondary (postgresql2) and following a leader (postgresql0) Sep 23 18:00:04 vs03.s013162.slurm.io patroni[2269]: 2021-09-23 18:00:04,035 INFO: no action. I am a secondary (postgresql2) and following a leader (postgresql0) Sep 23 18:00:14 vs03.s013162.slurm.io patroni[2269]: 2021-09-23 18:00:14,035 INFO: no action. I am a secondary (postgresql2) and following a leader (postgresql0) ``` ЗАМЕЧАНИЕ!!! Вместо переменной ${HOST1_IP} нужно поставить ip адреса сервера vs01. Вместо переменной ${HOST3_IP} нужно поставить ip адреса сервера vs03!!! 21. Проверяем, что PostgreSQL подцепился репликой ```bash vs03$ sudo -u postgres -i PGPASSWORD=postgres psql -h ${HOST3_IP} -c "select version();" pid | status | receive_start_lsn | receive_start_tli | received_lsn | received_tli | last_msg_send_time | last_msg_receipt_time | latest_end_lsn | latest_end_time | slot_name | sender_host | sender_port | conninfo ------+-----------+-------------------+-------------------+--------------+--------------+-------------------------------+-------------------------------+----------------+-------------------------------+-------------+--------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2316 | streaming | 0/5000000 | 1 | 0/5000060 | 1 | 2021-09-23 18:00:06.843126+00 | 2021-09-23 18:00:06.843694+00 | 0/5000060 | 2021-09-23 18:00:06.843126+00 | postgresql2 | 172.25.27.10 | 5432 | user=replicator passfile=/tmp/pgpass0 dbname=replication host=172.25.27.10 port=5432 application_name=postgresql2 fallback_application_name=cluster sslmode=prefer sslcompression=0 gssencmode=prefer krbsrvname=postgres target_session_attrs=any (1 row) ``` ЗАМЕЧАНИЕ!!! Вместо переменной ${HOST3_IP} нужно поставить ip адреса сервера vs03!!! ## Работа с patroni 22. Проверяем, что кластер patroni собран ```bash vs01$ patronictl -c /etc/patroni/patroni.yml list cluster + Cluster: cluster (7011188190946281796) --------+----+-----------+ | Member | Host | Role | State | TL | Lag in MB | +-------------+--------------+---------+---------+----+-----------+ | postgresql0 | 172.25.27.10 | Leader | running | 1 | | | postgresql1 | 172.25.27.11 | Replica | running | 1 | 0 | | postgresql2 | 172.25.27.12 | Replica | running | 1 | 0 | +-------------+--------------+---------+---------+----+-----------+ ``` 23. Давайте посмотрим, что у нас в etcd. Структура каталогов выглядит так ```bash vs01$ etcdctl ls /service/cluster /service/cluster/leader /service/cluster/status /service/cluster/members /service/cluster/initialize /service/cluster/config ``` 24. Кто сейчас лидер можно увидеть в ключе leader ```bash vs01$ etcdctl get /service/cluster/leader postgresql0 ``` 25. Параметры лидера можно увидеть тут ```bash vs01$ etcdctl get service/cluster/members/postgresql0 {"conn_url":"postgres://172.25.27.10:5432/postgres","api_url":"http://127.0.0.1:8008/patroni","state":"running","role":"master","version":"2.1.1","xlog_location":83886176,"timeline":1} ``` 26. Эмулируем выход лидера из строя ```bash vs01$ service patroni stop Redirecting to /bin/systemctl stop patroni.service ``` 27. Проверяем статуса кластера patroni ```bash vs01$ patronictl -c /etc/patroni/patroni.yml list cluster + Cluster: cluster (7011188190946281796) --------+----+-----------+ | Member | Host | Role | State | TL | Lag in MB | +-------------+--------------+---------+---------+----+-----------+ | postgresql0 | 172.25.27.10 | Replica | stopped | | unknown | | postgresql1 | 172.25.27.11 | Leader | running | 2 | | | postgresql2 | 172.25.27.12 | Replica | running | 2 | 0 | +-------------+--------------+---------+---------+----+-----------+ ``` 28. Возвращаем лидера в стой ```bash vs01$ sleep 10 vs01$ service patroni start Redirecting to /bin/systemctl start patroni.service ``` 29. Проверяем статуса кластера patroni ```bash vs01$ patronictl -c /etc/patroni/patroni.yml list cluster + Cluster: cluster (7011188190946281796) --------+----+-----------+ | Member | Host | Role | State | TL | Lag in MB | +-------------+--------------+---------+---------+----+-----------+ | postgresql0 | 172.25.27.10 | Replica | running | 2 | 0 | | postgresql1 | 172.25.27.11 | Leader | running | 2 | | | postgresql2 | 172.25.27.12 | Replica | running | 2 | 0 | +-------------+--------------+---------+---------+----+-----------+ ``` Видим, что лидер переехал с postgresql0 на postgresql1 ## Домашняя работа ## Высокодоступный кластер Patroni 1. Прогоняем скрипт, которые лежит в домашней директории сервера `sbox.slurm.io` ``` [s000000@sbox.slurm.io ~]$ ~/patroni.sh ... Показана концовка вывода файла ... + Cluster: cluster (7011154030226532723) --------+----+-----------+ | Member | Host | Role | State | TL | Lag in MB | +-------------+--------------+---------+---------+----+-----------+ | postgresql0 | 172.25.27.10 | Replica | running | 2 | 0 | | postgresql1 | 172.25.27.11 | Leader | running | 2 | | | postgresql2 | 172.25.27.12 | Replica | running | 2 | 0 | +-------------+--------------+---------+---------+----+-----------+ Видим, что лидер переехал с postgresql0 на postgresql1 Конец демонстрации ``` 2. Проверяем, что кластер Патрони у нас работает и что лидер находится на сервере `postgresql1`. Это должен подтверждать статус кластера. 3. Заходим на первый сервер `vs01` 4. Смотрим помощь к утилите `patronictl` 5. Находим нужную команду для переезда  6. Запускаем ее, указав путь до конфигурационного файла 7. Подтверждаем, что лидер сейчас это `postgresql1` 8. Указываем нового кондидата на переезд `postgresql0` 9. Подтверждаем, что переезд должен быть сейчас 10. Подтверждаем сам факт переезда 11. Проверяем, что сервер переехал ### Чек-лист домашнего задания. Лекция "Высокодоступный кластер" 1. Создан высокодоступный кластер Patroni; 2. Вызвана команда изменения мастера в высокодоступном кластере Patroni;
Решение: 1. Прогоняем скрипт, которые лежит в домашней директории сервера `sbox.slurm.io` ``` [s000000@sbox.slurm.io ~]$ ~/patroni.sh ... Показана концовка вывода файла ... + Cluster: cluster (7011154030226532723) --------+----+-----------+ | Member | Host | Role | State | TL | Lag in MB | +-------------+--------------+---------+---------+----+-----------+ | postgresql0 | 172.25.27.10 | Replica | running | 2 | 0 | | postgresql1 | 172.25.27.11 | Leader | running | 2 | | | postgresql2 | 172.25.27.12 | Replica | running | 2 | 0 | +-------------+--------------+---------+---------+----+-----------+ Видим, что лидер переехал с postgresql0 на postgresql1 Конец демонстрации ``` 2. Проверяем, что кластер Патрони у нас работает и что лидер находится на сервере `postgresql1`. Это должен подтверждать статус кластера. ``` | postgresql1 | 172.25.27.11 | Leader | running | 2 | ``` 3. Заходим на первый сервер `vs01` ``` ssh root@vs01.s000000 Warning: Permanently added 'vs01.s000000,172.25.27.10' (ECDSA) to the list of known hosts. Last login: Thu Sep 23 15:23:17 2021 from 172.20.100.50 [root@vs01.s000000.slurm.io ~]# ``` 4. Смотрим помощь к утилите `patronictl` ``` root$ patronictl --help Usage: patronictl [OPTIONS] COMMAND [ARGS]... Options: -c, --config-file TEXT Configuration file -d, --dcs TEXT Use this DCS -k, --insecure Allow connections to SSL sites without certs --help Show this message and exit. Commands: configure Create configuration file dsn Generate a dsn for the provided member,... edit-config Edit cluster configuration failover Failover to a replica flush Discard scheduled events history Show the history of failovers/switchovers list List the Patroni members for a given Patroni pause Disable auto failover query Query a Patroni PostgreSQL member reinit Reinitialize cluster member reload Reload cluster member configuration remove Remove cluster from DCS restart Restart cluster member resume Resume auto failover scaffold Create a structure for the cluster in DCS show-config Show cluster configuration switchover Switchover to a replica topology Prints ASCII topology for given cluster version Output version of patronictl command or a... ``` 5. Находим нужную команду для переезда  ``` switchover Switchover to a replica ``` 6. Запускаем ее, указав путь до конфигурационного файла ``` patronictl -c /etc/patroni/patroni.yml switchover ``` 7. Подтверждаем, что лидер сейчас это `postgresql1` ``` Master [postgresql1]: ``` 8. Указываем нового кондидата на переезд `postgresql0` ``` Candidate ['postgresql0', 'postgresql2'] []: postgresql0 ``` 9. Подтверждаем, что переезд должен быть сейчас ``` When should the switchover take place (e.g. 2021-09-23T16:41 ) [now]: ``` 10. Подтверждаем сам факт переезда ``` Are you sure you want to switchover cluster cluster, demoting current master postgresql1? [y/N]: y ``` 11. Проверяем, что сервер переехал ``` patronictl -c /etc/patroni/patroni.yml list cluster + Cluster: cluster (7011154030226532723) --------+----+-----------+ | Member | Host | Role | State | TL | Lag in MB | +-------------+--------------+---------+---------+----+-----------+ | postgresql0 | 172.25.27.10 | Leader | running | 3 | | | postgresql1 | 172.25.27.11 | Replica | stopped | | unknown | | postgresql2 | 172.25.27.12 | Replica | running | 2 | 0 | +-------------+--------------+---------+---------+----+-----------+ ```