ロードバランスlog-未解決-誰かトラシューして
nginx方式
LVS + Keepalived
https://keepalived.readthedocs.io/en/latest/installing_keepalived.html
https://keepalived.readthedocs.io/en/latest/installing_keepalived.html
https://keepalived.readthedocs.io/en/latest/installing_keepalived.html
LVSの設定
ロードバランスしてくれる
sudo apt install ipvsadm
sudo ipvsadm -C
sudo ipvsadm -A -t 192.168.100.222:80 -s wrr # VIPとポート80を追加、重みつきラウンドロビン方式
sudo ipvsadm -a -t 192.168.100.222:80 -r 192.168.100.155:80 -m # GS-01
sudo ipvsadm -a -t 192.168.100.222:80 -r 192.168.100.152:80 -m # GS-02
user@lb-01:/opt/nginx-l4$ sudo ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.100.222:80 rr
-> 192.168.100.152:80 Masq 1 0 0
-> 192.168.100.155:80 Masq 1 0 0
user@lb-01:/opt/nginx-l4$ sudo su -
root@lb-01:~# ipvsadm-save > /etc/sysconfig/ipvsadm
or
sudo service ipvsadm save
# ???
# /etc/ipvsadm.rules にいい感じ
もっとセーブ ?sudo nano /etc/default/ipvsadm(いらない
# ipvsadm
# if you want to start ipvsadm on boot set this to true
AUTO="true"
# daemon method (none|master|backup)
DAEMON="master"
# use interface (eth0,eth1...)
IFACE="eno1"
# syncid to use
# (0 means no filtering of syncids happen, that is the default)
# SYNCID="0"
user@lb-01:/opt/nginx-l4$ sudo systemctl enable ipvsadm
ipvsadm.service is not a native service, redirecting to systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable ipvsadm
user@lb-01:/opt/nginx-l4$ sudo systemctl status ipvsadm
● ipvsadm.service - LSB: ipvsadm daemon
Loaded: loaded (/etc/init.d/ipvsadm; generated)
Active: active (exited) since Fri 2024-12-06 12:56:20 JST; 3h 11min ago
Docs: man:systemd-sysv-generator(8)
Tasks: 0 (limit: 18852)
Memory: 0B
CGroup: /system.slice/ipvsadm.service
12月 06 12:56:20 lb-01 systemd[1]: Starting LSB: ipvsadm daemon...
12月 06 12:56:20 lb-01 ipvsadm[1079]: * ipvsadm is not configured to run. Please edit /etc/default/ipv>
12月 06 12:56:20 lb-01 systemd[1]: Started LSB: ipvsadm daemon.
user@lb-01:/etc/keepalived$ sudo systemctl status ipvsadm
● ipvsadm.service - LSB: ipvsadm daemon
Loaded: loaded (/etc/init.d/ipvsadm; generated)
Active: active (exited) since Fri 2024-12-06 18:29:39 JST; 2min 16s ago
Docs: man:systemd-sysv-generator(8)
Process: 27176 ExecStart=/etc/init.d/ipvsadm start (code=exited, status=0/SUCCESS)
12月 06 18:29:39 lb-01 systemd[1]: Starting LSB: ipvsadm daemon...
12月 06 18:29:39 lb-01 ipvsadm[27176]: * Clearing the current IPVS table...
12月 06 18:29:39 lb-01 ipvsadm[27176]: ...done.
12月 06 18:29:39 lb-01 ipvsadm[27176]: * Loading IPVS configuration...
12月 06 18:29:39 lb-01 ipvsadm[27176]: ...done.
12月 06 18:29:39 lb-01 ipvsadm[27176]: * Starting IPVS Connection Synchronization Daemon master
12月 06 18:29:39 lb-01 ipvsadm[27176]: ...done.
12月 06 18:29:39 lb-01 systemd[1]: Started LSB: ipvsadm daemon.
user@lb-01:/etc/keepalived$
現在の状態
┌───(hitto@hot)-[~]
└─$ curl http://192.168.100.222
┌───(hitto@hot)-[~]
└─$ curl http://192.168.100.222
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
┌───(hitto@hot)-[~]
└─$
一応ラウンドロビンでロードバランスしてくれる
Keepalivedの設定
フェイルオーバーやバックエンドのヘルスチェックしてくれる
vrrp_instance VI_1 {
state MASTER
interface eno1
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 12345
}
virtual_ipaddress {
192.168.100.222 # 仮想IP
}
}
user@lb-01:/etc/keepalived$ sudo systemctl enable keepalived
Synchronizing state of keepalived.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable keepalived
user@lb-01:/etc/keepalived$ sudo systemctl status keepalived
● keepalived.service - Keepalive Daemon (LVS and VRRP)
Loaded: loaded (/lib/systemd/system/keepalived.service; enabled; vendor preset: enabled)
Active: inactive (dead)
Condition: start condition failed at Fri 2024-12-06 16:11:06 JST; 2min 9s ago
12月 06 16:11:06 lb-01 systemd[1]: Condition check resulted in Keepalive Daemon (LVS and VRRP) being sk>
...skipping...
● keepalived.service - Keepalive Daemon (LVS and VRRP)
Loaded: loaded (/lib/systemd/system/keepalived.service; enabled; vendor preset: enabled)
Active: inactive (dead)
Condition: start condition failed at Fri 2024-12-06 16:11:06 JST; 2min 9s ago
12月 06 16:11:06 lb-01 systemd[1]: Condition check resulted in Keepalive Daemon (LVS and VRRP) being sk>
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
user@lb-01:/etc/keepalived$ sudo systemctl start keepalived
user@lb-01:/etc/keepalived$ sudo systemctl status keepalived
● keepalived.service - Keepalive Daemon (LVS and VRRP)
Loaded: loaded (/lib/systemd/system/keepalived.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2024-12-06 16:13:29 JST; 1s ago
Main PID: 9442 (keepalived)
Tasks: 2 (limit: 18852)
Memory: 2.9M
CGroup: /system.slice/keepalived.service
├─9442 /usr/sbin/keepalived --dont-fork
└─9443 /usr/sbin/keepalived --dont-fork
12月 06 16:13:29 lb-01 Keepalived[9442]: Starting Keepalived v2.0.19 (10/19,2019)
12月 06 16:13:29 lb-01 Keepalived[9442]: Running on Linux 5.15.0-113-generic #123~20.04.1-Ubuntu SMP We>
12月 06 16:13:29 lb-01 Keepalived[9442]: Command line: '/usr/sbin/keepalived' '--dont-fork'
12月 06 16:13:29 lb-01 Keepalived[9442]: Opening file '/etc/keepalived/keepalived.conf'.
12月 06 16:13:29 lb-01 Keepalived[9442]: Starting VRRP child process, pid=9443
12月 06 16:13:29 lb-01 Keepalived_vrrp[9443]: Registering Kernel netlink reflector
12月 06 16:13:29 lb-01 Keepalived_vrrp[9443]: Registering Kernel netlink command channel
12月 06 16:13:29 lb-01 Keepalived_vrrp[9443]: Opening file '/etc/keepalived/keepalived.conf'.
12月 06 16:13:29 lb-01 Keepalived_vrrp[9443]: Registering gratuitous ARP shared channel
12月 06 16:13:29 lb-01 Keepalived_vrrp[9443]: (VI_1) Entering BACKUP STATE (init)
lines 1-20/20 (END)
user@lb-01:/etc/netplan$ cat 01-network-manager-all.yaml
# Let NetworkManager manage all devices on this system
network:
version: 2
renderer: networkd
ethernets:
eno1:
addresses: [192.168.100.148/24]
gateway4: 192.168.100.1
nameservers:
addresses: [192.168.100.1]
user@lb-01:/etc/netplan$ ip addr show eno1
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether d0:50:99:ff:e4:03 brd ff:ff:ff:ff:ff:ff
altname enp4s0
inet 192.168.100.148/24 brd 192.168.100.255 scope global eno1 // netplanで設定
valid_lft forever preferred_lft forever
inet 192.168.100.222/32 scope global eno1 // Keepalivedで設定
valid_lft forever preferred_lft forever
inet6 fe80::216a:b3a8:8c54:cdc9/64 scope link noprefixroute
valid_lft forever preferred_lft forever
user@lb-01:/etc/netplan$
ロードバランサの設定
~~~
virtual_server 192.168.100.222 80 {
delay_loop 5 # ポーリング間隔
lb_algo rr # ラウンドロビン
lb_kind NAT # NATモード
persistence_timeout 50
protocol TCP
real_server 192.168.100.155 80 {
weight 1
MISC_CHECK {
misc_path "/etc/keepalived/check_gs01.sh"
misc_timeout 5
misc_dynamic
user root
}
}
real_server 192.168.100.152 80 {
weight 1
MISC_CHECK {
misc_path "/etc/keepalived/check_gs02.sh"
misc_timeout 5
misc_dynamic
user root
}
}
}
#!/bin/bash
curl -s --max-time 2 http://192.168.100.152:80 > /dev/null
if [ $? -ne 0 ]; then
exit 1
fi
exit 0
#!/bin/bash
# CPU使用率を取得(1分平均の負荷)
CPU_LOAD=$(ssh -o ConnectTimeout=3 root@192.168.100.152 "awk '{print $1}' /proc/loadavg")
CPU_THRESHOLD=2.0 # 負荷の閾値(例:2.0)
# CPU負荷が閾値を超えた場合、異常を通知
if (( $(echo "$CPU_LOAD > $CPU_THRESHOLD" | bc -l) )); then
exit 1 # 異常状態
fi
exit 0 # 正常状態
#!/bin/bash
CPU_LOAD=$(ssh -o ConnectTimeout=3 root@192.168.100.155 "awk '{print $1}' /proc/loadavg")
if [ $? -ne 0 ]; then
echo "Failed to connect to 192.168.100.155"
exit 1 # ssh失敗
fi
CPU_THRESHOLD=2.0
if (( $(echo "$CPU_LOAD > $CPU_THRESHOLD" | bc -l) )); then
echo "CPU load too high: $CPU_LOAD"
exit 1 # CPUやばい
fi
echo "CPU load normal: $CPU_LOAD"
exit 0
user@lb-01:/etc/netplan$ sudo chmod +x /etc/keepalived/check_gs01.sh
user@lb-01:/etc/netplan$ sudo chmod +x /etc/keepalived/check_gs02.sh
// チェック これ信用な~~らん~~る
keepalived -t
sudo systemctl restart keepalived
#!/bin/bash
# リモートホスト情報
HOST="192.168.100.155"
USER="user"
THRESHOLD=80
# SSHコマンド
OUTPUT=$(ssh -o ConnectTimeout=3 ${USER}@${HOST} "free | awk '/Mem:/ {print \$3/\$2*100}'" 2>/dev/null)
# SSH接続失敗時の処理
if [ $? -ne 0 ]; then
echo "Failed to connect to ${HOST}."
exit 1
fi
# メモリ使用量の計算と評価
MEMORY_USAGE=$(printf "%.0f" "${OUTPUT}")
if [ -z "${MEMORY_USAGE}" ]; then
echo "Failed to retrieve memory usage from ${HOST}."
exit 1
fi
if [ "${MEMORY_USAGE}" -ge "${THRESHOLD}" ]; then
echo "Memory usage too high: ${MEMORY_USAGE}% on ${HOST}."
exit 1
else
echo "Memory usage normal: ${MEMORY_USAGE}% on ${HOST}."
exit 0
fi
sudo apt remove keepalived
apt-get install curl gcc libssl-dev libnl-3-dev libnl-genl-3-dev libsnmp-dev
wget https://www.keepalived.org/software/keepalived-2.3.2.tar.gz
tar -xzf keepalived-2.3.2.tar.gz
cd keepalived-2.3.2
./configure
#!/bin/bash
# ログファイルパス
LOGFILE="/var/log/keepalived/check_gs01.log"
# リモートホスト情報
HOST="192.168.100.152"
USER="user"
PASSWORD="wfer3K5V"
THRESHOLD=50
# SSHコマンド
OUTPUT=$(sshpass -p "${PASSWORD}" ssh -o ConnectTimeout=3 -o StrictHostKeyChecking=no ${USER}@${HOST} "free | awk '/Mem:/ {print \$3/\$2*100}'" 2>/dev/null)
# SSH接続失敗時の処理
if [ $? -ne 0 ]; then
echo "$(date): Failed to connect to ${HOST}." >> "${LOGFILE}"
exit 1
fi
# メモリ使用量の計算と評価
MEMORY_USAGE=$(printf "%.0f" "${OUTPUT}")
if [ -z "${MEMORY_USAGE}" ]; then
echo "$(date): Failed to retrieve memory usage from ${HOST}." >> "${LOGFILE}"
exit 1
fi
if [ "${MEMORY_USAGE}" -ge "${THRESHOLD}" ]; then
echo "$(date): Memory usage too high: ${MEMORY_USAGE}% on ${HOST}." >> "${LOGFILE}"
exit 1
else
echo "$(date): Memory usage normal: ${MEMORY_USAGE}% on ${HOST}." >> "${LOGFILE}"
exit 0
fi
トラブルシューティング
vi /etc/sysctl.conf
net.ipv4.ip_forward = 1
net.ipv4.conf.default.rp_filter = 0
global
maxconn 2000
stats socket /tmp/haproxy mode 660 user haproxy group haproxy level admin
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
frontend http-in
bind 192.168.100.222:80
default_backend servers
frontend http-8080
bind 192.168.100.222:8080
default_backend backend-8080
frontend http-8000
bind 192.168.100.222:8000
default_backend backend-8000
backend servers
balance leastconn
server backend1 192.168.100.155:80 check
server backend2 192.168.100.152:80 check
backend backend-8080
balance leastconn
server backend1 192.168.100.155:8080 check
server backend2 192.168.100.152:8080 check
backend backend-8000
balance leastconn
server backend1 192.168.100.155:8000 check
server backend2 192.168.100.152:8000 check
listen stats
bind *:1936
stats enable
stats uri /stats
stats refresh 10s
stats auth admin:password
global
maxconn 2000
stats socket /tmp/haproxy mode 660 user haproxy group haproxy level admin
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
frontend http-in
bind 192.168.100.222:80
default_backend servers
backend servers
balance leastconn
server backend1 192.168.100.155:80 check
server backend2 192.168.100.152:80 check
listen stats
bind *:1936
stats enable
stats uri /stats
stats refresh 10s
stats auth admin:password
← Go home