ロードバランスlog-未解決-誰かトラシューして

nginx方式

  • ラウンドロビン…振り分け先のサーバーへ均等にリクエストがとぶように振り分ける方式。(デフォルト)リクエスト数の比重を変えることも可。
  • リース接続…アクティブな接続数が最も少ないサーバーに振り分けられるような設定です。
  • IPハッシュ…同じIPアドレスからのリクエストは、同じ振り分け先サーバーへ振り分けられるような設定です。1つの処理が複数のリクエストにまたがっており、かつ同一のサーバーで行われる必要があるときに有効な設定です。
  • LVS + Keepalived

    https://keepalived.readthedocs.io/en/latest/installing_keepalived.html

    https://keepalived.readthedocs.io/en/latest/installing_keepalived.html

    https://keepalived.readthedocs.io/en/latest/installing_keepalived.html

    LVSの設定

    ロードバランスしてくれる

  • install
      sudo apt install ipvsadm
  • LVS設定の初期化
      sudo ipvsadm -C
  • LVSルールの設定(いらない
      sudo ipvsadm -A -t 192.168.100.222:80 -s wrr  # VIPとポート80を追加、重みつきラウンドロビン方式
      sudo ipvsadm -a -t 192.168.100.222:80 -r 192.168.100.155:80 -m  # GS-01
      sudo ipvsadm -a -t 192.168.100.222:80 -r 192.168.100.152:80 -m  # GS-02
  • 確認よし!
      user@lb-01:/opt/nginx-l4$ sudo ipvsadm -Ln
      IP Virtual Server version 1.2.1 (size=4096)
      Prot LocalAddress:Port Scheduler Flags
        -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
      TCP  192.168.100.222:80 rr
        -> 192.168.100.152:80           Masq    1      0          0         
        -> 192.168.100.155:80           Masq    1      0          0   
  • save(いらない
      user@lb-01:/opt/nginx-l4$ sudo su -
      root@lb-01:~# ipvsadm-save > /etc/sysconfig/ipvsadm
      or
      sudo service ipvsadm save
      # ???
      # /etc/ipvsadm.rules にいい感じ

      もっとセーブ ?sudo nano /etc/default/ipvsadm(いらない

      # ipvsadm
      
      # if you want to start ipvsadm on boot set this to true
      AUTO="true"
      
      # daemon method (none|master|backup)
      DAEMON="master"
      
      # use interface (eth0,eth1...)
      IFACE="eno1"
      
      # syncid to use 
      # (0 means no filtering of syncids happen, that is the default)
      # SYNCID="0"
      
  • よし 追記- (全然よくない)
      user@lb-01:/opt/nginx-l4$ sudo systemctl enable ipvsadm
      ipvsadm.service is not a native service, redirecting to systemd-sysv-install.
      Executing: /lib/systemd/systemd-sysv-install enable ipvsadm
      user@lb-01:/opt/nginx-l4$ sudo systemctl status ipvsadm
      ● ipvsadm.service - LSB: ipvsadm daemon
           Loaded: loaded (/etc/init.d/ipvsadm; generated)
           Active: active (exited) since Fri 2024-12-06 12:56:20 JST; 3h 11min ago
             Docs: man:systemd-sysv-generator(8)
            Tasks: 0 (limit: 18852)
           Memory: 0B
           CGroup: /system.slice/ipvsadm.service
      
      12月 06 12:56:20 lb-01 systemd[1]: Starting LSB: ipvsadm daemon...
      12月 06 12:56:20 lb-01 ipvsadm[1079]:  * ipvsadm is not configured to run. Please edit /etc/default/ipv>
      12月 06 12:56:20 lb-01 systemd[1]: Started LSB: ipvsadm daemon.
  • よし!
      user@lb-01:/etc/keepalived$ sudo systemctl status ipvsadm
      ● ipvsadm.service - LSB: ipvsadm daemon
           Loaded: loaded (/etc/init.d/ipvsadm; generated)
           Active: active (exited) since Fri 2024-12-06 18:29:39 JST; 2min 16s ago
             Docs: man:systemd-sysv-generator(8)
          Process: 27176 ExecStart=/etc/init.d/ipvsadm start (code=exited, status=0/SUCCESS)
      
      12月 06 18:29:39 lb-01 systemd[1]: Starting LSB: ipvsadm daemon...
      12月 06 18:29:39 lb-01 ipvsadm[27176]:  * Clearing the current IPVS table...
      12月 06 18:29:39 lb-01 ipvsadm[27176]:    ...done.
      12月 06 18:29:39 lb-01 ipvsadm[27176]:  * Loading IPVS configuration...
      12月 06 18:29:39 lb-01 ipvsadm[27176]:    ...done.
      12月 06 18:29:39 lb-01 ipvsadm[27176]:  * Starting IPVS Connection Synchronization Daemon master
      12月 06 18:29:39 lb-01 ipvsadm[27176]:    ...done.
      12月 06 18:29:39 lb-01 systemd[1]: Started LSB: ipvsadm daemon.
      user@lb-01:/etc/keepalived$ 
      
  • 現在の状態

    ┌───(hitto@hot)-[~]
    └─$ curl http://192.168.100.222
    ┌───(hitto@hot)-[~]
    └─$ curl http://192.168.100.222
    <!DOCTYPE html>
    <html>
    <head>
    <title>Welcome to nginx!</title>
    <style>
        body {
            width: 35em;
            margin: 0 auto;
            font-family: Tahoma, Verdana, Arial, sans-serif;
        }
    </style>
    </head>
    <body>
    <h1>Welcome to nginx!</h1>
    <p>If you see this page, the nginx web server is successfully installed and
    working. Further configuration is required.</p>
    
    <p>For online documentation and support please refer to
    <a href="http://nginx.org/">nginx.org</a>.<br/>
    Commercial support is available at
    <a href="http://nginx.com/">nginx.com</a>.</p>
    
    <p><em>Thank you for using nginx.</em></p>
    </body>
    </html>
    ┌───(hitto@hot)-[~]
    └─$ 
    

    一応ラウンドロビンでロードバランスしてくれる

    Keepalivedの設定

    フェイルオーバーやバックエンドのヘルスチェックしてくれる

  • cat /etc/keepalived/keepalived.conf
      vrrp_instance VI_1 {
          state MASTER
          interface eno1
          virtual_router_id 51
          priority 100
          advert_int 1
          authentication {
              auth_type PASS
              auth_pass 12345
          }
          virtual_ipaddress {
              192.168.100.222  # 仮想IP
          }
      }
      
  • install
  • user@lb-01:/etc/keepalived$ sudo systemctl enable keepalived
    Synchronizing state of keepalived.service with SysV service script with /lib/systemd/systemd-sysv-install.
    Executing: /lib/systemd/systemd-sysv-install enable keepalived
    user@lb-01:/etc/keepalived$ sudo systemctl status keepalived
    ● keepalived.service - Keepalive Daemon (LVS and VRRP)
         Loaded: loaded (/lib/systemd/system/keepalived.service; enabled; vendor preset: enabled)
         Active: inactive (dead)
      Condition: start condition failed at Fri 2024-12-06 16:11:06 JST; 2min 9s ago
    
    12月 06 16:11:06 lb-01 systemd[1]: Condition check resulted in Keepalive Daemon (LVS and VRRP) being sk>
    ...skipping...
    ● keepalived.service - Keepalive Daemon (LVS and VRRP)
         Loaded: loaded (/lib/systemd/system/keepalived.service; enabled; vendor preset: enabled)
         Active: inactive (dead)
      Condition: start condition failed at Fri 2024-12-06 16:11:06 JST; 2min 9s ago
    
    12月 06 16:11:06 lb-01 systemd[1]: Condition check resulted in Keepalive Daemon (LVS and VRRP) being sk>
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    user@lb-01:/etc/keepalived$ sudo systemctl start keepalived
    user@lb-01:/etc/keepalived$ sudo systemctl status keepalived
    ● keepalived.service - Keepalive Daemon (LVS and VRRP)
         Loaded: loaded (/lib/systemd/system/keepalived.service; enabled; vendor preset: enabled)
         Active: active (running) since Fri 2024-12-06 16:13:29 JST; 1s ago
       Main PID: 9442 (keepalived)
          Tasks: 2 (limit: 18852)
         Memory: 2.9M
         CGroup: /system.slice/keepalived.service
                 ├─9442 /usr/sbin/keepalived --dont-fork
                 └─9443 /usr/sbin/keepalived --dont-fork
    
    12月 06 16:13:29 lb-01 Keepalived[9442]: Starting Keepalived v2.0.19 (10/19,2019)
    12月 06 16:13:29 lb-01 Keepalived[9442]: Running on Linux 5.15.0-113-generic #123~20.04.1-Ubuntu SMP We>
    12月 06 16:13:29 lb-01 Keepalived[9442]: Command line: '/usr/sbin/keepalived' '--dont-fork'
    12月 06 16:13:29 lb-01 Keepalived[9442]: Opening file '/etc/keepalived/keepalived.conf'.
    12月 06 16:13:29 lb-01 Keepalived[9442]: Starting VRRP child process, pid=9443
    12月 06 16:13:29 lb-01 Keepalived_vrrp[9443]: Registering Kernel netlink reflector
    12月 06 16:13:29 lb-01 Keepalived_vrrp[9443]: Registering Kernel netlink command channel
    12月 06 16:13:29 lb-01 Keepalived_vrrp[9443]: Opening file '/etc/keepalived/keepalived.conf'.
    12月 06 16:13:29 lb-01 Keepalived_vrrp[9443]: Registering gratuitous ARP shared channel
    12月 06 16:13:29 lb-01 Keepalived_vrrp[9443]: (VI_1) Entering BACKUP STATE (init)
    lines 1-20/20 (END)
    
  • netplan
  • user@lb-01:/etc/netplan$ cat 01-network-manager-all.yaml
    # Let NetworkManager manage all devices on this system
    network:
      version: 2
      renderer: networkd
      ethernets:
        eno1:
          addresses: [192.168.100.148/24]
          gateway4: 192.168.100.1
          nameservers:
            addresses: [192.168.100.1]
  • ip a
  • user@lb-01:/etc/netplan$ ip addr show eno1
    2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
        link/ether d0:50:99:ff:e4:03 brd ff:ff:ff:ff:ff:ff
        altname enp4s0
        inet 192.168.100.148/24 brd 192.168.100.255 scope global eno1 // netplanで設定
           valid_lft forever preferred_lft forever
        inet 192.168.100.222/32 scope global eno1                     // Keepalivedで設定
           valid_lft forever preferred_lft forever
        inet6 fe80::216a:b3a8:8c54:cdc9/64 scope link noprefixroute 
           valid_lft forever preferred_lft forever
    user@lb-01:/etc/netplan$ 

    ロードバランサの設定

  • /etc/keepalived/keepalived.confvirtual_serverセクションを追加して、KeepalivedにLVSの設定を統合
      ~~~
      
      virtual_server 192.168.100.222 80 {
          delay_loop 5  # ポーリング間隔
          lb_algo rr    # ラウンドロビン
          lb_kind NAT   # NATモード
          persistence_timeout 50
          protocol TCP
      
          real_server 192.168.100.155 80 {
              weight 1
              MISC_CHECK {
                  misc_path "/etc/keepalived/check_gs01.sh"
                  misc_timeout 5
                  misc_dynamic
                  user root
              }
          }
      
          real_server 192.168.100.152 80 {
              weight 1
              MISC_CHECK {
                  misc_path "/etc/keepalived/check_gs02.sh"
                  misc_timeout 5
                  misc_dynamic
                  user root
              }
          }
      }
      
      
      
  • 死活監視
  • 01
      #!/bin/bash
      curl -s --max-time 2 http://192.168.100.152:80 > /dev/null
      if [ $? -ne 0 ]; then
          exit 1
      fi
      exit 0
      
  • いい感じ監視
  • /etc/keepalived/check_gs01.sh
      #!/bin/bash
      
      # CPU使用率を取得(1分平均の負荷)
      CPU_LOAD=$(ssh -o ConnectTimeout=3 root@192.168.100.152 "awk '{print $1}' /proc/loadavg")
      CPU_THRESHOLD=2.0  # 負荷の閾値(例:2.0)
      
      # CPU負荷が閾値を超えた場合、異常を通知
      if (( $(echo "$CPU_LOAD > $CPU_THRESHOLD" | bc -l) )); then
          exit 1  # 異常状態
      fi
      
      exit 0  # 正常状態
      
  • /etc/keepalived/check_gs02.sh
      #!/bin/bash
      
      CPU_LOAD=$(ssh -o ConnectTimeout=3 root@192.168.100.155 "awk '{print $1}' /proc/loadavg")
      if [ $? -ne 0 ]; then
          echo "Failed to connect to 192.168.100.155"
          exit 1 # ssh失敗
      fi
      
      CPU_THRESHOLD=2.0
      if (( $(echo "$CPU_LOAD > $CPU_THRESHOLD" | bc -l) )); then
          echo "CPU load too high: $CPU_LOAD"
          exit 1 # CPUやばい
      fi
      
      echo "CPU load normal: $CPU_LOAD"
      exit 0
      
      user@lb-01:/etc/netplan$ sudo chmod +x /etc/keepalived/check_gs01.sh
      user@lb-01:/etc/netplan$ sudo chmod +x /etc/keepalived/check_gs02.sh
  • 設定変更したら
      // チェック これ信用な~~らん~~る
      keepalived -t
      
      sudo systemctl restart keepalived
      
  • メモリ監視
      #!/bin/bash
      
      # リモートホスト情報
      HOST="192.168.100.155"
      USER="user"
      THRESHOLD=80
      
      # SSHコマンド
      OUTPUT=$(ssh -o ConnectTimeout=3 ${USER}@${HOST} "free | awk '/Mem:/ {print \$3/\$2*100}'" 2>/dev/null)
      
      # SSH接続失敗時の処理
      if [ $? -ne 0 ]; then
          echo "Failed to connect to ${HOST}."
          exit 1
      fi
      
      # メモリ使用量の計算と評価
      MEMORY_USAGE=$(printf "%.0f" "${OUTPUT}")
      if [ -z "${MEMORY_USAGE}" ]; then
          echo "Failed to retrieve memory usage from ${HOST}."
          exit 1
      fi
      
      if [ "${MEMORY_USAGE}" -ge "${THRESHOLD}" ]; then
          echo "Memory usage too high: ${MEMORY_USAGE}% on ${HOST}."
          exit 1
      else
          echo "Memory usage normal: ${MEMORY_USAGE}% on ${HOST}."
          exit 0
      fi
      
  • バージョンアップ
      sudo apt remove keepalived
      
      apt-get install curl gcc libssl-dev libnl-3-dev libnl-genl-3-dev libsnmp-dev
      
      
      wget https://www.keepalived.org/software/keepalived-2.3.2.tar.gz
      tar -xzf keepalived-2.3.2.tar.gz
      cd keepalived-2.3.2
      ./configure
      
  • メモリ監視
      #!/bin/bash
      
      # ログファイルパス
      LOGFILE="/var/log/keepalived/check_gs01.log"
      
      # リモートホスト情報
      HOST="192.168.100.152"
      USER="user"
      PASSWORD="wfer3K5V"
      THRESHOLD=50
      
      # SSHコマンド
      OUTPUT=$(sshpass -p "${PASSWORD}" ssh -o ConnectTimeout=3 -o StrictHostKeyChecking=no ${USER}@${HOST} "free | awk '/Mem:/ {print \$3/\$2*100}'" 2>/dev/null)
      
      # SSH接続失敗時の処理
      if [ $? -ne 0 ]; then
          echo "$(date): Failed to connect to ${HOST}." >> "${LOGFILE}"
          exit 1
      fi
      
      # メモリ使用量の計算と評価
      MEMORY_USAGE=$(printf "%.0f" "${OUTPUT}")
      if [ -z "${MEMORY_USAGE}" ]; then
          echo "$(date): Failed to retrieve memory usage from ${HOST}." >> "${LOGFILE}"
          exit 1
      fi
      
      if [ "${MEMORY_USAGE}" -ge "${THRESHOLD}" ]; then
          echo "$(date): Memory usage too high: ${MEMORY_USAGE}% on ${HOST}." >> "${LOGFILE}"
          exit 1
      else
          echo "$(date): Memory usage normal: ${MEMORY_USAGE}% on ${HOST}." >> "${LOGFILE}"
          exit 0
      fi
      
  • トラブルシューティング

    vi /etc/sysctl.conf

    net.ipv4.ip_forward = 1
    net.ipv4.conf.default.rp_filter = 0

    global
        maxconn 2000
        stats socket /tmp/haproxy mode 660 user haproxy group haproxy level admin
    
    defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000ms
        timeout client  50000ms
        timeout server  50000ms
    
    frontend http-in
        bind 192.168.100.222:80
        default_backend servers
    
    frontend http-8080
        bind 192.168.100.222:8080
        default_backend backend-8080
    
    frontend http-8000
        bind 192.168.100.222:8000
        default_backend backend-8000
    
    backend servers
        balance leastconn
        server backend1 192.168.100.155:80 check
        server backend2 192.168.100.152:80 check
    
    backend backend-8080
        balance leastconn
        server backend1 192.168.100.155:8080 check
        server backend2 192.168.100.152:8080 check
    
    backend backend-8000
        balance leastconn
        server backend1 192.168.100.155:8000 check
        server backend2 192.168.100.152:8000 check
    
    listen stats
        bind *:1936
        stats enable
        stats uri /stats
        stats refresh 10s
        stats auth admin:password
    
    global
        maxconn 2000
        stats socket /tmp/haproxy mode 660 user haproxy group haproxy level admin
    
    defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000ms
        timeout client  50000ms
        timeout server  50000ms
    
    frontend http-in
        bind 192.168.100.222:80
        default_backend servers
    
    backend servers
        balance leastconn
        server backend1 192.168.100.155:80 check
        server backend2 192.168.100.152:80 check
    
    listen stats
        bind *:1936
        stats enable
        stats uri /stats
        stats refresh 10s
        stats auth admin:password
    ← Go home