Need help to tune kernel/Network settings on Haproxy hosts and backend servers

Discussion in 'Server Operation' started by mera_linux, Jan 4, 2012.

  1. mera_linux

    mera_linux New Member

    Hi,


    We are using HAproxy for http traffic and around 60-80 servers in rotations

    We are getting network errors on backend hosts. We are not getting any logs in dmesg and messages. Please refer the netstat and ifconfig outputs as follow on backend servers..
    Please let us know if we need to change any kernel settings. We have upgraded network drivers but issue still there.

    Operating system- CentOS release 5.4 (Final)
    Kernel - 2.6.18-274

    ====================================================
    ifconfig

    RX packets:97198166 errors:3104 dropped:11 overruns:0 frame:3104
    TX packets:27200159 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX packets:110829843 errors:5 dropped:14 overruns:0 frame:5
    TX packets:132306261 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000

    =========================================
    [[email protected]]# netstat -s
    Ip:
    54308864 total packets received
    688 with invalid addresses
    0 forwarded
    0 incoming packets discarded
    53901555 incoming packets delivered
    64669990 requests sent out
    6 dropped because of missing route
    Icmp:
    9997 ICMP messages received
    3 input ICMP message failed.
    ICMP input histogram:
    destination unreachable: 163
    echo requests: 9828
    echo replies: 6
    10554 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
    destination unreachable: 720
    echo request: 6
    echo replies: 9828
    IcmpMsg:
    InType0: 6
    InType3: 163
    InType8: 9828
    OutType0: 9828
    OutType3: 720
    OutType8: 6
    Tcp:
    6581507 active connections openings
    1607052 passive connection openings
    3526 failed connection attempts
    15216 connection resets received
    2 connections established
    52338714 segments received
    62884099 segments send out
    10711 segments retransmited
    0 bad segments received.
    6295 resets sent
    Udp:
    1550750 packets received
    698 packets to unknown port received.
    0 packet receive errors
    1764626 packets sent
    TcpExt:
    169 resets received for embryonic SYN_RECV sockets
    188 TCP sockets finished time wait in fast timer
    28 time wait sockets recycled by time stamp
    7987454 TCP sockets finished time wait in slow timer
    1 packets rejects in established connections because of timestamp
    471256 delayed acks sent
    1277 delayed acks further delayed because of locked socket
    Quick ack mode was activated 2550 times
    1164815 packets directly queued to recvmsg prequeue.
    2419208 packets directly received from backlog
    1339602178 packets directly received from prequeue
    16421204 packets header predicted
    411964 packets header predicted and directly queued to user
    12740664 acknowledgments not containing data received
    7724081 predicted acknowledgments
    341 times recovered from packet loss due to SACK data
    Detected reordering 1 times using FACK
    Detected reordering 5 times using SACK
    Detected reordering 2 times using time stamp
    4 congestion windows fully recovered
    53 congestion windows partially recovered using Hoe heuristic
    756 congestion windows recovered after partial ack
    1186 TCP data loss events
    TCPLostRetransmit: 2
    643 timeouts after SACK recovery
    98 timeouts in loss state
    1091 fast retransmits
    108 forward retransmits
    1719 retransmits in slow start
    6094 other TCP timeouts
    87 sack retransmits failed
    2 times receiver scheduled too late for direct processing
    2550 DSACKs sent for old packets
    496 DSACKs received
    365 connections reset due to unexpected data
    213 connections aborted due to timeout
    IpExt:
    InBcastPkts: 1396

    ++++++++++++++++++++++++++

    Proto Recv-Q Send-Q State PID/Program name
    tcp 0 4365 ESTABLISHED 11352/httpd
    tcp 0 4365 ESTABLISHED 19753/httpd
    tcp 0 4365 ESTABLISHED -
    tcp 0 4374 FIN_WAIT1 14063/httpd
    tcp 0 4435 ESTABLISHED 6652/httpd
    tcp 0 4505 FIN_WAIT1 3232/httpd
    tcp 0 4531 FIN_WAIT1 1917/httpd
    tcp 0 4532 FIN_WAIT1 15552/httpd
    tcp 0 4611 FIN_WAIT1 16370/httpd
    tcp 0 4788 ESTABLISHED 5170/httpd
    tcp 0 4818 FIN_WAIT1 15557/httpd
    tcp 0 5026 ESTABLISHED 5494/httpd
    tcp 0 5130 ESTABLISHED -
    tcp 0 5497 FIN_WAIT1 12466/httpd
    tcp 0 5645 ESTABLISHED 22611/httpd
    tcp 0 5773 ESTABLISHED 28445/httpd
    tcp 0 5841 ESTABLISHED 1653/httpd
    tcp 0 6010 ESTABLISHED 22245/httpd
    tcp 0 6228 ESTABLISHED 23534/httpd
    tcp 0 6392 FIN_WAIT1 24441/httpd
    tcp 0 6429 ESTABLISHED 1917/httpd
    tcp 0 6457 FIN_WAIT1 23745/httpd
    tcp 0 6922 FIN_WAIT1 6649/httpd
    tcp 0 6988 FIN_WAIT1 14158/httpd
    tcp 0 7256 ESTABLISHED 29844/httpd
    tcp 0 7560 FIN_WAIT1 26647/httpd
    tcp 0 7619 FIN_WAIT1 12465/httpd
    tcp 0 7742 ESTABLISHED 26388/httpd
    tcp 0 7922 ESTABLISHED 14985/httpd
    tcp 0 7922 ESTABLISHED 14158/httpd
    tcp 0 8529 ESTABLISHED 12462/httpd
    tcp 0 8569 FIN_WAIT1 23170/httpd
    tcp 0 8630 ESTABLISHED 20565/httpd
    tcp 0 8694 ESTABLISHED 27032/httpd
    tcp 0 8705 ESTABLISHED 2373/httpd
    tcp 0 8772 ESTABLISHED -
    tcp 0 8779 ESTABLISHED 6649/httpd
    tcp 0 8902 ESTABLISHED 14334/httpd
    tcp 0 9028 FIN_WAIT1 12118/httpd
    tcp 0 9183 FIN_WAIT1 14985/httpd
    tcp 0 9982 FIN_WAIT1 16363/httpd
    tcp 0 10566 FIN_WAIT1 25911/httpd
    tcp 0 16432 ESTABLISHED 31379/sshd: prod [p
    tcp 0 16432 ESTABLISHED 22215/sshd: prod [p
    tcp 0 25125 ESTABLISHED 30203/bash
    tcp 0 25346 FIN_WAIT1 -

    ========================================================
    Present Kernel settings-

    net.ipv4.tcp_syncookies = 0
    net.ipv4.tcp_keepalive_time = 600
    net.ipv4.tcp_synack_retries = 3
    net.ipv4.tcp_syn_retries = 3

    net.ipv4.conf.all.log_martians = 1
    net.ipv4.inet_peer_gc_mintime = 5

    net.ipv4.tcp_ecn = 0
    net.ipv4.tcp_window_scaling = 1
    net.ipv4.tcp_timestamps = 1

    net.ipv4.tcp_sack = 1
    net.ipv4.tcp_fack = 1
    net.ipv4.tcp_dsack = 1
    net.ipv4.ip_forward = 0

    net.ipv4.conf.default.rp_filter = 0
    net.ipv4.tcp_tw_recycle = 1
    net.ipv4.tcp_orphan_retries = 1


    net.ipv4.tcp_fin_timeout = 20

    net.ipv4.tcp_no_metrics_save = 1
    net.ipv4.tcp_moderate_rcvbuf = 1

    net.ipv4.tcp_rfc1337 = 1

    -thanks in Adv
     

Share This Page