all websites taking too much time to load.

Discussion in 'ISPConfig 3 Priority Support' started by pawan, Jun 11, 2018.

  1. pawan

    pawan Member HowtoForge Supporter

    Suddenly my server is responding too slow. any website from the server is taking too much time to load.
    to check any issue with the server I checked for certain issues, which might help in suggesting a solution.
    1. server-load is ok. like 0.25
    2. ping response from server is also fine.
    I also did a server benchmark and the results are like this:
    [email protected]:/# sysbench --test=cpu --cpu-max-prime=20000 run
    sysbench 0.4.12: multi-threaded system evaluation benchmark

    Running the test with following options:
    Number of threads: 1

    Doing CPU performance benchmark

    Threads started!
    Done.

    Maximum prime number checked in CPU test: 20000


    Test execution summary:
    total time: 34.1539s
    total number of events: 10000
    total time taken by event execution: 34.1526
    per-request statistics:
    min: 3.41ms
    avg: 3.42ms
    max: 4.61ms
    approx. 95 percentile: 3.42ms

    Threads fairness:
    events (avg/stddev): 10000.0000/0.00
    execution time (avg/stddev): 34.1526/0.00

    [email protected]:/# sysbench --test=fileio --file-total-size=150G --file-test-mode=rndrw --init-rng=on --max-time=300 --max-requests=0 run
    sysbench 0.4.12: multi-threaded system evaluation benchmark

    Running the test with following options:
    Number of threads: 1
    Initializing random number generator from timer.


    Extra file open flags: 0
    128 files, 1.1719Gb each
    150Gb total file size
    Block size 16Kb
    Number of random requests for random IO: 0
    Read/Write ratio for combined random IO test: 1.50
    Periodic FSYNC enabled, calling fsync() each 100 requests.
    Calling fsync() at the end of test, Enabled.
    Using synchronous I/O mode
    Doing random r/w test
    Threads started!
    Time limit exceeded, exiting...
    Done.

    Operations performed: 9058 Read, 6038 Write, 19200 Other = 34296 Total
    Read 141.53Mb Written 94.344Mb Total transferred 235.88Mb (805.09Kb/sec)
    50.32 Requests/sec executed

    Test execution summary:
    total time: 300.0116s
    total number of events: 15096
    total time taken by event execution: 184.7932
    per-request statistics:
    min: 0.01ms
    avg: 12.24ms
    max: 701.53ms
    approx. 95 percentile: 39.78ms

    Threads fairness:
    events (avg/stddev): 15096.0000/0.00
    execution time (avg/stddev): 184.7932/0.00



    [email protected]:/# sysbench --test=oltp --oltp-table-size=1000000 --db-driver=mysql --mysql-db=test --mysql-user=root --mysql-password=password --max-time=60 --oltp-read-only=on --max-requests=0 --num-threads=8 run
    sysbench 0.4.12: multi-threaded system evaluation benchmark

    Running the test with following options:
    Number of threads: 8

    Doing OLTP test.
    Running mixed OLTP test
    Doing read-only test
    Using Special distribution (12 iterations, 1 pct of values are returned in 75 pct cases)
    Using "BEGIN" for starting transactions
    Using auto_inc on the id column
    Threads started!
    Time limit exceeded, exiting...
    (last message repeated 7 times)
    Done.

    OLTP test statistics:
    queries performed:
    read: 1148700
    write: 0
    other: 164100
    total: 1312800
    transactions: 82050 (1367.38 per sec.)
    deadlocks: 0 (0.00 per sec.)
    read/write requests: 1148700 (19143.32 per sec.)
    other operations: 164100 (2734.76 per sec.)

    Test execution summary:
    total time: 60.0053s
    total number of events: 82050
    total time taken by event execution: 479.5003
    per-request statistics:
    min: 1.67ms
    avg: 5.84ms
    max: 193.21ms
    approx. 95 percentile: 8.28ms

    Threads fairness:
    events (avg/stddev): 10256.2500/79.58
    execution time (avg/stddev): 59.9375/0.00


    I have done a test for a webpage at webpagetest.org the result is like this:
    Code:
    Request Details
    
        Before Start Render
        
    
        Before On Load
        
    
        After On Load
    
    Request Details #     Resource     Content Type     Request Start     DNS Lookup     Initial Connection     Time to First Byte     Content Download     Bytes Downloaded     Error/Status Code     IP
    1     http://skretail.in/     text/html     3.091 s     25 ms     3060 ms     86932 ms     7 ms     2.2 KB     200     69.197.135.186
    2     http://skretail.in/s...yle_sheet/style1.css     text/css     90.046 s     -     -     59 ms     3 ms     2.2 KB     200     69.197.135.186
    3     http://skretail.in/i...es/logo-skretail.jpg     image/jpeg     90.116 s     -     -     59 ms     81 ms     27.3 KB     200     69.197.135.186
    4     http://skretail.in/images/logo1.jpg     -     90.13 s     -     55 ms     -     -90130 ms     -     -1     -
    5     http://skretail.in/images/bullot.gif     image/gif     90.258 s     -     -     58 ms     -     0.3 KB     200     69.197.135.186
    6     http://skretail.in/images/body.jpg     image/jpeg     90.317 s     -     -     69 ms     76 ms     51.1 KB     200     69.197.135.186
    7     http://skretail.in/images/shadow.png     image/png     90.463 s     -     -     58 ms     1 ms     0.4 KB     200     69.197.135.186
    8     http://skretail.in/i...ges/work_zone_bg.gif     image/gif     90.523 s     -     -     59 ms     -     0.6 KB     200     69.197.135.186
    9     http://skretail.in/images/border.gif     image/gif     90.584 s     -     -     58 ms     1 ms     0.3 KB     200     69.197.135.186
    10     http://skretail.in/images/box_bg.gif     image/gif     90.644 s     -     -     59 ms     1 ms     0.9 KB     200     69.197.135.186
    11     http://skretail.in/images/menu_bg.gif     image/gif     90.704 s     -     -     58 ms     1 ms     0.5 KB     200     69.197.135.186
    12     http://skretail.in/i...ages/menu_border.gif     image/gif     90.765 s     -     -     58 ms     1 ms     0.6 KB     200     69.197.135.186
    13     http://skretail.in/i...ages/header_box1.png     image/png     90.826 s     -     -     59 ms     1 ms     1.0 KB     200     69.197.135.186
    14     http://skretail.in/images/menu_path.gif     image/gif     90.888 s     -     -     58 ms     1 ms     0.3 KB     200     69.197.135.186
    15     http://skretail.in/images/pic3.jpg     image/jpeg     90.948 s     -     -     61 ms     1 ms     2.2 KB     200     69.197.135.186
    16     http://skretail.in/i...premier-agencies.jpg     image/jpeg     91.011 s     -     -     62 ms     27 ms     18.8 KB     200     69.197.135.186
    17     http://skretail.in/i...ges/logo-sktyres.png     image/png     91.103 s     -     -     67 ms     3 ms     7.6 KB     200     69.197.135.186
    18     http://skretail.in/images/pic1.png     image/png     91.175 s     -     -     67 ms     25 ms     20.6 KB     200     69.197.135.186
    19     http://skretail.in/images/pic2.png     -     91.175 s     -     1057 ms     -     -91175 ms     -     -1     -
    20     http://skretail.in/images/read.gif     -     91.21 s     -     1090 ms     -1 ms     -     -     -1     69.197.135.186
    21     http://skretail.in/images/footer_bg.gif     -     91.21 s     -     1090 ms     -1 ms     -     -     -1     69.197.135.186
    clearly the result show the problem in connectivity. but am not able find the source of the issue.

    I also done a speedtest-cli the result of which is like below:
    speedtest-cli 100%[===================>] 59.98K --.-KB/s in 0.04s

    2018-06-11 17:01:48 (1.32 MB/s) - 'speedtest-cli' saved [61423/61423]

    [email protected]:/home/pawan# chmod +x speedtest-cli [email protected]:/home/pawan# ./speedtest-cli --server 13628 Retrieving speedtest.net configuration...
    Testing from WholeSale Internet (69.197.135.186)...
    Retrieving speedtest.net server list...
    Retrieving information for the selected server...
    Hosted by Nocix (Kansas City, MO) [1.32 km]: 2.261 ms
    Testing download speed.........................
    .......................................................
    Download: 926.30 Mbit/s
    Testing upload speed................................................................................................
    Upload: 833.40 Mbit/s
    [email protected]:/home/pawan#

    Where else I should look or do to fix the issue.
    Thanks.
     
  2. pawan

    pawan Member HowtoForge Supporter

    another observation:
    In ISPCONFIG apache service shows offline.
     
  3. Taleman

    Taleman Active Member HowtoForge Supporter

    Start apache. Or restart it. Or nginx if you webserver is not apache.
    If the ISPConfig Monitor shows any other service is down, start that also.
    Maybe try rebooting the server?
    Those benchmarks are not helpful here. Unless you moved to a slower host and want to compare measurements to the old host.
    Try command "top", see what it show for CPU wa state.
     
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    Plus you might check if the nameservers in /etc/resolv.conf are reachable and responsive from your server and you might install and use the tool iotop to check io load beside using the top command that @Taleman suggested already.
     
  5. pawan

    pawan Member HowtoForge Supporter

    run the top command - cpu wa state shows load like 1.0, 0.6 etc.
    already rebooted the server twice, but no luck, still facing the same issue.
    when checking apache status on command line, it shows fine.
    for iotop: the result is like this:
    Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s
    Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s
    TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
    15627 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.01 % [kworker/u50:1]
    2048 be/4 web201 0.00 B/s 0.00 B/s 0.00 % 0.00 % php-fpm: pool web201
    1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init splash
    2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd]
    3 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0]
    2052 be/4 web204 0.00 B/s 0.00 B/s 0.00 % 0.00 % php-fpm: pool web204
    5 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/0:0H]
    6 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/u48:0]
    7 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/u49:0]
    8 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_sched]
    9 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_bh]
    10 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/0]
    11 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/0]
    12 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/1]
    13 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/1]
    14 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/1]
    2063 be/4 web23 0.00 B/s 0.00 B/s 0.00 % 0.00 % php-fpm: pool web23

    as regarding etc/resolve.conf the contents of the file is like this:
    Code:
    # Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
    #     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
    nameserver 192.187.107.16
    nameserver 69.30.209.16
    nameserver 8.8.8.8
    check - nameservers in /etc/resolv.conf are reachable
    how do I check this?
     
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    you can try something like this:

    dig @8.8.8.8 google.com
     
  7. Taleman

    Taleman Active Member HowtoForge Supporter

    Is this a virtual server or a real iron host? If virtual, what is the load on the host server? Is the network overloaded? Can you check how much traffic there is in the switch or subnet?
    What does command
    Code:
    free -h
    show about memory and swap usage?
    Install dstat, and check with it's various options what is slowing the host down.
     
  8. pawan

    pawan Member HowtoForge Supporter

    HI Till -
    [email protected] google.com looks fine.
    result is like this:
    ; <<>> DiG 9.10.3-P4-Ubuntu <<>> @8.8.8.8 google.com
    ; (1 server found)
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49961
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags:; udp: 512
    ;; QUESTION SECTION:
    ;google.com. IN A

    ;; ANSWER SECTION:
    google.com. 110 IN A 172.217.9.14

    dig 192.187.107.16

    ; <<>> DiG 9.10.3-P4-Ubuntu <<>> 192.187.107.16
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21842
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

    ;; QUESTION SECTION:
    ;192.187.107.16. IN A

    ;; ANSWER SECTION:
    192.187.107.16. 86400 IN A 0.0.0.0

    ;; Query time: 0 msec
    ;; SERVER: 192.187.107.16#53(192.187.107.16)
    ;; WHEN: Mon Jun 11 21:03:40 IST 2018
    ;; MSG SIZE rcvd: 48

    [email protected]:/var/log/mysql# free -h

    [email protected]:/var/log/mysql# free -h
    total used free shared buff/cache available
    Mem: 15G 1.5G 13G 54M 990M 13G
    Swap: 23G 0B 23G
     
  9. pawan

    pawan Member HowtoForge Supporter

    Hi Till - I think somewhere something is messed up, don't know how, but as I stated earlier as well that
    ISPCONFIG MONITOR - showing APACHE server as offline.
    but in the terminal when I run the command it is showing as online.
     
  10. till

    till Super Moderator Staff Member ISPConfig Developer

    The ispconfig monitor tests port 80 on localhost (127.0.0.1). Maybe you altered the hosts file that localhost is not IP 127.0.0.1 anymore or ypou reconfigured apache to not listen on localhost or not listen on port 80.
     
  11. pawan

    pawan Member HowtoForge Supporter

    No Till
    I didn't altered anything.
    but now everything is working fine and fast now.
    besides as you told that I might have altered hosts file, then it shouldn't have worked now on its own.
    so I don't know what had happened.
    I even contacted server support, but they told that there is no issue it there end.
     

Share This Page