server overload on running a Magento command

Discussion in 'ISPConfig 3 Priority Support' started by pawan, Dec 29, 2017.

  1. pawan

    pawan Member HowtoForge Supporter

    I am not aware why and how this is happening and how to fix the issue.
    when I am running any of the commands as root or as chroot user
    php bin/magento setup:upgrade
    php bin/magento cache:flush
    the command processing never ends and on running top command the server load increases continuously and there are so many process starts with www-data
    here is the snapshot
    Code:
    top - 18:37:27 up 44 min,  3 users,  load average: 55.12, 31.33, 15.15
    Tasks: 511 total,   2 running, 509 sleeping,   0 stopped,   0 zombie
    %Cpu(s):  0.1 us,  0.1 sy,  0.0 ni, 34.9 id, 64.9 wa,  0.0 hi,  0.0 si,  0.0 st
    KiB Mem : 16420712 total,   182924 free, 16070896 used,   166892 buff/cache
    KiB Swap: 25150460 total,  9407484 free, 15742976 used.    54316 avail Mem
    
      PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    2154 root      20   0 1172972   2224      0 S   2.0  0.0   1:07.80 fail2ban-server
      916 root      39  19       0      0      0 S   1.3  0.0  20:20.91 kipmi0
    7468 root      20   0   39636    832      0 R   0.7  0.0   0:04.82 top
        8 root      20   0       0      0      0 S   0.3  0.0   0:12.26 rcu_sched
    8839 www-data  20   0  223280   6420     12 D   0.3  0.0   0:00.18 apache2
    8840 www-data  20   0  223280   6400     12 D   0.3  0.0   0:00.18 apache2
    8856 www-data  20   0  223280   6448     12 D   0.3  0.0   0:00.15 apache2
    8880 www-data  20   0  616932   4872   2196 D   0.3  0.0   0:00.02 apache2
    8885 www-data  20   0  223280   7328     44 D   0.3  0.0   0:00.06 apache2
    8887 www-data  20   0  223296   7292     44 D   0.3  0.0   0:00.05 apache2
    8888 www-data  20   0  223304   7396     44 D   0.3  0.0   0:00.08 apache2
    8889 www-data  20   0  223312   7372     44 D   0.3  0.0   0:00.07 apache2
    8897 www-data  20   0  616784   5064   2164 D   0.3  0.0   0:00.01 apache2
        1 root      20   0   38456   1468    796 S   0.0  0.0   0:03.47 systemd
        2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kthreadd
        3 root      20   0       0      0      0 S   0.0  0.0   0:00.02 ksoftirqd/0
        5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
        6 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/u48:0
        9 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
       10 root      rt   0       0      0      0 S   0.0  0.0   0:00.03 migration/0
       11 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 watchdog/0
       12 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 watchdog/1
       13 root      rt   0       0      0      0 S   0.0  0.0   0:00.03 migration/1
       14 root      20   0       0      0      0 S   0.0  0.0   0:00.01 ksoftirqd/1
       16 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0H
       17 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 watchdog/2
       18 root      rt   0       0      0      0 S   0.0  0.0   0:00.03 migration/2
       19 root      20   0       0      0      0 S   0.0  0.0   0:00.01 ksoftirqd/2
       21 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/2:0H
       22 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 watchdog/3
       23 root      rt   0       0      0      0 S   0.0  0.0   0:00.03 migration/3
       24 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/3
       26 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/3:0H
       27 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 watchdog/4
       28 root      rt   0       0      0      0 S   0.0  0.0   0:00.03 migration/4
       29 root      20   0       0      0      0 S   0.0  0.0   0:00.01 ksoftirqd/4
       31 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/4:0H
    
    Need some help, as what could be the reason of such an issue.
    I didn't face this issue earlier.
    Thanks.

    Note: There are also many processes comes up for other vhosts, though the command is run for say web197. but the processes comes up for web158, web77 etc.
     
  2. Croydon

    Croydon ISPConfig Developer ISPConfig Developer

    The top does not show any unusual load on the processes. Please install and execute "iotop" to see the HDD load. The high cpu wait state % is often related to hdd load.
     
  3. florian030

    florian030 ISPConfig Developer ISPConfig Developer

    Or use htop instead of top. I don't think, that a load above 50 comes from a magento php script.
     
  4. pawan

    pawan Member HowtoForge Supporter

    It is really very awkward, but it's the Magento script as above
    even the command php bin/magento cache:clean
    also stucks and increase the server load to a unacceptable level.
    I am attaching snapshots taken from top, htop and iotop here.
    I am not able to run any cli command of Magneto for this specific installation.
    Interestingly other vhost web77, web136 all these come up after I run the command for a different website.

    [​IMG]
    [​IMG]

    [​IMG]
     
  5. Croydon

    Croydon ISPConfig Developer ISPConfig Developer

    Maybe one of your hard disks is failing. Try checking with smartctl -a /dev/sda or whatever device your disk is.
     
  6. Taleman

    Taleman Active Member HowtoForge Supporter

    There seems to be lots of sending e-mail going on. Is that expected?
    Since CPUs are idle and lots of memory is unused, I would examine why disk is busy. Read some guide like this for example: http://bencane.com/2012/08/06/troubleshooting-high-io-wait-in-linux/
    Maybe our disk setup is too slow for the load? What kind of disks and disk controller does that host have? And what file system?
     
  7. pawan

    pawan Member HowtoForge Supporter

    Hi Croydon,
    I am not able to analyze the data what it says, I am putting the result here for the command you suggested.
    Code:
    [email protected]:/var/www/indianartzone.mywebsolutions.co.in/web# smartctl -a /dev/sda
    smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-93-generic] (local build)
    Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
    
    === START OF INFORMATION SECTION ===
    Model Family:     Seagate Barracuda 7200.14 (AF)
    Device Model:     ST2000DM001-1CH164
    Serial Number:    S1E339VD
    LU WWN Device Id: 5 000c50 06d56b01a
    Firmware Version: CC27
    User Capacity:    2,000,398,934,016 bytes [2.00 TB]
    Sector Sizes:     512 bytes logical, 4096 bytes physical
    Rotation Rate:    7200 rpm
    Form Factor:      3.5 inches
    Device is:        In smartctl database [for details use: -P show]
    ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
    SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
    Local Time is:    Sat Jan 13 00:07:06 2018 IST
    SMART support is: Available - device has SMART capability.
    SMART support is: Enabled
    
    === START OF READ SMART DATA SECTION ===
    SMART overall-health self-assessment test result: PASSED
    See vendor-specific Attribute list for marginal Attributes.
    
    General SMART Values:
    Offline data collection status:  (0x82) Offline data collection activity
                                            was completed without error.
                                            Auto Offline Data Collection: Enabled.
    Self-test execution status:      (   0) The previous self-test routine completed
                                            without error or no self-test has ever
                                            been run.
    Total time to complete Offline
    data collection:                (  584) seconds.
    
    Offline data collection
    capabilities:                    (0x7b) SMART execute Offline immediate.
                                            Auto Offline data collection on/off support.
                                            Suspend Offline collection upon new
                                            command.
                                            Offline surface scan supported.
                                            Self-test supported.
                                            Conveyance Self-test supported.
                                            Selective Self-test supported.
    SMART capabilities:            (0x0003) Saves SMART data before entering
                                            power-saving mode.
                                            Supports SMART auto save timer.
    Error logging capability:        (0x01) Error logging supported.
                                            General Purpose Logging supported.
    Short self-test routine
    recommended polling time:        (   1) minutes.
    Extended self-test routine
    recommended polling time:        ( 223) minutes.
    Conveyance self-test routine
    recommended polling time:        (   2) minutes.
    SCT capabilities:              (0x3085) SCT Status supported.
    
    SMART Attributes Data Structure revision number: 10
    Vendor Specific SMART Attributes with Thresholds:
    ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
      1 Raw_Read_Error_Rate     0x000f   118   099   006    Pre-fail  Always       -       196563816
      3 Spin_Up_Time            0x0003   096   096   000    Pre-fail  Always       -       0
      4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       51
      5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       33
      7 Seek_Error_Rate         0x000f   075   060   030    Pre-fail  Always       -       112508599648
      9 Power_On_Hours          0x0032   071   071   000    Old_age   Always       -       25779
    10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
    12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       52
    183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
    184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
    187 Reported_Uncorrect      0x0032   098   098   000    Old_age   Always       -       2
    188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       1 1 1
    189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
    190 Airflow_Temperature_Cel 0x0022   084   001   045    Old_age   Always   In_the_past 16 (0 58 26 14 0)
    191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
    192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       41
    193 Load_Cycle_Count        0x0032   047   047   000    Old_age   Always       -       107892
    194 Temperature_Celsius     0x0022   016   123   000    Old_age   Always       -       16 (0 12 0 0 0)
    197 Current_Pending_Sector  0x0012   086   086   000    Old_age   Always       -       2351
    198 Offline_Uncorrectable   0x0010   086   086   000    Old_age   Offline      -       2351
    199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
    240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       24882h+06m+30.965s
    241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       44702979623
    242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       39448843481
    
    SMART Error Log Version: 1
    ATA Error Count: 2
            CR = Command Register [HEX]
            FR = Features Register [HEX]
            SC = Sector Count Register [HEX]
            SN = Sector Number Register [HEX]
            CL = Cylinder Low Register [HEX]
            CH = Cylinder High Register [HEX]
            DH = Device/Head Register [HEX]
            DC = Device Command Register [HEX]
            ER = Error register [HEX]
            ST = Status register [HEX]
    Powered_Up_Time is measured from power on, and printed as
    DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
    SS=sec, and sss=millisec. It "wraps" after 49.710 days.
    
    Error 2 occurred at disk power-on lifetime: 22465 hours (936 days + 1 hours)
      When the command that caused the error occurred, the device was active or idle.
    
      After command completion occurred, registers were:
      ER ST SC SN CL CH DH
      -- -- -- -- -- -- --
      40 51 00 37 0c 2a 00  Error: UNC at LBA = 0x002a0c37 = 2755639
    
      Commands leading to the command that caused the error were:
      CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
      -- -- -- -- -- -- -- --  ----------------  --------------------
      25 d8 08 37 0c 2a e0 00      00:49:55.708  READ DMA EXT
      25 d8 08 37 0c 2a e0 00      00:49:55.189  READ DMA EXT
      25 d8 08 3f 0c 2a e0 00      00:49:55.124  READ DMA EXT
      25 d8 08 2f 0c 2a e0 00      00:49:55.089  READ DMA EXT
      25 d8 08 27 0c 2a e0 00      00:49:55.081  READ DMA EXT
    
    Error 1 occurred at disk power-on lifetime: 22465 hours (936 days + 1 hours)
      When the command that caused the error occurred, the device was active or idle.
    
      After command completion occurred, registers were:
      ER ST SC SN CL CH DH
      -- -- -- -- -- -- --
      40 51 00 37 0c 2a 00  Error: UNC at LBA = 0x002a0c37 = 2755639
    
      Commands leading to the command that caused the error were:
      CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
      -- -- -- -- -- -- -- --  ----------------  --------------------
      25 d8 08 37 0c 2a e0 00      00:49:55.189  READ DMA EXT
      25 d8 08 3f 0c 2a e0 00      00:49:55.124  READ DMA EXT
      25 d8 08 2f 0c 2a e0 00      00:49:55.089  READ DMA EXT
      25 d8 08 27 0c 2a e0 00      00:49:55.081  READ DMA EXT
      25 d8 08 2f f7 70 e2 00      00:49:55.056  READ DMA EXT
    SMART Self-test log structure revision number 1
    Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
    # 1  Short offline       Completed without error       00%      3980         -
    
    SMART Selective self-test log data structure revision number 1
    SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
        1        0        0  Not_testing
        2        0        0  Not_testing
        3        0        0  Not_testing
        4        0        0  Not_testing
        5        0        0  Not_testing
    Selective self-test flags (0x0):
      After scanning selected spans, do NOT read-scan remainder of disk.
    If Selective self-test is pending on power-up, resume after 0 minute delay.
    
    Please suggest a proper solution, if it's really Hdd issue, I will raise it with the Host.
    Thanks.
     
  8. pawan

    pawan Member HowtoForge Supporter

    I also run a test with sysbench if this helps.
    Code:
    [email protected]:/# sysbench --test=fileio --file-total-size=150G --file-test-mode=rndrw --init-rng=on --max-time=300 --max-requests=0 run
    sysbench 0.4.12:  multi-threaded system evaluation benchmark
    
    Running the test with following options:
    Number of threads: 1
    Initializing random number generator from timer.
    
    
    Extra file open flags: 0
    128 files, 1.1719Gb each
    150Gb total file size
    Block size 16Kb
    Number of random requests for random IO: 0
    Read/Write ratio for combined random IO test: 1.50
    Periodic FSYNC enabled, calling fsync() each 100 requests.
    Calling fsync() at the end of test, Enabled.
    Using synchronous I/O mode
    Doing random r/w test
    Threads started!
    Time limit exceeded, exiting...
    Done.
    
    Operations performed:  11857 Read, 7904 Write, 25216 Other = 44977 Total
    Read 185.27Mb  Written 123.5Mb  Total transferred 308.77Mb  (1.0292Mb/sec)
       65.87 Requests/sec executed
    
    Test execution summary:
        total time:                          300.0036s
        total number of events:              19761
        total time taken by event execution: 190.4599
        per-request statistics:
             min:                                  0.01ms
             avg:                                  9.64ms
             max:                               1359.13ms
             approx.  95 percentile:              25.46ms
    
    Threads fairness:
        events (avg/stddev):           19761.0000/0.00
        execution time (avg/stddev):   190.4599/0.00
    
     
  9. Croydon

    Croydon ISPConfig Developer ISPConfig Developer

    Have you checked your my SQL process list?
     
  10. pawan

    pawan Member HowtoForge Supporter

    how do I check that, can you help me on that?
     
  11. Croydon

    Croydon ISPConfig Developer ISPConfig Developer

    Log in to mysql
    mysql -u root -p
    and then run
    SHOW PROCESSLIST
    or
    SHOW FULL PROCESSLIST
     

Share This Page