[SOLVED] The following changes are not yet populated to all servers

Discussion in 'General' started by clamp2x, Jan 21, 2021.

  1. clamp2x

    clamp2x New Member

    Hi,

    While creating a cronjob through the ispconfig panel, I noticed that the job was not reflected and remains in the job queue.
    After creating the cronjob, nothing happened and I tried other changes in the ispconfig panel, but it is not being applied to the server.

    The contents currently in the job queue are as follows.

    Delete cron job: 1
    Create cron job: 1
    Update cron job: 1
    Update server settings: 2
    Update shell user: 3
    Update website settings: 2


    My ispconfig version is 3.2.2 (stable), and I reconfigured permissions in master database and reconfigured your services when updating.


    This is my information.
    Code:
    OS: Ubuntu 16.04
    PHP CLI version: 7.0 (I tried versions 7.2 and 7.4, but got the same result.)
    The result of executing /usr/local/ispconfig/server/server.sh to check the debugging log is as follows.
    Code:
    22.01.2021-01:25 - WARNING - There is already a lockfile set, but no process running with this pid (14734). Continuing.
    22.01.2021-01:25 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
    22.01.2021-01:25 - DEBUG - Found 11 changes, starting update process.
    22.01.2021-01:25 - DEBUG - Calling function 'insert' from plugin 'cron_jailkit_plugin' raised by event 'cron_insert'.
    22.01.2021-01:25 - DEBUG - safe_exec cmd: chattr -i '/var/www/clients/client1/web2' - return code: 0
    22.01.2021-01:25 - DEBUG - safe_exec cmd: chattr +i '/var/www/clients/client1/web2' - return code: 0
    22.01.2021-01:25 - DEBUG - safe_exec cmd: chattr -i '/var/www/clients/client1/web2' - return code: 0
    22.01.2021-01:25 - DEBUG - update_jailkit_chroot called for /var/www/clients/client1/web2 with options Array
    (
        [0] => force
    )
    
    22.01.2021-01:25 - DEBUG - update_jailkit_chroot: searching for hardlinks in /var/www/clients/client1/web2/bin
    22.01.2021-01:25 - DEBUG - update_jailkit_chroot: searching for hardlinks in /var/www/clients/client1/web2/dev
    22.01.2021-01:25 - DEBUG - update_jailkit_chroot: searching for hardlinks in /var/www/clients/client1/web2/etc
    22.01.2021-01:25 - DEBUG - removing broken symlink /var/www/clients/client1/web2/lib/x86_64-linux-gnu/libncursesw.so.5
    22.01.2021-01:25 - DEBUG - update_jailkit_chroot: searching for hardlinks in /var/www/clients/client1/web2/lib
    22.01.2021-01:25 - DEBUG - update_jailkit_chroot: searching for hardlinks in /var/www/clients/client1/web2/lib64
    22.01.2021-01:25 - DEBUG - update_jailkit_chroot: searching for hardlinks in /var/www/clients/client1/web2/usr
    22.01.2021-01:25 - DEBUG - update_jailkit_chroot: searching for hardlinks in /var/www/clients/client1/web2/var
    22.01.2021-01:25 - DEBUG - safe_exec cmd: jk_update --jail='/var/www/clients/client1/web2' --skip=/lib32 --skip=/opt --skip=/sys - return code: 0
    22.01.2021-01:25 - DEBUG - jk_update returned: Array
    (
        [0] => skip /var/www/clients/client1/web2/opt/
    )
    
    There was no reaction here, so I pressed Ctrl+C to escape.

    Code:
    ^CTraceback (most recent call last):
      File "/usr/sbin/jk_init", line 247, in <module>
        main()
      File "/usr/sbin/jk_init", line 244, in main
        activateConfig(config, jail, args)
      File "/usr/sbin/jk_init", line 162, in activateConfig
        ji.handle_cfg_section(config,jail,cfg,section)
      File "/usr/sbin/jk_init", line 99, in handle_cfg_section
        self.handle_cfg_section(config,chroot,cfg,tmp)
      File "/usr/sbin/jk_init", line 108, in handle_cfg_section
        self.didfiles = jk_lib.copy_binaries_and_libs(chroot, paths2, config['force'], config['verbose'], 1, try_hardlink=config['hardlink'],try_glob_matching=1,handledfiles=self.didfiles)
      File "/usr/share/jailkit/jk_lib.py", line 633, in copy_binaries_and_libs
        handledfiles = copy_binaries_and_libs(chroot, ret, force_overwrite, be_verbose, check_libs, try_hardlink=try_hardlink, retain_owner=retain_owner, try_glob_matching=0, handledfiles=handledfiles)
      File "/usr/share/jailkit/jk_lib.py", line 713, in copy_binaries_and_libs
        libs = lddlist_libraries(file)
      File "/usr/share/jailkit/jk_lib.py", line 233, in lddlist_libraries
        return lddlist_libraries_linux(executable)
      File "/usr/share/jailkit/jk_lib.py", line 129, in lddlist_libraries_linux
        p = subprocess.Popen('ldd '+executable, shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True)
      File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
        errread, errwrite)
      File "/usr/lib/python2.7/subprocess.py", line 1319, in _execute_child
        data = _eintr_retry_call(os.read, errpipe_read, 1048576)
      File "/usr/lib/python2.7/subprocess.py", line 476, in _eintr_retry_call
        return func(*args)
    KeyboardInterrupt
    The changes requested by the panel up to two days ago were well reflected in the server, but it is assumed that the problem occurred after adding the Cron Job in the panel today.
    The user who created the cron job is an account with a shell user created using jailkit.

    Thanks.
     
  2. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    How long did you wait? The traceback looks like jk_init was running when you hit Ctrl-C. If you run server.sh again I imagine it will attempt to update the same jail (web2) - see what related commands are running when it "hangs" (eg. ps auxww | grep -E 'web2|jk').
     
  3. clamp2x

    clamp2x New Member

    Hi,

    After creating the cron job for the first time, there was no reaction even after waiting about 30 minutes, so I tried additional tasks.
    The first time I worked on it was 3 hours ago.
    I ran /usr/local/ispconfig/server/server.sh and waited about 10 minutes, but there was still no response.

    While there was no response, the currently running process was checked with the command below.
    Code:
    ps auxww | grep -E 'web2|jk'
    The results
    Code:
    root     24394  0.0  0.0  26988  6664 pts/2    R+   02:06   0:00 /usr/bin/python /usr/sbin/jk_init -f -c /etc/jailkit/jk_init.ini -j /var/www/clients/client1/web2 basicshell editors extendedshell netutils ssh sftp scp groups jk_lsh git php composer
    I didn't feel much when using version 3.1, but it seems that it takes a lot of time to select the Chroot shell as jailkit when creating a shell user after updating to version 3.2.
    Thanks.
     
  4. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    So jk_init hasn't exited after 10 minutes (or hours earlier), it sure shouldn't take that long unless you have extremely slow disk i/o.

    You could install strace and attach to that process to see if it's doing anything, or what syscall it's waiting on if not (strace -p 24394).

    Try hitting ctrl-c again, then run the exact same jk_init command from cli as root, see if it prints any errors/clues.

    What version of jailkit do you have installed? There are bugfixes published for it periodically, if it's pretty old, try installing the latest version and see what happens.
     
  5. clamp2x

    clamp2x New Member

    Little progress
    After posting the first question in this thread, running the server.sh file again, about an hour later, the jailkit files were copied, and then moved on to the next process.

    However, it is stopped once again in this part.

    Code:
    22.01.2021-02:53 - DEBUG - ssh-rsa setup shelluser_base
    22.01.2021-02:53 - DEBUG - ssh-rsa key updated in /var/www/clients/client1/web2/.ssh/authorized_keys
    22.01.2021-02:53 - DEBUG - safe_exec cmd: chown -R 'web2':'client1' '/var/www/clients/client1/web2/.ssh' - return code: 0
    22.01.2021-02:53 - DEBUG - safe_exec cmd: chmod 600 '/var/www/clients/client1/web2/.ssh/authorized_keys' - return code: 0
    22.01.2021-02:53 - DEBUG - safe_exec cmd: chattr +i '/var/www/clients/client1/web2' - return code: 0
    22.01.2021-02:53 - DEBUG - Calling function 'update' from plugin 'shelluser_jailkit_plugin' raised by event 'shell_user_update'.
    22.01.2021-02:53 - DEBUG - safe_exec cmd: chattr -i '/var/www/clients/client1/web2' - return code: 0
    22.01.2021-02:53 - DEBUG - safe_exec cmd: chattr +i '/var/www/clients/client1/web2' - return code: 0
    22.01.2021-02:53 - DEBUG - safe_exec cmd: chattr -i '/var/www/clients/client1/web2' - return code: 0
    PHP Warning:  Invalid argument supplied for foreach() in /usr/local/ispconfig/server/lib/classes/system.inc.php on line 2437
    ps auxww | grep -E 'web2|jk'

    If you check the currently running process using this command, jailkit continues to exist, but the pid is changed every time it is checked, making it difficult to trace with strace.

    My jailkit version is version 2.19-1 provided by Ubuntu 16.04. It was confirmed by the following method.
    dpkg -l | grep jailkit
    Code:
    ii  jailkit                               2.19-1                                                             amd64        chroot jail utilities

    In the middle of writing this answer, all the work in the work queue was done.

    I'm seeing this as the cause of the slow copying of jailkit.
    I'm using an SSD, and when I use commands like rsync or zip, it works fast.
     
  6. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    Hmm, this should only be called when a jail wasn't already setup (tested by presence of /etc/jailkit directory within the jail). Can you reproduce this? And/or do you still have the debug output that came before that?

    That's good, and there's no need to run strace if processes are completing/changing (strace was just to troubleshoot why it might be stopped).
     
  7. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    For that shell user being created above, what do you have set for 'Base Dir' under the shell user options?
     
  8. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    I uncovered one bug though I don't think it's the issue(s) you're having right offhand - when changing the shell user Base dir, in my case from /var/www/clients/client1/web20 to /var/www/clients/client1/web20/blah, I see this error:
    Code:
    21.01.2021-13:08 - DEBUG - Calling function 'update' from plugin 'shelluser_base_plugin' raised by event 'shell_user_update'.
    21.01.2021-13:08 - DEBUG - Homedir New: /var/www/clients/client1/web20/blah
    21.01.2021-13:08 - DEBUG - Homedir Old: /var/www/clients/client1/web20
    21.01.2021-13:08 - DEBUG - safe_exec cmd: chattr -i '/var/www/clients/client1/web20' - return code: 0
    PHP Warning:  rename(/var/www/clients/client1/web20,/var/www/clients/client1/web20/blah): Invalid argument in /usr/local/ispconfig/server/lib/classes/
    system.inc.php on line 894
    
    I'll note when changing the Base Dir I did see a stall in exactly the same place you initially reported having one; I caught the jk_init process still running and was able to tell it was very actively doing it's thing. The timestamps only show minute resolution, and according to that it was less than 1 minute until complete.
     
  9. clamp2x

    clamp2x New Member

    Nothing.
    The shelluser's openbase dir is /var/www/clients/client1/web2 .
    It is default setting.
     
  10. clamp2x

    clamp2x New Member

    Hi,
    Tested by setting the environment similar to the current server in another server.

    Same OS, Ubuntu 16.04
    Same jailkit settings in ispconfig panel
    Same jk_init.ini setting
    It was tested using this configuration.

    The test server also uses SSD.
    I used the command below to see which part was the problem.
    Code:
    jk_init -c /etc/jailkit/jk_init.ini -f -k -j /var/www/clients/client1/web1565 php
    Code:
    Creating symlink /var/www/clients/client1/web1565/usr/share/zoneinfo/posix/Europe/Paris to ../../Europe/Paris
    Trying to link /usr/share/zoneinfo/Europe/Paris to /var/www/clients/client1/web1565/usr/share/zoneinfo/Europe/Paris
    The original server takes quite a long time to create each symlink as above.
    But on the test server, it went very fast.

    In the beginning, the process of creating symlinks with jailkit was very fast in the original server, but now it is slow.

    Probably, this is a problem with my server rather than a panel problem.
    Now, to solve this problem, we need to find out what causes symlinks to slow down.

    This is my disk usage info.
    Code:
    [email protected]:~# df -h
    Filesystem      Size  Used Avail Use% Mounted on
    udev             32G     0   32G   0% /dev
    tmpfs           6.3G   59M  6.3G   1% /run
    /dev/sda1       688G  359G  295G  55% /
    tmpfs            32G     0   32G   0% /dev/shm
    tmpfs           5.0M  4.0K  5.0M   1% /run/lock
    tmpfs            32G     0   32G   0% /sys/fs/cgroup
    tmpfs           6.3G     0  6.3G   0% /run/user/0
    
    Code:
    [email protected]:~# df -i
    Filesystem       Inodes   IUsed    IFree IUse% Mounted on
    udev            8227337     398  8226939    1% /dev
    tmpfs           8232394    1032  8231362    1% /run
    /dev/sda1      45793280 4366759 41426521   10% /
    tmpfs           8232394       1  8232393    1% /dev/shm
    tmpfs           8232394      15  8232379    1% /run/lock
    tmpfs           8232394      16  8232378    1% /sys/fs/cgroup
    tmpfs           8232394       4  8232390    1% /run/user/0
    
    Thank you for your help.
     
    Last edited: Jan 22, 2021
  11. Hm. I think I have something similar on my setup (Ubuntu 20.04.1 LTS):

    Code:
    Mon 25 Jan 00:16:34 WET 2021 Beginning jailkit maintenance for domain some.domain.name.here at /var/www/clients/client6/web128
    Mon 25 Jan 00:16:34 WET 2021 chroot --userspec web128:client6 /var/www/clients/client6/web128 /bin/bash -c true 2>/dev/null
    Mon 25 Jan 00:16:34 WET 2021 /bin/bash test failed, forcing update
    Mon 25 Jan 00:16:47 WET 2021 Traceback (most recent call last):
    Mon 25 Jan 00:16:47 WET 2021 File "/usr/sbin/jk_update", line 290, in <module>
    Mon 25 Jan 00:16:47 WET 2021 main()
    Mon 25 Jan 00:16:47 WET 2021 File "/usr/sbin/jk_update", line 272, in main
    Mon 25 Jan 00:16:47 WET 2021 if (not config.has_key('hardlink')):
    Mon 25 Jan 00:16:47 WET 2021 AttributeError: 'dict' object has no attribute 'has_key'
    Mon 25 Jan 00:17:01 WET 2021 finished server.php.
    
    This is something that popped up just after midnight, after I've noticed a surge in CPU load.
    And, indeed, all jk_* functions seem to be teeming with python errors, e.g:

    Code:
    # jk_list
    Traceback (most recent call last):
      File "/usr/sbin/jk_list", line 159, in <module>
        main()
      File "/usr/sbin/jk_list", line 156, in main
        printResults(results,wide)
      File "/usr/sbin/jk_list", line 92, in printResults
        results.sort()
    TypeError: '<' not supported between instances of 'ListResult' and 'ListResult'
    
    This one is even more cryptic:

    Code:
    # jk_check
    ERROR: user web1 does not exist in /var/www/clients/client1/web1//etc/passwd
    ERROR: user web2 does not exist in /var/www/clients/client2/web2//etc/passwd
    ERROR: user web6 does not exist in /var/www/clients/client4/web6//etc/passwd
    ERROR: user web9 does not exist in /var/www/clients/client5/web9//etc/passwd
    ERROR: user web11 does not exist in /var/www/clients/client6/web11//etc/passwd
    ERROR: user web12 does not exist in /var/www/clients/client6/web12//etc/passwd
    ERROR: user web39 does not exist in /var/www/clients/client13/web39//etc/passwd
    ERROR: user web61 does not exist in /var/www/clients/client6/web61//etc/passwd
    ERROR: user web69 does not exist in /var/www/clients/client6/web69//etc/passwd
    ERROR: user web105 does not exist in /var/www/clients/client3/web105//etc/passwd
    ERROR: user web122 does not exist in /var/www/clients/client7/web122//etc/passwd
    ERROR: user web123 does not exist in /var/www/clients/client22/web123//etc/passwd
    ERROR: user web124 does not exist in /var/www/clients/client23/web124//etc/passwd
    ERROR: user web126 does not exist in /var/www/clients/client3/web126//etc/passwd
    ERROR: user web127 does not exist in /var/www/clients/client3/web127//etc/passwd
    ERROR: user web128 does not exist in /var/www/clients/client6/web128//etc/passwd
    ERROR: user web129 does not exist in /var/www/clients/client6/web129//etc/passwd
    ERROR: user web130 does not exist in /var/www/clients/client6/web130//etc/passwd
    ERROR: user web131 does not exist in /var/www/clients/client24/web131//etc/passwd
    ERROR: user web137 does not exist in /var/www/clients/client25/web137//etc/passwd
    ERROR: cannot lstat() /home/testchroot
    cannot check files in /home/testchroot/, it does not exist
    ERROR: cannot read /var/www/clients/client1/web1/usr/share/terminfo/r/rxvt-unicode-256color or /usr/share/terminfo/r/rxvt-unicode-256color !
    ERROR: cannot read /var/www/clients/client1/web1/usr/share/terminfo/s/screen.xterm-256color or /usr/share/terminfo/s/screen.xterm-256color !
    ERROR: cannot read /var/www/clients/client1/web1/usr/share/terminfo/s/screen-bce.mlterm or /usr/share/terminfo/s/screen-bce.mlterm !
    ERROR: cannot read /var/www/clients/client1/web1/usr/share/terminfo/t/tmux or /usr/share/terminfo/t/tmux !
    ERROR: cannot read /var/www/clients/client1/web1/usr/share/terminfo/t/tmux-256color or /usr/share/terminfo/t/tmux-256color !
    ERROR: cannot read /var/www/clients/client1/web1/usr/share/vim/vim74/debian.vim or /usr/share/vim/vim74/debian.vim !
    ERROR: cannot read /var/www/clients/client1/web1/usr/share/vim/vim74/doc/README.Debian or /usr/share/vim/vim74/doc/README.Debian !
    ERROR: cannot read /var/www/clients/client1/web1/usr/share/vim/vim74/doc/help.txt or /usr/share/vim/vim74/doc/help.txt !
    ERROR: cannot read /var/www/clients/client1/web1/usr/share/vim/vim74/doc/tags or /usr/share/vim/vim74/doc/tags !
    ERROR: cannot read /var/www/clients/client1/web1/usr/lib/libdns.so.95.1.1 or /usr/lib/libdns.so.95.1.1 !
    ERROR: symlinks /var/www/clients/client1/web1/usr/lib/libbind9.so.90 and /usr/lib/libbind9.so.90 point to different files!
    ERROR: cannot read /var/www/clients/client1/web1/usr/lib/libisc.so.92 or /usr/lib/libisc.so.92 !
    ERROR: symlinks /var/www/clients/client1/web1/usr/lib/libisccfg.so.90 and /usr/lib/libisccfg.so.90 point to different files!
    ERROR: cannot read /var/www/clients/client1/web1/usr/lib/liblwres.so.90.0.1 or /usr/lib/liblwres.so.90.0.1 !
    ERROR: cannot read /var/www/clients/client1/web1/usr/lib/libisccfg.so.90.0.2 or /usr/lib/libisccfg.so.90.0.2 !
    ERROR: cannot read /var/www/clients/client1/web1/usr/lib/libdns.so.95 or /usr/lib/libdns.so.95 !
    ERROR: cannot read /var/www/clients/client1/web1/usr/lib/libisc.so.92.1.0 or /usr/lib/libisc.so.92.1.0 !
    ERROR: cannot read /var/www/clients/client1/web1/usr/lib/x86_64-linux-gnu/libisc.so.160.0.0 or /usr/lib/x86_64-linux-gnu/libisc.so.160.0.0 !
    [...]
    
    All this is a bit weird.
    I'm not really proficient in python development, but it looks like a wrong/missing library is being called at some point...?
     
    Last edited: Jan 25, 2021
  12. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    I think (not positive) that is an error with a python 2 program running under python 3. Are you using Jailkit 2.21?
     
  13. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    Curious; I don't have many ideas right off, short of digging through the code to see exactly what it's doing there. You could attach (to a running process) with or run under strace real quick and see if it's stuck on some system call. Or do the same with ltrace and it will give even more info from libraries.

    You could check logfiles for errors, too, who knows what might turn up. Filesystem corruption? Apparmor interfering?
     
  14. clamp2x

    clamp2x New Member

    Hi,

    https://www.howtoforge.com/community/threads/jailkit-problems.80852/

    Referencing this thread, I thought that Linux's kernel could be causing this problem.
    Ubuntu 16.04's linux-image-generic version was 4.4.0.
    I proceeded with the planned Ubuntu release update. I wanted jailkit's problem to be solved.
    As the Ubuntu version was updated from 16.04 to 20.04, the version of linux-image-generic was changed to 5.4.0, and when I tried jailkit, I noticed that a symbolic link was created very quickly.

    Thanks for your help!
     
  15. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    Curious, thanks for reporting your solution.
     
  16. Whoops sorry for the late reply. Indeed, I'm using Jailkit 2.21-2.
     

Share This Page