Jailkit problem

Discussion in 'General' started by Greenhorn2013, Apr 3, 2020.

  1. Greenhorn2013

    Greenhorn2013 Member

    Get this message in auth.log:
    Code:
    Apr  3 16:24:20 systemd: pam_unix(systemd-user:session): session opened for user web9 by (uid=0)
    Apr  3 16:24:20 jk_chrootsh[6710]: now entering jail /var/www/clients/client1/web9 for user c1_rosental (5006) with arguments -c /usr/lib/openssh/sftp-server
    Apr  3 16:24:20 jk_chrootsh[6710]: path /bin/bash is not owned by user 0
    Apr  3 16:24:20 jk_chrootsh[6710]: path /bin/bash is not owned by group 0
    when i make
    Code:
    ls -la /var/www/clients/client1/web9/bin/bash
    i get
    Code:
    -rwxr-xr-x 4 5007 client1 1029624 Nov  5  2016 /var/www/clients/client1/web9/bin/bash
    user 5007 is unknown
     
  2. Jesse Norell

    Jesse Norell Well-Known Member

    If this is on debian/ubuntu, search for jk_updater_ispc (a script) and run with -reinit.
     
  3. Greenhorn2013

    Greenhorn2013 Member

    There is no script with this name on Debian GNU/Linux 9.12 (stretch) (4.9.0-12-amd64).
     
  4. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Search with Internet Search Engines like so:
    Code:
    site:howtoforge.com jk_updater_ispc script
     
    Jesse Norell likes this.
  5. Jesse Norell

    Jesse Norell Well-Known Member

    Yeah, sorry, I meant search here in the forums or google or such.
     
  6. Greenhorn2013

    Greenhorn2013 Member

    Found it, run it, but users cant still login. After changing config for shelluser without jailkit, login works. changed back, user cant login.
    in auth.log found this:
    Code:
    Apr  7 16:51:37 vcawws01 sshd[2712]: Accepted password for c1_calidris from 37.24.124.86 port 51378 ssh2
    Apr  7 16:51:37 vcawws01 sshd[2712]: pam_unix(sshd:session): session opened for user c1_calidris by (uid=0)
    Apr  7 16:51:37 vcawws01 systemd-logind[574]: New session 35503 of user web9.
    Apr  7 16:51:37 vcawws01 jk_chrootsh[2778]: now entering jail /var/www/clients/client1/web9 for user c1_calidris (5006) with arguments -c /usr/lib/openssh/sftp-server
    Apr  7 16:51:37 vcawws01 sshd[2710]: Received disconnect from 122.51.213.140 port 60042:11: Bye Bye [preauth]
    Apr  7 16:51:37 vcawws01 sshd[2710]: Disconnected from 122.51.213.140 port 60042 [preauth]
    Apr  7 16:51:37 vcawws01 sshd[2712]: pam_unix(sshd:session): session closed for user c1_calidris
    Apr  7 16:51:37 vcawws01 systemd-logind[574]: Removed session 35503.
    
    In Syslog found this one:
    Code:
    Apr  7 16:51:37 vcawws01 kernel: [371512.752371] bash[2778]: segfault at 0 ip           (null) sp 00007ffe84d5b698 error 14 in bash[400000+100000]
    
     
    Last edited: Apr 7, 2020
  7. Jesse Norell

    Jesse Norell Well-Known Member

    Do you get a bash segfault every time you try to login? Are you loging in with an ssh client or sftp? (If sftp, did you install the sftp server in your jail?)
     
  8. Greenhorn2013

    Greenhorn2013 Member

    we are using in 90% winscp, error came every time. error is new since the last 3 or 4 days. running this configuration for over a year. installed ispconfig with perfect server tutorial.

    There is enough disk and ram space and cpu is max @ 40%.
     
  9. Jesse Norell

    Jesse Norell Well-Known Member

    Curious. Are there any system updates? Update, reboot and hope it's fixed is worth a shot.

    If that doesn't work, can you login with ssh at all? Does a new jail work, and only old ones fail? Can you run anything chroot in that directory (eg. chroot
    /var/www/clients/client1/web9 /bin/ls -l / )? Is /bin/sh still bash, and didn't inadvertently get changed to dash?
     
  10. Steini86

    Steini86 Active Member

    Seems like your jail is broken. A segfault usually happens when a process tries to access a memory position it is not allowed / deos not exist. In a jail, however, this can occur when critical shared libs, config file or /dev/ entries are missing.

    To debug:
    a) Connect to jail, but do not login (do not enter password)
    b) Find the process id: "ps axu|grep sshd|grep jailuser"
    c) Debug the process: strace -p PROCESS-ID -ff -o /tmp/debug
    d) complete login, then read /tmp/debug


    Edit:
    Btw, does "ls -la /var/www/clients/client1/web9/bin/bash" now show the right permissions?
     
  11. Greenhorn2013

    Greenhorn2013 Member

    ls -la /var/www/clients/client1/web9/bin/bash:
    Code:
    -rwxr-xr-x 1 root root 1099016 Mai 15  2017 /var/www/clients/client1/web9/bin/bash
    After Connect 2 Jail, i got 2 processes
    Code:
    root      1087  0.0  0.0  95068  6352 ?        Ss   06:18   0:00 sshd: c1_demo [priv]
    sshd      1088  0.0  0.0  69952  3212 ?        S    06:18   0:00 sshd: c1_demo [net]
    
    Entering Password => terminal window closed
    Debug Data attached as file

    @Jesse Norell Updated to current jailkit and reboot, no changes. MayBe error occurs after jk_updater_ispc.
     

    Attached Files:

    Last edited: Apr 8, 2020
  12. Greenhorn2013

    Greenhorn2013 Member

    Anyone an idea?
     
  13. Jesse Norell

    Jesse Norell Well-Known Member

    Based on file size, the last one is "no", but the others? I'll take a quick look at the strace output.
     
  14. Jesse Norell

    Jesse Norell Well-Known Member

    There should be some more debug files in /tmp, eg. debug.8574 and debug.8614, and maybe others.

    Be sure to change your password or delete your demo account when done here.
     
  15. Greenhorn2013

    Greenhorn2013 Member

    New Jails and Old Jails fails the same, demo account is a new one. I will debug a new login after easter holiday, thanks.
     
  16. Greenhorn2013

    Greenhorn2013 Member

    I started a new session, logs attached.
     

    Attached Files:

  17. Greenhorn2013

    Greenhorn2013 Member

    Any further information needed?
     
  18. Jesse Norell

    Jesse Norell Well-Known Member

    Well, I don't gather much more from the strace other than bash was killed with a SEGV. From your output (eg. debug_1.8615) this happens very early in bash execution, right after reading libc and doing some memory mapping:
    Code:
    open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
    read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\4\2\0\0\0\0\0"..., 832) = 832
    fstat(3, {st_mode=S_IFREG|0755, st_size=1689360, ...}) = 0
    mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f14eb833000
    mmap(NULL, 3795296, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f14eae4c000
    mprotect(0x7f14eafe1000, 2097152, PROT_NONE) = 0
    mmap(0x7f14eb1e1000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x195000) = 0x7f14eb1e1000
    mmap(0x7f14eb1e7000, 14688, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f14eb1e7000
    close(3)                                = 0
    mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f14eb832000
    mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f14eb831000
    mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f14eb830000
    arch_prctl(ARCH_SET_FS, 0x7f14eb831700) = 0
    --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=NULL} ---
    +++ killed by SIGSEGV +++
    
    I traced the same way on a machine here, and right at the point yours crashed I see additional memory protection calls happening:

    Code:
    openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
    read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260A\2\0\0\0\0\0"..., 832) = 832
    fstat(3, {st_mode=S_IFREG|0755, st_size=1824496, ...}) = 0
    mmap(NULL, 1837056, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f1caf56e000
    mprotect(0x7f1caf590000, 1658880, PROT_NONE) = 0
    mmap(0x7f1caf590000, 1343488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x22000) = 0x7f1caf590000
    mmap(0x7f1caf6d8000, 311296, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16a000) = 0x7f1caf6d8000
    mmap(0x7f1caf725000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b6000) = 0x7f1caf725000
    mmap(0x7f1caf72b000, 14336, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f1caf72b000
    close(3)                                = 0
    mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1caf56b000
    arch_prctl(ARCH_SET_FS, 0x7f1caf56b740) = 0
    mprotect(0x7f1caf725000, 16384, PROT_READ) = 0
    mprotect(0x7f1caf732000, 4096, PROT_READ) = 0
    mprotect(0x7f1caf75d000, 16384, PROT_READ) = 0
    mprotect(0x55e03d000000, 12288, PROT_READ) = 0
    mprotect(0x7f1caf78e000, 4096, PROT_READ) = 0
    munmap(0x7f1caf764000, 8734)            = 0
    openat(AT_FDCWD, "/dev/tty", O_RDWR|O_NONBLOCK) = 3
    close(3)                                = 0
    ...
    
    At this point you can go one of 2 routes, 1) try to figure out what changed in your system/environment that started this (package updates, in libc or the kernel? container/memory settings? security related changes? etc.), or 2) try to debug bash further to see more specifically what/where it's failing, to identify what changed.

    For 1), check command history, package and file timestamps, any records or container/vps type settings, etc. For 2) try installing ltrace and run similar to strace (eg. in your example command output above, you'd run 'ltrace -f -p 1087 -p 1088 -o /tmp/ltrace.out' at the point you ran strace, that will show more info on what's happening. And the next step would be to setup bash to create a core file when it crashes, I've not looked into how to do that specifically, but I'd try to run 'ulimit -c unlimited' ahead of starting bash from jk_chrootsh or so.
     
  19. Greenhorn2013

    Greenhorn2013 Member

    Cant find any significant changes. So i do 2 step, attachment is to big to attach, so here is the link:
    https://we.tl/t-c1ELCel5Y1
     
  20. Jesse Norell

    Jesse Norell Well-Known Member

    Of that, process 7668 is the interesting one, and just before it exits (last few lines of the file):

    Code:
    7668 syslog(3, "abort, home directory %s differs"..., "/home/c1_demo", "", "c1_demo", 5006, "/var/www/clients/client1/web9") = <void>
    
    Sounds like a problem in the passwd files (I think the one inside the chroot). What do you get from "grep -E 'c1|client1/web9' /etc/passwd" and "cat /var/www/clients/client1/web9/etc/passwd" ?

    That jk_updater_ispc script doesn't touch/try to validate/recreate the passwd file, maybe I should look at adding that.
     

Share This Page