Server Concerns

Discussion in 'Server Operation' started by FrozenPixel, Mar 26, 2008.

  1. FrozenPixel

    FrozenPixel New Member

    I recently had a few issues with our Debian server and want to start some sort of recovery or backup procedure. I am definitely a green horn when it comes to Linux and this is my first job working within a Linux / Windows environment. Right now we use our server as the File / Domain Server for 30 + XP machines. We do not host our website or email internally so basically its function is to authenticate our windows users using SAMBA.

    For the first time since I started, over a year ago I had my first real glitch where all our users lost their connection to the server and were unable to copy their files back to their home directories. The server would not even come up on the monitor nor could I restart it using the on/off button, had to disconnect the power cord. After restarting the server and the separate linux firewall box (Debian as well, no GUI's on either) everything came back. What kinds of steps can I now take to setting up some sort of disaster recovery. Really the only thing I know to do is to restart the systems through the command prompt plus I do have an older version of Webmin installed that I use to troubleshoot / add / delete/ user accounts but aside from that I'm really lost. I did pick up an Administration manual for Redhat but it uses more of the GUI than anything else and our systems do not have that installed.

    We are using a XEON system as the file server and a small Celeron system as the firewall. I manually do an encrypted data backup using a USB drive and True Crypt from my XP machine but I have no backup of things such as firewall settings, SAMBA settings etc. just our user information is being copied.

    Any suggestions what kinds of steps I should do and please remember there is no backup at this point and it is our main server, hence the reason I've been a bit scared to touch anything and screw it all up.


    Thanks
     
    Last edited: Mar 26, 2008
  2. technick

    technick New Member

    Check your logs to see if it gives you any information regarding the lock up. If the Logs don't provide any useful information then check your system for a kernel dump.

    As far as backing up your system, its a good idea to back up everything under /etc.

    Assuming you are using iptables on your debian machine, you can run (must be root) the command "iptables-save > iptable_rules" and this will copy all of the rules out to a the file iptable_rules.

    For iptables information, please check out the netfilter website. Their is tons of information plus tutorials there.
     
  3. falko

    falko Super Moderator ISPConfig Developer

  4. chipsafts

    chipsafts New Member

    update the WebMin and then take a look at what it provides in the backup area.
     
  5. FrozenPixel

    FrozenPixel New Member

    I don't want to do any updating at all since I have no idea what I'm doing at the moment. This is our main server and screwing it up would bring our entire office to a halt.
     
  6. falko

    falko Super Moderator ISPConfig Developer

    I think SystemImager is the best option in this case. It allows you to create an identic image of the system while it's running, and if the system crashes, you can restore the system from the image within a few minutes.
     
  7. FrozenPixel

    FrozenPixel New Member

    I will have a look into that Falko. We just had our system freeze up again today and all the Windows users lost their network connection so I have to do something.

    I also have to figure how to read and locate logs to see if it can give me any indication of a problem.

    Thank you.
     
  8. chipsafts

    chipsafts New Member

    do a
    Code:
    locate smbd.log 
    this should lead you to the directory holding the logs.
     
  9. FrozenPixel

    FrozenPixel New Member

    I opened up the Webmin interface through my PC and have access to different logs but what kinds of things should I be looking for?

    I'm going to go do a search too.
     
  10. FrozenPixel

    FrozenPixel New Member

    I am just trying to pinpoint the time when the server crashed and had some interesting finds in auth.log that I maybe just jumping to conclusions but at
    6.25am (which no one is in the office) and this entry was in the log.

    Mar 31 06:25:02 myserver su[6988]: + ??? root:nobody
    Mar 31 06:25:02 myserver su[6988]: (pam_unix) session opened for user nobody by (uid=0)

    I have no idea what it means.

    and then this showed up a couple of times as well and at the time of the crash

    Mar 31 10:54:21 myserver perl: (pam_unix) authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=root
    Mar 31 10:54:21 myserver perl: (pam_unix) authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=root
    Mar 31 10:54:23 myserver usermin[1317]: Usermin starting
    Mar 31 10:54:23 myserver webmin[1320]: Webmin starting

    Syslog

    Mar 31 11:34:35 myserver dhcpd: DHCPACK on 10.0.0.53 to 00:15:f2:e0:2c:fa via eth0
    Mar 31 11:37:01 myserver syslogd 1.4.1#16: restart.
    Mar 31 11:37:01 myserver kernel: klogd 1.4.1#16, log source = /proc/kmsg started.

    I think it just restarted on its own.


    Does this look normal?


    And I was not on Webmin till after 1pm. I'm a bit concerned I definitely have to get on top of this kind of stuff.
     
    Last edited: Mar 31, 2008
  11. falko

    falko Super Moderator ISPConfig Developer

  12. chipsafts

    chipsafts New Member

    Does not look normal to me. If I was in your position, I would change the root password, change the password to the WEBMIN and USERMIN.
    Then check your sshd_config (/etc/ssh/sshd_config ?)
    and ensure there is a line that says: PermitRootLogin no .
    if you had to put that in, do /etc/rc.d/init.d/sshd restart
     
  13. FrozenPixel

    FrozenPixel New Member

    I've brought in a spare machine from home and just installed the base system for Debian, so I'm going to tyr and set it up similar to the server so now I can at least test these commands first then try it on the server once I'm successful.

    Going to try all the suggetions.

    Thanks
     
  14. FrozenPixel

    FrozenPixel New Member

    What does it mean when you see this line in your syslog?

    Apr 4 06:28:44 myserver syslogd 1.4.1#16: restart

    It's appeared in the log today 3 times in between 6.28am and 8.29am. Is our server actually rebooting. I should mention I have a separate firewall server as well running Debian Etch and both the file server and firewall connect on it's own via wallports.

    Once again our server failed to respond to anything at around 9.15am, whether it be Webmin by remote or keyboard.

    This is frustrating.
     
  15. FrozenPixel

    FrozenPixel New Member

    It's me again, I'm really starting to get frustrated with this installation. Firstly it does not alow mw to use vim to edit files and secondly vi does not work or something. In vim when I hit shift 'i' it goes into INSERT mode and when I save hitting shift'ZZ' causes it save but this crappy vi, nothing happens. I've download a 'how to use vi' but none of the commands allow me to insert or save.

    I have a bunch of services running like proftpd that we don't use and want to eleminate any services that allowing external connections plus I got some weird service running by the 'root' owner saying
    socket://IP Address:pORT smbprn 000002516 USERNAME Obituaries | Death Notice, have no idea what that is.

    :mad:
     
  16. chipsafts

    chipsafts New Member

    have you done any of the changes suggested to you?
     
  17. FrozenPixel

    FrozenPixel New Member

    Well I have been trying a few different things. First thing was too make sure the server itself is ok, not overheating etc..Then I looked at the logs and in the syslog almost like every couple of hours it saying

    Apr 4 06:28:44 myserver syslogd 1.4.1#16: restart

    Then I tried to upgrade the system using apt-get dist-upgrade, which that did not appear to go over very well because it failed cause of errors. I just going through some of services running and came across the ftp and the other weird thing, all of which we don't need running, cause we are only this server as a file server.

    I went over the partition table and everything looked good there. I'm going to try falkos suggestion for looking for malware or trojans but that will have to wait till everyone leaves today.

    Like I mentioned this is new to me and all command work i'm finding difficult, espcially since the person who set it up years ago, all they gave me was the root password and no other info. I just happened to discover it had webmin by accident and I installed Putty on my Windows machine so I don't have to keep running back and forth.

    I haven't changed the password yet, and I really want to try and stop some of these services and outside connections. I think the original admin was using some sort of VPN to log in from home to do anything so I'd like to check and if so stop that service as well. I need no outside access plus this server itself gets it's internet connection from a seperate Debian machine sitting next it. So basically we are running the Debian Firewall (machine #2) and then this Debian server that acts as our DomainController / SAMBA file server.

    I'm learning some stuff but still afraid to wreck everything. I have copied at least our main directories for the windows users but none of the OS directories, going to try and connect a usb external drive and again try that imaging thing.

    Thanks
     
  18. FrozenPixel

    FrozenPixel New Member

    Tried some upgrading and failed, server unresponsive and upon reboot I now have no on board ethernet card working. Now I'm really in shit.:confused:

    When I execute /etc/init.d/networking restart

    I get

    Setting up IP spoofing protection: rp_filter
    Reconfiguring network interfaces...ifup: interface lo already configured

    eth0: ERROR while getting interface flags: No such device
    Bind socket to interface: No such device

    When I go /sbin/ifconfig I get

    lo Link encap:Local Loopback
    inet address:127.0.0.1 Mask:255.0.0.0
    UP LOOPBACK RUNNING MTU:16436 Metric:1

    then some packet info, something is severly screwed up.
     
    Last edited: Apr 5, 2008
  19. FrozenPixel

    FrozenPixel New Member

    I just checked that file and it says: PermitRootLogin yes

    I have nothing else to look at since I can't get the onboard ethernet to work anymore.
     
  20. falko

    falko Super Moderator ISPConfig Developer

    I have no idea what it is, but it might be a hardware problem (mainboard, RAM, HDD, etc.).
     

Share This Page