ISPConfig feature request: cron jobs order direction

Discussion in 'ISPConfig 3 Priority Support' started by Erih-Gres, Sep 13, 2021.

  1. Erih-Gres

    Erih-Gres New Member HowtoForge Supporter

    Would be great to have ability to change cron jobs order from FIFO to LIFO.
    Also would be better to have ability to setup bounty for feature request :)

    Kind regards,
    Erih
     
  2. Th0m

    Th0m ISPConfig Developer Staff Member ISPConfig Developer

    This is not a good idea. For example, when you disable Let's Encrypt and re-enable it, it would go the wrong way around. So the last action should always be done last.

    It might be that we introduce this in the feature ;)
     
  3. Erih-Gres

    Erih-Gres New Member HowtoForge Supporter

    I understand, it make sense.
    Let me explain problem better:
    I have a lot of cron jobs. Now it more than 4M. Every day cron can execute only 180k jobs.
    For me cron jobs count it's not a big problem, but if I did any change I should wait for something like 22 days :)
    For me would be better to change job order, but I'm open for any other suggestion.
     
  4. Th0m

    Th0m ISPConfig Developer Staff Member ISPConfig Developer

    Wait, so you have 4 million cronjobs?!

    I think the best solution would be to spread your sites/cronjobs over separate servers...
     
    Erih-Gres likes this.
  5. Erih-Gres

    Erih-Gres New Member HowtoForge Supporter

    At the beginning I had 16M....
    Thank you, for your suggestion. Do you have any more effective suggestions?
     
  6. Th0m

    Th0m ISPConfig Developer Staff Member ISPConfig Developer

    Can you share some details about the cronjobs? I'm still wondering how you have so many on one server.
     
  7. Erih-Gres

    Erih-Gres New Member HowtoForge Supporter

    Yes. Sure.
    Current result: 4729798
    every 24 hours server can resolve 180k records.
    Biggest part of this updates about DNS. But for me it doesn't matter because I can't change anything.
    All of the time server receive new updates by API. In this case if I made any changes my updates would be at the end of this queue.
    As I told before for me doesn't matter how much time need to resolve all this updates, but I want to have ability make my changes a little bit faster.
    Also I trued to increase instance CPU and memory but it doesn't make better.
     
  8. till

    till Super Moderator Staff Member ISPConfig Developer

    Ok, so I guess there is a misunderstanding here. the sys_datalog is not what is commonly referred to as cronjobs, cronjobs are in the cron table only. sys_datalog is the config transaction log, and config transactions have to be executed in order. If you would execute them out of order or in reverse order, then the system will fail for most actions. Example: Reversing the order for the creation of a DNS tone would result in DNS records to be added before the zone itself exists, this means you end up with a bunch of errors plus an empty zone at the end.

    One solution might be to run multiple servers, e.g. if you have so many automated DNS transactions and you don't want to slow down transactions when adding a website or database, then you can achieve that by using a multiserver setup where dns nodes are separated from other nodes as each slave node in a multiserver system processes changes on it's own.
     
    Erih-Gres and Th0m like this.
  9. till

    till Super Moderator Staff Member ISPConfig Developer

    Do you use BIND with the normal file backend in ISPConfig? If yes, then you might try to modify it to remove the named zone check to speed things up, but you might have to add other checks then to avoid errors.
     
  10. Erih-Gres

    Erih-Gres New Member HowtoForge Supporter

    Sorry. My English not so good.

    Yes, I'm fully understand this from first Th0m's message.

    Yes. Can you explain how can I do this.
     
  11. till

    till Super Moderator Staff Member ISPConfig Developer

    You can try to disable the zone check like this:

    Edit the file /usr/local/ispconfig/server/plugin-available/bind_plugin.inc.php

    comment out line 337 - 339:

    Code:
    $app->system->exec_safe('named-checkzone ? ?', $zone['origin'], $filename);
                $out = $app->system->last_exec_out();
                $return_status = $app->system->last_exec_retcode();
    and add in that place:

    Code:
    $return_status = 0;
    you might end up with some zone file loading errors in the syslog, but bind ignores such zones anyway as far as I know. So your log will contain some errors of files which would not be included in the bind config otherwise, but it should speed up things when not every change gets checked in this way.
     
    Erih-Gres likes this.
  12. till

    till Super Moderator Staff Member ISPConfig Developer

    another possibility might be to switch from file-based BIND setup to database-based powerdns setup. The powerdns plugin is not officially supported and documented, but I know that some ISP's use it and it might be faster when you have such a large amount of config changes to process as no files need to be written for the zones.
     
  13. Erih-Gres

    Erih-Gres New Member HowtoForge Supporter

    Thanks, gentlemen's!
    If you have any other suggestion how to make it faster I'll be happy to know about it.
    In my case I have 100% that all of requests valid and in correct order.
     

Share This Page