Migrating from Ensim Webppliance to ISPConfig

Discussion in 'General' started by lorello, Mar 15, 2006.

  1. lorello

    lorello New Member

    Hi,
    I've 100 domains and 3000 users in my old server with Ensim 3.1 and I've installed a new server with ISPConfig, now I'm thinking about migration of domains and users.

    Using Ensim API I can obtain a php array with whole data about domains, users... , so I could write a script that take this array and make some query on ISPConfig database inserting records for domains & users. Is this correct? I mean, is all ispconfig data stored in mysql?

    Is there some documentation about data structure of ISPConfig?

    Have someone already done this ensim_to_ispc.php? :rolleyes:
     
  2. falko

    falko Super Moderator

    You can do it with the ISPConfig remoting framework which is in beta right now and therefore not available for download. But it will be in a few weeks.
     
  3. lorello

    lorello New Member

    I'm doing my migration :)

    Thank you Falko & Tim for Remoting Plugin!

    Using "Ensim PHP Class" & "ISPConfig Remoting Plugin" I'm moving my 120 sites & 4000 users from Ensim to ISPConfig.

    Doing this I've encountered some problems with ISPConfig admin interface. I've a domain with 3.500 users, when I open Users page on admin interface I get a long not paginated list of users... what about adding "pagination on Users list" in ISPConfig todolist?
     
  4. till

    till Super Moderator

    This will be really useful indeed :)
     
  5. lorello

    lorello New Member

    On fat domains

    In Italy we have 50% of internet connections with dialup modems at 56Kbps: users list on that domain require 4 minutes each time to load... it should be very useful :)

    Another problem with scalability: when I change a password for some user, ISPConfig make some routine from crontab that make me some doubt about scalability.

    Code:
    root@vandana:/# tail -f /home/admispconfig/ispconfig/ispconfig.log
    21.08.2006 - 17:40:51 => INFO - USER:
    web43_devgiovanna:x:11289:10043:fabio masciullo:/home/www/web43/user/web43_devgiovanna:/dev/null
    21.08.2006 - 17:40:51 => INFO - /root/ispconfig/scripts/lib/config.lib.php, Line 868: setquota -u web43_devgiovanna 5120 5120 0 0 -a &> /dev/null
    21.08.2006 - 17:40:51 => INFO - /root/ispconfig/scripts/lib/classes/ispconfig_procmail.lib.php, Line 57: cp -f /root/ispconfig/isp/conf/forward.master /home/www/web43/user/web43_devgiovanna/.forward
    21.08.2006 - 17:40:52 => INFO - USER:
    web43_fabiogiovanna:x:11290:10043:fabio masciullo:/home/www/web43/user/web43_fabiogiovanna:/dev/null
    21.08.2006 - 17:40:52 => INFO - /root/ispconfig/scripts/lib/config.lib.php, Line 868: setquota -u web43_fabiogiovanna 5120 5120 0 0 -a &> /dev/null
    21.08.2006 - 17:40:52 => INFO - /root/ispconfig/scripts/lib/classes/ispconfig_procmail.lib.php, Line 57: cp -f /root/ispconfig/isp/conf/forward.master /home/www/web43/user/web43_fabiogiovanna/.forward
    21.08.2006 - 17:40:53 => INFO - USER:
    web43_dfee:x:11291:10043:paola negri:/home/www/web43/user/web43_dfee:/dev/null
    21.08.2006 - 17:40:53 => INFO - /root/ispconfig/scripts/lib/config.lib.php, Line 868: setquota -u web43_dfee 5120 5120 0 0 -a &> /dev/null
    21.08.2006 - 17:40:53 => INFO - /root/ispconfig/scripts/lib/classes/ispconfig_procmail.lib.php, Line 57: cp -f /root/ispconfig/isp/conf/forward.master /home/www/web43/user/web43_dfee/.forward
    
    
    1 password changed, 2 seconds for 3500 users, 2 hours of hard work for ISPConfig... for one password... too much :eek:
     
  6. falko

    falko Super Moderator

    2 hours? Where do you see that? :confused:
     
  7. kea13

    kea13 ISPConfig Developer

    Hi lorello,

    since I'm stuck with an Ensim 3.1 box, too, I'd like to inquire whether you'd mind sharing this gem ?

    Thanks in advance for your consideration,
    Greetings,
    Roman
     
  8. lorello

    lorello New Member

    I see the time for this operations in
    /home/admispconfig/ispconfig/ispconfig.log

    Code:
    05.09.2006 - 16:42:06 => INFO - /root/ispconfig/scripts/lib/config.lib.php, Line 254: setquota -g web
    43 5120000 5120000 0 0 -a &> /dev/null
    web43_manto-gas:x:13980:10043:manto-gas:/home/www/web43/user/web43_manto-gas:/dev/null
    05.09.2006 - 16:42:09 => INFO - /root/ispconfig/scripts/lib/config.lib.php, Line 649: cp -fr /root/is
    pconfig/isp/user_standard_index.html_it /home/www/web43/user/web43_manto-gas/web/index.html
    05.09.2006 - 16:42:09 => INFO - /root/ispconfig/scripts/lib/config.lib.php, Line 708: setquota -u web
    43_manto-gas 0 0 0 0 -a &> /dev/null
    05.09.2006 - 16:42:09 => INFO - /root/ispconfig/scripts/lib/classes/ispconfig_procmail.lib.php, Line
    57: cp -f /root/ispconfig/isp/conf/forward.master /home/www/web43/user/web43_manto-gas/.forward
    05.09.2006 - 16:42:09 => INFO - maildirmake /home/www/web43/user/web43_manto-gas/Maildir &> /dev/null
    , Line 109: maildirmake /home/www/web43/user/web43_manto-gas/Maildir &> /dev/null
    web43_lorello:x:10195:10043:lorellone:/home/www/web43/user/web43_lorello:/dev/null
    05.09.2006 - 16:42:10 => INFO - /root/ispconfig/scripts/lib/config.lib.php, Line 868: setquota -u web
    43_lorello 5120 5120 0 0 -a &> /dev/null
    05.09.2006 - 16:42:10 => INFO - /root/ispconfig/scripts/lib/classes/ispconfig_procmail.lib.php, Line
    57: cp -f /root/ispconfig/isp/conf/forward.master /home/www/web43/user/web43_lorello/.forward
    web43_sciarab:x:10196:10043:Michele Sciarabba:/home/www/web43/user/web43_sciarab:/dev/null
    05.09.2006 - 16:42:11 => INFO - /root/ispconfig/scripts/lib/config.lib.php, Line 868: setquota -u web
    43_sciarab 204800 204800 0 0 -a &> /dev/null
    
    ...
    
    05.09.2006 - 18:34:13 => INFO - /root/ispconfig/scripts/lib/config.lib.php, Line 868: setquota -u web
    43_massi.pilati 51200 51200 0 0 -a &> /dev/null
    05.09.2006 - 18:34:13 => INFO - /root/ispconfig/scripts/lib/classes/ispconfig_procmail.lib.php, Line
    57: cp -f /root/ispconfig/isp/conf/forward.master /home/www/web43/user/web43_massi.pilati/.forward
    web43_manto-gas:x:13980:10043:manto-gas:/home/www/web43/user/web43_manto-gas:/dev/null
    05.09.2006 - 18:34:14 => INFO - /root/ispconfig/scripts/lib/config.lib.php, Line 868: setquota -u web
    43_manto-gas 0 0 0 0 -a &> /dev/null
    05.09.2006 - 18:34:14 => INFO - /root/ispconfig/scripts/lib/classes/ispconfig_procmail.lib.php, Line
    57: cp -f /root/ispconfig/isp/conf/forward.master /home/www/web43/user/web43_manto-gas/.forward
    

    I've another problem with this big domain: usernames containing uppercase letters doesn't receive messages and they can't access webmail... they can't do anything :)

    In logs I can see:

    Code:
    05.09.2006 - 18:34:15 => WARN - /root/ispconfig/scripts/lib/config.lib.php, Line 937: WARNING: could
    not setquota -u web43_Ake 0 0 0 0 -a &> /dev/null
    web43_Ake:x:::::/dev/null
    
    05.09.2006 - 18:34:15 => WARN - /root/ispconfig/scripts/lib/config.lib.php, Line 937: WARNING: could
    not setquota -u web43_Micotti 0 0 0 0 -a &> /dev/null
    web43_Micotti:x:::::/dev/null
    
    05.09.2006 - 18:34:16 => WARN - /root/ispconfig/scripts/lib/config.lib.php, Line 937: WARNING: could
    not setquota -u web43_Umberto.Micotti 0 0 0 0 -a &> /dev/null
    web43_Umberto.Micotti:x:::::/dev/null
    
     
  9. till

    till Super Moderator

    Uppercase letters are not supported in linux usernames, thats why you did not receive emails for them.
     
  10. lorello

    lorello New Member

    Hi, I've not made a general purpose toolkit to make my migration, I've only used the Remoting Framework of ISPConfig to create quickly 50% of my sites, the other half has been created manually.

    My old server with ensim has died when I was programming the migration and writing scripts to make it easier, so I've done many errors and if I could return on my steps I should make it better....

    The steps I've followed are (should be):

    - move backups of sites, database, majordomo, mailqueue, users' home, user preferences for squirrelmail from ensim box to ispconfig box
    (exclude in mysql dumps the commands to create & select database, because they will change names on the ISPconfig box)

    - launch many times a script that using remote framework create quickly: client, site, dns records for mail.domain.tld and mysql database (i've made it all with the same password for admin users, database etc.., we could do it better...)

    - launch another script many times to restore backup of sites' files, set correct ownership & permissions.

    - restore mysql database (i've done it manually, because i've made )

    - recreate users on ispconfig sites: a) manually for small sites, because you don't need a default admin user as on Ensim, so you can change something during the migration; b) automatically for some bigger domains, because i've one domain with more than 3200 users...

    - foreach domain, launch a script that take old mailboxes in mbox format and convert it in Maildir format (and write a log where you can read what mailboxes you have forgotten to create)
    - install squirrelmail on ispconfig server and import user preferences & addressbook (i've not done it until now)

    I've found two main problems after migration:

    1. mixed uppercase & lowercase usernames (Some.User@some_ensim_domain.tld) that ISPConfig doesn't like.

    2. ISPConfig performances running big domains: my old box was AMD 2GHz of cpu, 1Gb of ram, one ATA disk, using MailScanner (Spamassassin+clamd), my new box is a dual Xeon 3GHz with 2Gb of ram and 2 SCSI disks but if I activate spamassassin & clamd an all users, as it was on the old server, my load average jump over 200 (!!!) and the server became unusable. I think this depends on how ispconfig use spamassasin&clam (through procmail), I must work on this now to remove spam in earlier stages (on handshake, with RBLs, greylistings, etc...).

    I hope this rapid notes could help you, ask on the forum if you want more info... I could have forgotten some steps.

    See you,
    lorello.
     
  11. till

    till Super Moderator

    2) The problem is that ISPCOnfig calls spamassassin instead of the spamc / spamd combination. We can not simply switch beacuse spamd is not able to read the current spamassassin config files. A solution might be to symlink from the file location spamd expects to the current spamassassin config file for every useraccount. We will start working on a solution for that in the ISPConfig dev branch shortly.
     
  12. lorello

    lorello New Member

    Thanks Till, in the mean time we have disabled spamassasin for the 3 thousand users of our bigger domain and we are configuring graylisting and adding RBL in postfix, trying to reduce the amount of mail received from each account.

    At now we have some time during each day where the load average jump over 100 for the concurrent presence of many clamscan processes.

    There is also another task that is always present during the day, but I suppose it should terminate during the night

    15111 root 39 17 3084 904 624 R 99.9 0.0 5:20.82 grep -iw postfix /var/log/mail.log.07-09-06_23-59-01

    I will study some code...
     

Share This Page