Migrating from Ensim Webppliance to ISPConfig

[lorello]

Hi,
I've 100 domains and 3000 users in my old server with Ensim 3.1 and I've installed a new server with ISPConfig, now I'm thinking about migration of domains and users.

Using Ensim API I can obtain a php array with whole data about domains, users... , so I could write a script that take this array and make some query on ISPConfig database inserting records for domains & users. Is this correct? I mean, is all ispconfig data stored in mysql?

Is there some documentation about data structure of ISPConfig?

Have someone already done this ensim_to_ispc.php? :rolleyes:

[falko]

You can do it with the ISPConfig remoting framework which is in beta right now and therefore not available for download. But it will be in a few weeks.

[lorello]

Thank you Falko & Tim for Remoting Plugin!

Using "Ensim PHP Class" & "ISPConfig Remoting Plugin" I'm moving my 120 sites & 4000 users from Ensim to ISPConfig.

Doing this I've encountered some problems with ISPConfig admin interface. I've a domain with 3.500 users, when I open Users page on admin interface I get a long not paginated list of users... what about adding "pagination on Users list" in ISPConfig todolist?

[till]

Doing this I've encountered some problems with ISPConfig admin interface. I've a domain with 3.500 users, when I open Users page on admin interface I get a long not paginated list of users... what about adding "pagination on Users list" in ISPConfig todolist?

This will be really useful indeed :)

[lorello]

This will be really useful indeed :)

In Italy we have 50% of internet connections with dialup modems at 56Kbps: users list on that domain require 4 minutes each time to load... it should be very useful :-)

Another problem with scalability: when I change a password for some user, ISPConfig make some routine from crontab that make me some doubt about scalability.


root@vandana:/# tail -f /home/admispconfig/ispconfig/ispconfig.log
21.08.2006 - 17:40:51 => INFO - USER:
web43_devgiovanna:x:11289:10043:fabio masciullo:/home/www/web43/user/web43_devgiovanna:/dev/null
21.08.2006 - 17:40:51 => INFO - /root/ispconfig/scripts/lib/config.lib.php, Line 868: setquota -u web43_devgiovanna 5120 5120 0 0 -a &> /dev/null
21.08.2006 - 17:40:51 => INFO - /root/ispconfig/scripts/lib/classes/ispconfig_procmail.lib.php, Line 57: cp -f /root/ispconfig/isp/conf/forward.master /home/www/web43/user/web43_devgiovanna/.forward
21.08.2006 - 17:40:52 => INFO - USER:
web43_fabiogiovanna:x:11290:10043:fabio masciullo:/home/www/web43/user/web43_fabiogiovanna:/dev/null
21.08.2006 - 17:40:52 => INFO - /root/ispconfig/scripts/lib/config.lib.php, Line 868: setquota -u web43_fabiogiovanna 5120 5120 0 0 -a &> /dev/null
21.08.2006 - 17:40:52 => INFO - /root/ispconfig/scripts/lib/classes/ispconfig_procmail.lib.php, Line 57: cp -f /root/ispconfig/isp/conf/forward.master /home/www/web43/user/web43_fabiogiovanna/.forward
21.08.2006 - 17:40:53 => INFO - USER:
web43_dfee:x:11291:10043:paola negri:/home/www/web43/user/web43_dfee:/dev/null
21.08.2006 - 17:40:53 => INFO - /root/ispconfig/scripts/lib/config.lib.php, Line 868: setquota -u web43_dfee 5120 5120 0 0 -a &> /dev/null
21.08.2006 - 17:40:53 => INFO - /root/ispconfig/scripts/lib/classes/ispconfig_procmail.lib.php, Line 57: cp -f /root/ispconfig/isp/conf/forward.master /home/www/web43/user/web43_dfee/.forward



1 password changed, 2 seconds for 3500 users, 2 hours of hard work for ISPConfig... for one password... too much :eek:

[falko]

2 hours of hard work for ISPConfig...
2 hours? Where do you see that? :confused:

[kea13]

Hi lorello,

since I'm stuck with an Ensim 3.1 box, too, I'd like to inquire whether you'd mind sharing this gem ?

Thanks in advance for your consideration,
Greetings,
Roman

[lorello]

I see the time for this operations in
/home/admispconfig/ispconfig/ispconfig.log


05.09.2006 - 16:42:06 => INFO - /root/ispconfig/scripts/lib/config.lib.php, Line 254: setquota -g web
43 5120000 5120000 0 0 -a &> /dev/null
web43_manto-gas:x:13980:10043:manto-gas:/home/www/web43/user/web43_manto-gas:/dev/null
05.09.2006 - 16:42:09 => INFO - /root/ispconfig/scripts/lib/config.lib.php, Line 649: cp -fr /root/is
pconfig/isp/user_standard_index.html_it /home/www/web43/user/web43_manto-gas/web/index.html
05.09.2006 - 16:42:09 => INFO - /root/ispconfig/scripts/lib/config.lib.php, Line 708: setquota -u web
43_manto-gas 0 0 0 0 -a &> /dev/null
05.09.2006 - 16:42:09 => INFO - /root/ispconfig/scripts/lib/classes/ispconfig_procmail.lib.php, Line
57: cp -f /root/ispconfig/isp/conf/forward.master /home/www/web43/user/web43_manto-gas/.forward
05.09.2006 - 16:42:09 => INFO - maildirmake /home/www/web43/user/web43_manto-gas/Maildir &> /dev/null
, Line 109: maildirmake /home/www/web43/user/web43_manto-gas/Maildir &> /dev/null
web43_lorello:x:10195:10043:lorellone:/home/www/web43/user/web43_lorello:/dev/null
05.09.2006 - 16:42:10 => INFO - /root/ispconfig/scripts/lib/config.lib.php, Line 868: setquota -u web
43_lorello 5120 5120 0 0 -a &> /dev/null
05.09.2006 - 16:42:10 => INFO - /root/ispconfig/scripts/lib/classes/ispconfig_procmail.lib.php, Line
57: cp -f /root/ispconfig/isp/conf/forward.master /home/www/web43/user/web43_lorello/.forward
web43_sciarab:x:10196:10043:Michele Sciarabba:/home/www/web43/user/web43_sciarab:/dev/null
05.09.2006 - 16:42:11 => INFO - /root/ispconfig/scripts/lib/config.lib.php, Line 868: setquota -u web
43_sciarab 204800 204800 0 0 -a &> /dev/null

...

05.09.2006 - 18:34:13 => INFO - /root/ispconfig/scripts/lib/config.lib.php, Line 868: setquota -u web
43_massi.pilati 51200 51200 0 0 -a &> /dev/null
05.09.2006 - 18:34:13 => INFO - /root/ispconfig/scripts/lib/classes/ispconfig_procmail.lib.php, Line
57: cp -f /root/ispconfig/isp/conf/forward.master /home/www/web43/user/web43_massi.pilati/.forward
web43_manto-gas:x:13980:10043:manto-gas:/home/www/web43/user/web43_manto-gas:/dev/null
05.09.2006 - 18:34:14 => INFO - /root/ispconfig/scripts/lib/config.lib.php, Line 868: setquota -u web
43_manto-gas 0 0 0 0 -a &> /dev/null
05.09.2006 - 18:34:14 => INFO - /root/ispconfig/scripts/lib/classes/ispconfig_procmail.lib.php, Line
57: cp -f /root/ispconfig/isp/conf/forward.master /home/www/web43/user/web43_manto-gas/.forward



I've another problem with this big domain: usernames containing uppercase letters doesn't receive messages and they can't access webmail... they can't do anything :-)

In logs I can see:


05.09.2006 - 18:34:15 => WARN - /root/ispconfig/scripts/lib/config.lib.php, Line 937: WARNING: could
not setquota -u web43_Ake 0 0 0 0 -a &> /dev/null
web43_Ake:x:::::/dev/null

05.09.2006 - 18:34:15 => WARN - /root/ispconfig/scripts/lib/config.lib.php, Line 937: WARNING: could
not setquota -u web43_Micotti 0 0 0 0 -a &> /dev/null
web43_Micotti:x:::::/dev/null

05.09.2006 - 18:34:16 => WARN - /root/ispconfig/scripts/lib/config.lib.php, Line 937: WARNING: could
not setquota -u web43_Umberto.Micotti 0 0 0 0 -a &> /dev/null
web43_Umberto.Micotti:x:::::/dev/null

[till]

Uppercase letters are not supported in linux usernames, thats why you did not receive emails for them.

[lorello]

Hi lorello,

since I'm stuck with an Ensim 3.1 box, too, I'd like to inquire whether you'd mind sharing this gem ?


Hi, I've not made a general purpose toolkit to make my migration, I've only used the Remoting Framework of ISPConfig to create quickly 50% of my sites, the other half has been created manually.

My old server with ensim has died when I was programming the migration and writing scripts to make it easier, so I've done many errors and if I could return on my steps I should make it better....

The steps I've followed are (should be):

- move backups of sites, database, majordomo, mailqueue, users' home, user preferences for squirrelmail from ensim box to ispconfig box
(exclude in mysql dumps the commands to create & select database, because they will change names on the ISPconfig box)

- launch many times a script that using remote framework create quickly: client, site, dns records for mail.domain.tld and mysql database (i've made it all with the same password for admin users, database etc.., we could do it better...)

- launch another script many times to restore backup of sites' files, set correct ownership & permissions.

- restore mysql database (i've done it manually, because i've made )

- recreate users on ispconfig sites: a) manually for small sites, because you don't need a default admin user as on Ensim, so you can change something during the migration; b) automatically for some bigger domains, because i've one domain with more than 3200 users...

- foreach domain, launch a script that take old mailboxes in mbox format and convert it in Maildir format (and write a log where you can read what mailboxes you have forgotten to create)
- install squirrelmail on ispconfig server and import user preferences & addressbook (i've not done it until now)

I've found two main problems after migration:

1. mixed uppercase & lowercase usernames (Some.User@some_ensim_domain.tld) that ISPConfig doesn't like.

2. ISPConfig performances running big domains: my old box was AMD 2GHz of cpu, 1Gb of ram, one ATA disk, using MailScanner (Spamassassin+clamd), my new box is a dual Xeon 3GHz with 2Gb of ram and 2 SCSI disks but if I activate spamassassin & clamd an all users, as it was on the old server, my load average jump over 200 (!!!) and the server became unusable. I think this depends on how ispconfig use spamassasin&clam (through procmail), I must work on this now to remove spam in earlier stages (on handshake, with RBLs, greylistings, etc...).

I hope this rapid notes could help you, ask on the forum if you want more info... I could have forgotten some steps.

See you,
lorello.

[till]

2) The problem is that ISPCOnfig calls spamassassin instead of the spamc / spamd combination. We can not simply switch beacuse spamd is not able to read the current spamassassin config files. A solution might be to symlink from the file location spamd expects to the current spamassassin config file for every useraccount. We will start working on a solution for that in the ISPConfig dev branch shortly.

[lorello]

2) The problem is that ISPCOnfig calls spamassassin instead of the spamc / spamd combination. We can not simply switch beacuse spamd is not able to read the current spamassassin config files. A solution might be to symlink from the file location spamd expects to the current spamassassin config file for every useraccount. We will start working on a solution for that in the ISPConfig dev branch shortly.

Thanks Till, in the mean time we have disabled spamassasin for the 3 thousand users of our bigger domain and we are configuring graylisting and adding RBL in postfix, trying to reduce the amount of mail received from each account.

At now we have some time during each day where the load average jump over 100 for the concurrent presence of many clamscan processes.

There is also another task that is always present during the day, but I suppose it should terminate during the night

15111 root 39 17 3084 904 624 R 99.9 0.0 5:20.82 grep -iw postfix /var/log/mail.log.07-09-06_23-59-01

I will study some code...