ISPConfig could be great but its riddled with issues. How to fix?

[yucktoufoo]

I've installed a multiserver setup (Centos 5.5) as per the guides (which are very good) but it seems I get different results every time.

Firstly, built a BIND DNS server and the monitor page repeatedly tells me MyDNS isn't running, also drops the zone files in the wrong place for a chrooted bind install.

Then built a mailserver, setup my domains to find I am unable to add a fetchmail setup with an alias address as the destination mailbox. Perhaps there is some golden rule about not doing this but I modified the database by hand and everything seems to work as expected.

Then built a secondary mailserver only to find I am unable to host the same domain on both servers (i do not wish the two servers to be mirrors, i just want to host a single domain on both servers).

Then I see I am unable to receive mail at the latest mail server I built, getting the error

pipe_command: execvp /usr/bin/maildrop: No such file or directory

I thought that path was just incorrect, and that is should be /var/spool/postfix/maildrop but when I started to poke around in the server table and found the following in the [MAIL] section...

pop3_imap_daemon=courier
mail_filter_syntax=maildrop

Well I don't know why it thinks courier is used, its a brand new build and courier has NEVER been installed. I guess it should read...

pop3_imap_daemon=dovecot
mail_filter_syntax=sieve

I also noted that removing uninstalling and reinstalling ispconfig on the remote mailserver results in another server added to the UI. This seems a bit silly to me if the server has the same hostname and ip addresses.

There are loads of other things I've had to manually modify in the database but that's really scary as some of these things can easily be "un-done" by innocently updating an alias record in the UI or by an ispconfig update.

I think its odd that someone would store an entire ini file in a single column in a database table. Everything is also done with raw SQL in the PHP scripts, now I've never developed anything in PHP by in my past experience, if your database engine supports stored procedures then you should use them. It would certainly make fudging the database a bit easier.

I am also amazed that there are no foreign key constraints on the database. That could be a real problem for a big installation with lots of managed hosts.

I did head over to the bugtracker site for ispconfig but I wasnt able to add any new tickets and any existing tickets in the system didn't look like they were being worked on.

Now, I assume its bad mojo and deemed UNSAFE to edit the content of the dbispconfig database manually, I know this but I have to make changes to get it to work properly. I'd like other peoples thoughts on how likely it is that these modifications will be overwritten by ispconfig updates....

As for my alias problem, I am thinking of adding a trigger to the mail_get table to ensure that the destination address is always forced to the correct alias address of someone updates the row.

Add a stored procedure or trigger to parse the config column of the server table to replace any instances of 'MyDNS' with 'bind' instead as I do not use MyDNS. I will probably do the same again to fix my maildrop issue.

I'd like to know how data flows between the various instances of mysql. I will do some digging through the PHP code later but if someone has any quick answers on this I would be greatful. Presumably the remote installations connect back to main ISPConfig database (as I've only allowed MySQL from client to server), push some stuff into the central database and pull any changes down at the same time?

On this basis, when I make the changes to the config column, am I better to do this on the remote server or the central ISPConfig server?

I really don't know what I will do to work around hosting the same email domain on two different servers, any pointers greatly appreciated.

[till]

I did head over to the bugtracker site for ispconfig but I wasnt able to add any new tickets and any existing tickets in the system didn't look like they were being worked on.

They are all being worked. If you see our blog you can see that we release new bugfix releases every 1-2 months. Yyou can post new items after you registered yourself in the bugtracker.

I think its odd that someone would store an entire ini file in a single column in a database table. Everything is also done with raw SQL in the PHP scripts, now I've never developed anything in PHP by in my past experience, if your database engine supports stored procedures then you should use them. It would certainly make fudging the database a bit easier.

Thats not odd at all and it works fine. Its much better to use a ini file instead of having a new column for every entry and the ini file is stored in mysql and not the filesystem as this includes it into the internal replication mechanism of ispconfig.

Now, I assume its bad mojo and deemed UNSAFE to edit the content of the dbispconfig database manually, I know this but I have to make changes to get it to work properly. I'd like other peoples thoughts on how likely it is that these modifications will be overwritten by ispconfig updates....

Yes it is unsafe and will break your setup. If you think that there is something missing, feel free to make a feature request in the bugtracker.

Presumably the remote installations connect back to main ISPConfig database (as I've only allowed MySQL from client to server), push some stuff into the central database and pull any changes down at the same time?

Yes. This makes the setup very fault tolerant. The master server keeps a transaction log and when a slave is offline for a while, it will sync itself with the master when it come back online.

Regarding your mail issues, please look at the setup guides and install your system exactly as described there. If you install packages in different locations or outside the path, then they can not be recognized by the installer.

Add a stored procedure or trigger to parse the config column of the server table to replace any instances of 'MyDNS' with 'bind' instead as I do not use MyDNS. I will probably do the same again to fix my maildrop issue.

This is not nescessary if you setup the server correctly before you install ispconfig. The issues that you describe are all caused by a incompatible setup which makes it impossible for the ispconfig installer to recognize the system components. For a correct setup, please read the perfect setup guides.

I really don't know what I will do to work around hosting the same email domain on two different servers, any pointers greatly appreciated.

Thats covered in this guide:

http://www.howtoforge.com/installing...th-ispconfig-3

ISPConfig is doing all the configuration mirroring for as many mirrors as you need. Then a shared filesystem that is used for the /var/vmail directory is used for data mirroring.

[yucktoufoo]

Originally Posted by till

They are all being worked. If you see our blog you can see that we release new bugfix releases every 1-2 months. Yyou can post new items after you registered yourself in the bugtracker.

I will check this out. I registered for the bug tracker but was unable to find any options for creating a new ticket.

Originally Posted by till

Thats not odd at all and it works fine. Its much better to use a ini file instead of having a new column for every entry and the ini file is stored in mysql and not the filesystem as this includes it into the internal replication mechanism of ispconfig.

I've worked on possibly some of the largest database installations in the UK and disagree with this comment but I guess we will have to agree to disagree

Originally Posted by till

Regarding your mail issues, please look at the setup guides and install your system exactly as described there. If you install packages in different locations or outside the path, then they can not be recognized by the installer.

I will go back over the guide but I have followed it several times now and am 99.9998% sure that I did everything as per the guide, certainly for the DNS servers which think they have MyDNS installed. The latest mailserver was a 32bit build rather than 64bit, perhaps this is where I have problems?

Originally Posted by till

This is not nescessary if you setup the server correctly before you install ispconfig. The issues that you describe are all caused by a incompatible setup which makes it impossible for the ispconfig installer to recognize the system components. For a correct setup, please read the perfect setup guides.

Hmm, I've followed that guide numerous times now so Im fairly sure I am doing everything correctly. How exactly does ispconfig determine if mydns is installed?

Originally Posted by till

Thats covered in this guide:

http://www.howtoforge.com/installing...th-ispconfig-3

ISPConfig is doing all the configuration mirroring for as many mirrors as you need. Then a shared filesystem that is used for the /var/vmail directory is used for data mirroring.

I dont have time to check that out but will have a look later this evening but I am not sure this is what I want, I do not want to mirror any of the mail data. The second server is internal and has no knowledge of the external server. The internal server should getmail from the external and simply inject it into the local server under the same recipient domain.... is this not supported?

[till]

I will check this out. I registered for the bug tracker but was unable to find any options for creating a new ticket.

There should be a "Add New Task" Button in the upper menu after you logged in.

I've worked on possibly some of the largest database installations in the UK and disagree with this comment but I guess we will have to agree to disagree

I think so I guess these were not mysql setups? Mysql has limits in the overall length of column names for a database table and we also needed a flexible solution.

I will go back over the guide but I have followed it several times now and am 99.9998% sure that I did everything as per the guide, certainly for the DNS servers which think they have MyDNS installed. The latest mailserver was a 32bit build rather than 64bit, perhaps this is where I have problems?

Thats should not make a difference. Please check with the command:

which dovecot

if the dovecot binary is in the path.

I dont have time to check that out but will have a look later this evening but I am not sure this is what I want, I do not want to mirror any of the mail data. The second server is internal and has no knowledge of the external server. The internal server should getmail from the external and simply inject it into the local server under the same recipient domain.... is this not supported?

Thats not supported, at least not without mirroring as ispconfig ensures for consistency that a domain can only be added once per cluster.

Why do you want to use getmail for this setup instead of a postfix transport? With a postfix transport, you setup a transport on the external server for the domain, then add the allowed email addresses as relay recipients. Then add the domain as email domain for the internal server in ispconfig and add mailboxes.

[yucktoufoo]

Originally Posted by till

The issues that you describe are all caused by a incompatible setup which makes it impossible for the ispconfig installer to recognize the system components. For a correct setup, please read the perfect setup guides.

Forgot to query... Does this include not being able to set an alias/forward address as a destination for a new fetchmail setup as I think thats a bug/missing feature rather than an implementation thing.

[till]

Originally Posted by yucktoufoo

Forgot to query... Does this include not being able to set an alias/forward address as a destination for a new fetchmail setup as I think thats a bug/missing feature rather than an implementation thing.

Adding a fetchmail job for the alias of a mailbox instead of the mailbox itself makes the list much larger as users tend to have many aliases for a single mailbox and so might become confusing. Thats why we have not added it.

But there is no technical reason why it should not work for aliases as well.

[yucktoufoo]

Originally Posted by till

I think so I guess these were not mysql setups? Mysql has limits in the overall length of column names for a database table and we also needed a flexible solution.

A mixture of MySQL, MSSQL & Postgres & DB2 and in some cases all flavours together. Some of these others have stupidly impossible limits on naming too

Originally Posted by till

Thats should not make a difference. Please check with the command:

which dovecot

if the dovecot binary is in the path.

Ahh.. now see, here's the thing, which is not installed. Presumably that answers my question although does this mean if dovecot is not found, ispconfig automatically assumes that courier is installed instead?

That leads me to think there are two problems here. ISPConfig should complain if which is not installed and also probably shouldn't just default to courier.

Originally Posted by till

Thats not supported, at least not without mirroring as ispconfig ensures for consistency that a domain can only be added once per cluster.

Why do you want to use getmail for this setup instead of a postfix transport? With a postfix transport, you setup a transport on the external server for the domain, then add the allowed email addresses as relay recipients. Then add the domain as email domain for the internal server in ispconfig and add mailboxes.

There are a few reasons for this. Initially I had no access to the external mail server to be able to configure transports so the internal server was simply configured to getmail from the external instead. Now that I am building a new external server I could do it via transport as you said but this would also require ports to be opened on our firewall which I was trying to avoid.


Aside from all of these small problems, where is the best place to get more documentation for under the bonnet of ispconfig? its incredibly difficult to support something when you don't understand how it works.

I purchased the user manual which will give me some bed time reading this week but I am looking for more in-depth knowledge of what happens behind the scenes.

[yucktoufoo]

Originally Posted by till

Adding a fetchmail job for the alias of a mailbox instead of the mailbox itself makes the list much larger as users tend to have many aliases for a single mailbox and so might become confusing. Thats why we have not added it.

But there is no technical reason why it should not work for aliases as well.

This is what I thought, for me it is not a problem to have big lists because I host a very controlled set of domains but perhaps if I revisit my getmail/transports setup as per your previous posting this issue will be null and void anyway.

[till]

although does this mean if dovecot is not found, ispconfig automatically assumes that courier is installed instead?

Yes, we should add that. As courier is the default pop3/imap daemon. Same with mydns and BIND.

SPConfig should complain if which is not installed

We should add that. Even if I've never seen a system yet where the Linux base utils had been missing.

Aside from all of these small problems, where is the best place to get more documentation for under the bonnet of ispconfig? its incredibly difficult to support something when you don't understand how it works.

I purchased the user manual which will give me some bed time reading this week but I am looking for more in-depth knowledge of what happens behind the scenes

Most infos are here in the forum. We do not have a full technical documentation available yet.

The basics are:

ISPConfig consists of two, mostly independant parts. The Server and the interface (/usr/local/ispconfig/interface and /usr/local/ispconfig/server). Server and interface communicate trogh the mysql db.

For the configuration, the most interesting part is the server. It is a very modular system consisting of modules and plugins (/usr/local/ispconfig/server/mods-available and /usr/local/ispconfig/server/plugins-available). If a plugin or module is enabled in your setup, then it has a symlink from the ...-enabled directory to the ...-available directory.
The modules corresposnd to the interface modules like dns, mail, web and database. So a server cluster can split into these parts. The modules provide / raise events like "mail_user_insert" or "mail_user_update" and the plugins can subscribe to these events and get called when such a event occurs. So the plugins do the actual config work by creating directories, copying files and executing commands etc.

So as an (very simplified) example, the difference between a mydns, powerdns and bind setup is basically which server plugin is enabled.

mydns: no plugin as mydns reads directly out of the db and needs no further configuration.
powerdns: Powerdns has its own mysql database as the ispconfig db layout is not compatible, so the powerdns plugin just syncs the records in the powerdns db with the ones in the ispconfig db.
Bind: The bind plugin writes / updates BIND config files when a DNS record gets changed. The place for the BIND config directrory can be configured in ispconfig under System > server config.

[yucktoufoo]

Originally Posted by till

We should add that. Even if I've never seen a system yet where the Linux base utils had been missing.

That system is a VPS provided by bytemark which I re-imaged this morning and would also have expected which to have been installed so yeah, probably should check for this.


I can't help but think that the whole process would have been much easier had I have used debian over CentOS. Either way it was a worth excersize and I've learnt a little in the process so can't be all bad.

Thanks very much for your assistance. I look forward to the day when there is some official tech documents but in the meantime I will take on board what you have said, between that and poking at the code im sure it will all become clearer over time.