ISPConfig3 plugable mail subsystem (for Google Apps) and nginx as alternative

[eugenesan]

Hi all,

Recently I was forced to migrate from dedicated to VPS.

Maybe I am too optimistic, but after small battle with low resources I've came with 3 decisions:

1) ISPConfig3 will be the CP (mydns support and single language code were main reasons).

2) Google Apps will replace mail subsystem (Saves tens of MB and solves spam headache). Therefore whole mail/spam supporting code must be disabled and replaced with simple DNS settings.

3) Nginx will be used instead of Apache.

Now, since I am pretty new to ISPConfig and a lot of work expected, it will be really nice to hear your ideas and if I'll have some luck even receive some code drafts from you ;-)

BTW: If some is interested I've built updated Ubuntu source for MyDNS 1.2.8.15.

[till]

2) There has no code to be changed. Just installl ispconfig 3 and then go to system > users and uncheck the checkbox for mail for the admin user.

3) Nginx is not supported, but you can write plugin for it which simply replaces the apache plugin. Just take a look at the apache plugin in /usr/local/ispconfig3/server/plugins-available/ and write a new plugin for nginx.

[eugenesan]

Thanks for the tip.

I am currently trying to understand relation between object/modules.
Are they all totally independent when items created?

For ex. when I add add site, does ISPC creates also DNS records for it?

Is ISPC will be affected by disabling all mail related services? Is it going to restart them at some point ad fail, lets say on DNS changes?

[till]

Are they all totally independent when items created?

yes.

For ex. when I add add site, does ISPC creates also DNS records for it?

no.

Is ISPC will be affected by disabling all mail related services? Is it going to restart them at some point ad fail, lets say on DNS changes?

No. ISPConfig is a multi server control panel, ist is built so that services are independant and dont have to be installed on the same server or be installed at all.

[eugenesan]

Thanks for information.
After install script inspection, I've also noticed same level of modules separation on install stage, very smart!

Are there any plans for more "user-friendly" modules? Maybe some work already done in this area?

And are myDNS System->Server Config->Nameservers used as upstream DNS?

Automated DNS zone, as an option, on site creation would be really nice.
In addition two NS (ns1,ns2) records by default is good choice also.

BTW: 3rd version looks very nice and solid, nice work.

[till]

The modules are laready very user friendly. They are separated as this is nescessary and this will not be changed. Otherwise you would not be able to run ispconfig without mail system ;)

Automated DNS zone, as an option, on site creation would be really nice.

We will add some more wizards later.

In addition two NS (ns1,ns2) records by default is good choice also.

Thats already included. Take a look at the dns wizard.

[eugenesan]

Looks like I was playing with too old version (3.0.0.7) :-(.

Just found latest RC :-)

[eugenesan]

Hi all,

After many hours of learning, coding and testing I have nginx plugin for ISPC3.
Currently it lacks SSL support but I've added protected folders support, hope it can be easily extended for apache plugin.
I'll post it soon for review.

Meanwhile I am going to deal with automatic DNS creation.
Here I need help, what would be most rational way to perform that?
I was thinking of creating tasks for DNS creation when apropriate option selected on site creation.
How can I "stream" such an actions to sys_datalog?

[eugenesan]

Hi all,

Here it comes, Nginx support for ISPC.

Basic support only:
1) No installation support
2) No automatic configuration of ISPC web.
3) No SSL support.
4) Currently only supports php-fpm as php fastcgi engine.
5) Tested and built on Debian based system, sorry rpm guys.

Few notes for maintainers:
*) I had to dos2unix few files so take it into account when diffing.
*) I've changed in some places hard coded references to apache by webserver.

To use:
*) Install php-fpm and mydns (google or contact me for patches for Debian based distros)
*) apply patch
*) install without web
*) configure ISPC in nginx by adding /etc/nginx/available-sites/ispconfig with next content:

# ISPConfig virtual host

server {
listen 8080;
server_name _;

access_log /var/log/nginx/ispconfig.access.log;

location / {
root /usr/local/ispconfig/interface/web/;
index index.php;
}

# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
location ~ \.php$ {
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME /usr/local/ispconfig/interface/web//$fastcgi_script_name;
include /etc/nginx/fastcgi_params;
}

# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
location ~ /\.ht {
deny all;
}
}

make link to /etc/nginx/enabled-sites/ and restart nginx.
If I didn't forget something, ISPC should be operational.
*) Now enter ISPC web and go to system->edit server->select server->config. Here modify next:

[web]
website_path=/var/clients/client[client_id]/web[website_id]
website_symlinks=/var/www/[website_domain]/:/var/clients/client[client_id]/[website_domain]/
vhost_conf_dir=/etc/nginx/sites-available
vhost_conf_enabled_dir=/etc/nginx/sites-enabled

[global]
webserver=nginx


*) if brave and going to update existing DB perform next changes:


- `apache_directives` text,
+ `webserver_directives` text,
+ `protected_folders` text,



Waiting for feedbacks.

[till]

Hi,

thank you very much for the patch. I'am not able to add it to SVN at the moment as there is a feature freeze until the final version but I will add it as soon as the final gets released.

One problem that I see with the patch is that you renamed some fields in the database and the logging directory which breaks the compatibility to existing ispconfig installations so that an update will fail when the patch gets applied to svn. We will have to find a workaround for this or rename the fields and folders to its original names.

[eugenesan]

Hi,

I understand the problem. But on other hand calling everything webserver related as "apache" is not good idea :-).
May you can push only changes that are not feature related as tables and folders names? That won't cause any problem now and will solve problems in future. Anyways no body was expecting RC to be 100% compatible to final. right?

[till]

With RC1 we had a complete feature freeze, so nothing that is not a bugfix gets into SVN anymore until we release the final version. Especially renaming tables is not possible during feature freeze. To rename the tables we would have to expand the installer first and add conditional code that repairs the installations after the table changes. There are already several thousand installations of ISPConfig 3 out there and I know several which are production systems, so we can not even break the compatibility between RC1 and Final.

I really like your plugin but it can be integrated first after the final release.

[eugenesan]

No problem, I am not on hurry with the patch.
I was trying to ease the pane of core modification.

Later, after I'll finish testing and fixing possible nginx runtime problems, I'll work on installation part.

Currently I've found some DB related issues and reported them in bug tracker, take a look at them please. They can become real pain for un-experienced user.

Now, I still want to add auto-DNS feature.
Seems it's not too difficult, but I need your help. I've created some DNS templates for Google Apps, normal templates already exists.
Currently I wonder how to initiate creation of records based on DNS-template and data provided for site that is going to be created. Missing data may be built using default prefixes as ns1, ns1 and so on.
What do you think?

BTW: I found framework used for ISPConfig as very nice :-).

[eugenesan]

Hi all,

I've prepared next version of patch.
Now it supports:
1) Seamless subdomain (without redirects).
2) Default config files and read-me inside (requires manual modification)
3) Optional support for user-friendly folders names. Renaming of sites and users supported. All paths will be updated automatically.
4) Added default DNS template for GoolgeApps.
5) Management of php-fpm pools, now PHP environments are separated (suFPM)
6) Synced with trunk-30/1/2009

Good luck.

[eugenesan]

Hi,

Some bugfixes and new features as webserver directives (regex locations) and php childs control.
Bug fixes and probably something I've forgot.

[KenMasters]

Thanks for this! I'm highly interested in this patch, and was wondering if there is a howto you'd recommend for the nginx install. Most of the howtos I'm seeing are using pre-built packages which are out of date or aren't using the far more efficient php-fpm patch for the php install. I hope to test your patch tonight, and will be keeping an eye out for new versions!

[eugenesan]

Hi,

I was planning to prepare some documentation, but not sure when I'll be find time for that :-(

I am testing updated version of ISPConfig patch and will try to release some additional information regarding building/installing/configuring nginx/php-fpm with ISPConfig, soon.

At the moment there is very simple readme inside the patch, I've posted earlier.
I can't post patches here, since forum has 100k limit. I'll check my options and post back.

Contact me for clarifications.
Good luck

[weldpua2008]

Hi,

I was planning to prepare some documentation, but not sure when I'll be find time for that :-(

I am testing updated version of ISPConfig patch and will try to release some additional information regarding building/installing/configuring nginx/php-fpm with ISPConfig, soon.

At the moment there is very simple readme inside the patch, I've posted earlier.
I can't post patches here, since forum has 100k limit. I'll check my options and post back.

Contact me for clarifications.
Good luck
Hello
I'm happy that ISPconfig3 start support for nginx. But...
I think that nginx(or other like that reverse proxy server: lighttp) can used in 3 ways:


nginx(or other like that) revers proxy all query to apache server for better performance(less memory use than apache process or other).

nginx (or other like that) for revers proxy query to apache server for websites, and one directive for ispconfig3 - only nginx+php-cgi

integrate nginx (or other like that) to ispconfig3, like suphp, php-cgi,modphp that can users chose apache or other server.

[eugenesan]

Hi,

First I want to state that nginx is not less functional then apache in most cases.
My target was VPS oriented hosting server with control panel.
My intention was eliminate mail server usage (thanks to gmail) and totally replace apache+libphp with nginx+php-fcgi-fpm and not to add nginx as reverse proxy.

Currently I am successfully running desired configuration and I am more then happy :-)
Footprint, security and speed are far beyond my expectations.
I was able to add to ISPConfig, support for nginx, with as much features as majority of websites would require.

I hope to find some time and cooperation from ISPConfig maintainers, to share my achievements with others, soon.

[weldpua2008]

Hi,
First I want to state that nginx is not less functional then apache in most cases.

I know, but many people's use it's features by default, and they need them.
Thats why I want implement to ISPconfig feature: chose your http-server. But I think that apache must bind on 127.0.0.1, and gateway to apache must be revers proxy application - for many reasons, but I don't think that I has right's to chose htp-server thats why ISPconfig3 must support nginx in many way's.
When I can see in svn your changes I can try write some patches.

My target was VPS oriented hosting server with control panel.
My intention was eliminate mail server usage (thanks to gmail) and totally replace apache+libphp with nginx+php-fcgi-fpm and not to add nginx as reverse proxy.

My target too.

[till]

ISPConfig has a plugin system for the server configuration part. If you want to add another webserver or implement a different way a webserver shall be configured, just write a new plugin like eugenesan did. Change in the existing apache plugin that change the way apache is configured in a default ispconfig setup will not be accepted as the ispconfig project trys always to be downwards compatible. but as long as you implement your features as new plugins there is no problem to add them to svn.

[KenMasters]

Hi,

First I want to state that nginx is not less functional then apache in most cases.
My target was VPS oriented hosting server with control panel.
My intention was eliminate mail server usage (thanks to gmail) and totally replace apache+libphp with nginx+php-fcgi-fpm and not to add nginx as reverse proxy.

Currently I am successfully running desired configuration and I am more then happy :-)
Footprint, security and speed are far beyond my expectations.
I was able to add to ISPConfig, support for nginx, with as much features as majority of websites would require.

I hope to find some time and cooperation from ISPConfig maintainers, to share my achievements with others, soon.

I, for one, am anxiously awaiting your configuration details. With the lack of decent documentation for ISPConfig 3, I'm having trouble with getting mail to point to an external mail server (Google Apps). Thanks for your hard work on this!

[till]

Pointing email tp google apps has nothing to do with the mail system itself. Just create a dns mx records tha points to googles servers as google describes it in their instructions.

[KenMasters]

Pointing email tp google apps has nothing to do with the mail system itself. Just create a dns mx records tha points to googles servers as google describes it in their instructions.

Let me clarify: I already have Google setup via my primary external DNS servers, but for ISPConfig2, I had to specify an external mail server for each domain and create the appropriate local DNS MX records to force my php applications and server (postfix) to relay to Google's servers for sending email, as described HERE (http://www.howtoforge.com/forums/showpost.php?p=21462&postcount=5). Has this changed to be automatic if I simply set the local DNS MX records to point to Google's servers? Thanks again!

[till]

ISPConfig 3 does not create automatic mail server entry for domains as ispconfig 2 did, so you dont have to configure anything in ispconfig 3 if you use a external mailserver like google.

[KenMasters]

ISPConfig 3 does not create automatic mail server entry for domains as ispconfig 2 did, so you dont have to configure anything in ispconfig 3 if you use a external mailserver like google.

Okay, fair enough, so you're saying that my server will automatically know to use Google's mail servers for sending email for each domain? That seems almost too simple. I've been hesitant to migrate to ISPConfig3 because of this issue, so I guess my fears were unfounded. I can't wait for the new release, and then some docs for the new interface -- which is VERY slick I might add. Nice work. :D

[masky]

Is this patch available as part of 3.0.1.3? Can someone provide instructions to swtich ISPConfig 3.0.1.3 to use Nginx?

Thanks in advance
Masky

[till]

Is this patch available as part of 3.0.1.3?

No. The patch is not part of ISPConfig as it renames several database fields which makes a patched system incompatible with any future ISPConfig updates and also breaks all installed systems if we would introduce it into the main branch. If I will find the time I will try to rewrite the patch so that it uses the correct database fields of ISPConfig so that it will not break exsiting systems.

[eugenesan]

Hi,

Actually I am not sure I am breaking anything in DB structure, I was just adding new fields.
But, unfortunately, I had to perform some changes in Webserver area that may be incompatible with already installed systems.

BTW: Since last time I've posted my patch I've enhanced my local installation and new patch must be prepared if some one is planning to use it.

Don't hesitate to contact me if you need.

[till]

If you have a new patch, I will try to find the time to rewrite it to make it compatible with installed releases. Please post the patch to the bugtracker so that it gets part of the review process.

http://bugtracker.ispconfig.org

[eugenesan]

Ok, I'll have to prepare the patch first :-)
I'll update you with details soon.

[blackbumer]

Hi all,

Here it comes, Nginx support for ISPC.

Basic support only:
1) No installation support
2) No automatic configuration of ISPC web.
3) No SSL support.
4) Currently only supports php-fpm as php fastcgi engine.
5) Tested and built on Debian based system, sorry rpm guys.

Few notes for maintainers:
*) I had to dos2unix few files so take it into account when diffing.
*) I've changed in some places hard coded references to apache by webserver.

To use:
*) Install php-fpm and mydns (google or contact me for patches for Debian based distros)
*) apply patch
*) install without web
*) configure ISPC in nginx by adding /etc/nginx/available-sites/ispconfig with next content:

# ISPConfig virtual host

server {
listen 8080;
server_name _;

access_log /var/log/nginx/ispconfig.access.log;

location / {
root /usr/local/ispconfig/interface/web/;
index index.php;
}

# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
location ~ \.php$ {
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME /usr/local/ispconfig/interface/web//$fastcgi_script_name;
include /etc/nginx/fastcgi_params;
}

# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
location ~ /\.ht {
deny all;
}
}

make link to /etc/nginx/enabled-sites/ and restart nginx.
If I didn't forget something, ISPC should be operational.
*) Now enter ISPC web and go to system->edit server->select server->config. Here modify next:

[web]
website_path=/var/clients/client[client_id]/web[website_id]
website_symlinks=/var/www/[website_domain]/:/var/clients/client[client_id]/[website_domain]/
vhost_conf_dir=/etc/nginx/sites-available
vhost_conf_enabled_dir=/etc/nginx/sites-enabled

[global]
webserver=nginx


*) if brave and going to update existing DB perform next changes:


- `apache_directives` text,
+ `webserver_directives` text,
+ `protected_folders` text,



Waiting for feedbacks.

is it right method of isntallation patch :
cd /usr/local/src/
svn checkout svn://svn.ispconfig.org/ispconfig3/trunk/
mv trunk ISPConfig-trunk
patch -p1 ../ispconfig-trunk.nginx-0.1.patch

should i apply all patches in this thread or just last one? (ispconfig-trunk.nginx-0.2.0.4)

[eugenesan]

Hi,

Only the latest patch should be applied, but I suspect there maybe a problem with last one. Seems like several files slipped into "original" tree before I've created patch.

At the moment I have running setup but can't find time to "rediff" it with latest version. In addition some functionality still not documented :-(

I can try to pack a tarball which can be used to merge with latest ISPC version.
What do you think?

[blackbumer]

it will be very nice !!!
very need it :)

[cubells]

I'm eager to test it!!

[eugenesan]

Hi,

At the moment I am "extracting" changes from local setup to installable tree.
I still need to prepare SQL template, that can take some time.
Hope to finish very soon, maybe tomorrow...

[till]

Few notes for maintainers:
*) I had to dos2unix few files so take it into account when diffing.

Normally all files should be with unix linebreaks. Do you know which filesw ere wrong?

*) I've changed in some places hard coded references to apache by webserver.

Thats sad as it breaks the compatibility with the existing ISPConfig 3 installations and will prevent that we add this patch to ispconfig main branch.

[eugenesan]

Normally all files should be with unix linebreaks. Do you know which filesw ere wrong?

I didn't write down file names, but when I'll post a patch we probably will see it.


Thats sad as it breaks the compatibility with the existing ISPConfig 3 installations and will prevent that we add this patch to ispconfig main branch.
Yes, I am aware of it, but as I recall that was the only rational solution.
Actually I am pretty sure everything will continue to work for Apache too.
And we can always assume Apache is the default.

[till]

Yes, I am aware of it, but as I recall that was the only rational solution.
Actually I am pretty sure everything will continue to work for Apache too.

That depends on what you changed. If you changed the name of a database field e.g. the apache_directives field to webserver_directives, all installations with your patch can not be updated with a normal ispconfig update in future and are not able to install any of the bugfixes or possible future security fixes. If they would install an update or use the ispconfig update function, they would loose the complete server configuration in the database as of the incompatible db scheme.

[eugenesan]

Yes, here you right.

But, for example, if next update will include updates to both DB fields and references in code, we may solve the problem.

In addition you may consider adding my changes to next major release.
That way people who wish to evaluate nginx will use trunk or will have to wait for final release.
In future, people willing to move to that release, from older ISPC, will be required to perform minor adaptation to their DB.

Unfortunately I don't recall all changes I've performed and after I'll review final patch I may reply with concrete examples.

What do you think about my proposal?

[blackbumer]

Yes, here you right.

But, for example, if next update will include updates to both DB fields and references in code, we may solve the problem.

In addition you may consider adding my changes to next major release.
That way people who wish to evaluate nginx will use trunk or will have to wait for final release.
In future, people willing to move to that release, from older ISPC, will be required to perform minor adaptation to their DB.

Unfortunately I don't recall all changes I've performed and after I'll review final patch I may reply with concrete examples.

What do you think about my proposal?

we are waiting files from you :) and ready to test it

[till]

But, for example, if next update will include updates to both DB fields and references in code, we may solve the problem.

As far as I know you changed also ptahs e.g. for satistics. This will mess up everything on update.

The only otion that I see at the moment for an inclusion is to rewrite your addons so that it follows the ispconfig paths and databse field names.

I'am sorry but I really can not include a piece of software even in a next major release that will break thousands of exsiting server installations.

Nevertehless I will review your code and see If it can be made compatible with the released ispconfig versions as it will be less work to chnage your plugin then to cahnge the whole ispconfig code to make it compatible with your plugin.

[Keeper]

This topic is very interesting and I'm glad there is progress in getting nginx support to ISPC3. However, when this gets finalized I hope it does not require php-fpm. php-fpm still requires modifying PHP source manually, which means automatic security updates won't work.

[eugenesan]

Actually there is a project that should allow php-fpm as addon to PHP.
http://php-fpm.org/Main_Page

Unfortunately progress is almost stopped, since I am very busy lately.
If someone willing create/rebase patches based on older version, I am ready to release the code.