Possible ISPConfig bug?

[stefanm]

Hi Till, Hi Falko, Hi everyone,

I hope you can help me out with the following problem: We have cluster setup of ISPConfig 3.0.3 running on Debian 6 (setup follows your tutorial for Lenny).

So far we have about 30 domains configured on the system (all configured with fastcgi/suexec). With 4 or 5 of the web domains we had a strange effect: After creating the domains, the owner id in the filesystem is too low by exactly 2.

Example: We created a domain yxz.org, the ispconfig interface says:
Owner: web111 Group: client37
But the filesystem for the web domain is created with web109:client37
(and thus making suexec fail)

Do you have any idea why this is happening? The difference in numbers is exactly 2 in all cases and so far it happend with 4 or 5 sites, but these sites do not seem to be special somehow compared to all the other sites that were created correctly.

Where does ISPconfig get the information from which user:group to use when creating the filesystem?

Hope, you can help. I am a little bit lost here, since I have no idea where to start with searching the problem.

An additional note: All the ids that are generated on our system are uneven(web103,web105,web107,...), so it seems the use the id of the last web created before.

Thanks for your help!
Stefan

Update:
Just encountered the phenomenon. Here is what was configured and what happens on the filesystem (this time the difference is not 2 anymore). Really strange. Does perhaps the debug message not show the real command that is executed?




Debug output for creating the site:

9.11.2011 21:55 	s3.xxx.yyy 	Debug 	Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 	
29.11.2011 21:55 	s3.xxx.yyy 	Debug 	Processed datalog_id 1109 	
29.11.2011 21:55 	s3.xxx.yyy 	Debug 	Apache online status after restart is: 1 	
29.11.2011 21:55 	s3.xxx.yyy 	Debug 	Calling function 'restartHttpd' from module 'web_module'. 	
29.11.2011 21:55 	s3.xxx.yyy 	Debug 	Apache status is: 1 	
29.11.2011 21:55 	s3.xxx.yyy 	Debug 	Writing the vhost file: /etc/apache2/sites-available/test.abc.vhost 	
29.11.2011 21:55 	s3.xxx.yyy 	Debug 	Creating fastcgi starter script: /var/www/php-fcgi-scripts/web117/.php-fcgi-starter 	
29.11.2011 21:55 	s3.xxx.yyy 	Debug 	Disable SSL for: 	
29.11.2011 21:55 	s3.xxx.yyy 	Debug 	exec: chown web117:client41 /var/www/clients/client41/web117/log/error.log 	
29.11.2011 21:55 	s3.xxx.yyy 	Debug 	exec: chown web117:client41 /var/www/clients/client41/web117 	
29.11.2011 21:55 	s3.xxx.yyy 	Debug 	exec: usermod --groups sshusers web117 	
29.11.2011 21:55 	s3.xxx.yyy 	Debug 	exec: chmod 755 /var/www/clients/client41/web117/log 	
29.11.2011 21:55 	s3.xxx.yyy 	Debug 	exec: chmod 777 /var/www/clients/client41/web117/tmp 	
29.11.2011 21:55 	s3.xxx.yyy 	Debug 	exec: chmod 710 /var/www/clients/client41/web117/web 	
29.11.2011 21:55 	s3.xxx.yyy 	Debug 	exec: chmod 751 /var/www/clients/client41/web117/*
29.11.2011 21:55 	s3.xxx.yyy 	Debug 	exec: chmod 777 /var/www/clients/client41/web117/tmp 	
29.11.2011 21:55 	s3.xxx.yyy 	Debug 	exec: chmod 710 /var/www/clients/client41/web117/web 	
29.11.2011 21:55 	s3.xxx.yyy 	Debug 	exec: chmod 751 /var/www/clients/client41/web117/* 	
29.11.2011 21:55 	s3.xxx.yyy 	Debug 	exec: chmod 751 /var/www/clients/client41/web117/ 	
29.11.2011 21:55 	s3.xxx.yyy 	Debug 	Creating Symlink: ln -s /var/www/clients/client41/web117/ /var/www/clients/client41/test.abc 	
29.11.2011 21:55 	s3.xxx.yyy 	Debug 	Creating Symlink: ln -s /var/www/clients/client41/web117/ /var/www/test.abc 	
29.11.2011 21:55 	s3.xxx.yyy 	Debug 	Moving site to new document root: mv /var/www/clients/client0/web117 /var/www/clients/client41 	
29.11.2011 21:55 	s3.xxx.yyy 	Debug 	Removed Symlink: rm -f /var/www/clients/client0/test.abc 	
29.11.2011 21:55 	s3.xxx.yyy 	Debug 	Calling function 'update' from plugin 'apache2_plugin' raised by event 'web_domain_update'. 	
29.11.2011 21:55 	s3.xxx.yyy 	Debug 	Calling function 'ssl' from plugin 'apache2_plugin' raised by event 'web_domain_update'. 	
29.11.2011 21:55 	s3.xxx.yyy 	Debug 	Found 1 changes, starting update process.

and this is how the filesystem for the domain looks like:

root@s3 /var/www/test.abc # ls -l
total 36K
drwxr-x--x 2 web111 client41 4.0K Nov 29 21:54 cgi-bin
lrwxrwxrwx 1 web111 client41   33 Nov 29 21:54 log -> /var/log/ispconfig/httpd/test.abc
drwxr-x--x 2 web111 client41 4.0K Nov 29 21:54 ssl
drwxrwxrwx 2 web111 client41 4.0K Nov 29 21:54 tmp
drwx--x--- 4 web111 client41 4.0K Nov 29 21:54 web

Update 2:
It seems that at some point there went something terribly wrong in ispconfig. I compared the master and the slave server and the user accounts are not in sync anymore. It seems that at some point when we deleted two unused shell accounts, these were correctly removed from the slave server, but not from the master server. So the passwd on the master has two additonal accounts which leads to different uid/gid for all accounts that were created after the failed delete. It seems that these two leftover accounts cause the above phenomenon, since they are not anywhere in ispconfig, but still exist in the passwd. On the slave, where the two accounts were correctly removed, the error explained above, does not occur and uid/gid are correct in the filesystem.
However that leaves me with quite a problem. On the master I have the mess with the wrong uids and the slave is also not usable, because it actually got the uid/gid that the master selected, but they do not match the uid/gid in the passwd.
Any suggestions how to get the servers back in sync and what might have caused the error with the two shell accounts?

[stefanm]

Hi Till, Hi Falko,

do you perhaps have any suggestions how to cope with the above problem?
It seems that my only choice might be to delete all web domains and shell users that were created since the first false shell account, remove the two orphaned accounts manually from passwd, create all the accounts and hope, that this will help ispconfig to recover. A horrible lot of work

And do you perhaps have an idea what causes the error? Recreating all the accounts might help to recover for now, but how long will it take until the next undeleted shell account.
Sigh, ispconfig is really a great panel and I am using it since several years now, but this is really a downfall.
Any suggestions or help is greatly appreciated!

Stefan

[till]

I have several miultiserver systems and test systems here but I never had it yet that a shell account was not deleted when you delete it in ispconfig.

What you can do is that you change the uids of the ispconfig users that were different in the passwd file on one server so that both files match again and then chown -R the affected website directories manually to this user and group.

We have a bugreport about this issue here:

http://bugtracker.ispconfig.org/inde...s&task_id=1518

The issue can happen if you add a shell user manually on one system without adding it on the second server as well. The only solution for that is to use a fixed uid range that ispconfig enforces on both servers, thats a feature which is on our todo list already but it requires some bigger changes.

[stefanm]

Hi Till,

had a quick look at your sources and noticed that normal shell users are deleted with "userdel -f", while chrooted users are only deleted with "userdel" though the jailkit-Plugin. Do you have a specific reason for omitting the -f flag?
Might the bug simply arise, when a chrooted user that is deleted still has an open shell to the system? Might also explain, why the users get correctly deleted from the slave, since it is only used for failover purposes and normally has no actives shell accesses.
If I found the right parts in the source (apache2_plugin), the web-domain accounts (webXX) are deleted without the -f flag, too. I am not quite sure, but I think userdel without the -f might fail, if the account has still a process running, so together with suxec, it might be that there simply was access to the website while it was deleted ? (Would explain the behaviour described in the bug report, you pointed me to)

Stefan