Go Back   HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials > ISPConfig 3 > Developers' Forum

Do you like HowtoForge? Please consider supporting us by becoming a subscriber.
Reply
 
Thread Tools Display Modes
  #1  
Old 26th August 2010, 15:35
ispcomm ispcomm is offline
Senior Member
 
Join Date: Aug 2010
Posts: 166
Thanks: 19
Thanked 11 Times in 11 Posts
Default Web filesystem layout change

I am trying to imagine a large web host running a 5 figures numbers of sites on top of ispconfig and clustered filesystem storage (don't do this at home!).

Normally hosts solve this problem by installing more servers, each hosting a number of sites and perhaps 2-server clusters for high availability and processing power.

What if the whole directory tree was available to all servers of the host and what if each server could serve any of the hosts of the hoster? Load balancing on the fly would be possible and pushing low-traffic sites to weaker machines would be possible (by means of linux-vserver or firewall load balancing tricks).

A problem is that such a host would need to be very efficient on resource usage and have high degree of redundancy.

I would also need to share the web tree across the "workers" cluster of servers

I'm wondering if the current filesystem layout on web directories will hit it's limit at the symlink level when a directory approaches the 100K+ mark of symlinks... for each access to a file this giant directory will need to be scanned (btree/hash is fast... but) and there will be millions of small images that will be served.

Normally this kind of stuff is solved by just using 2+ levels of indirection, effectively using the filesystem as a search tree:

/var/sites/f/fi/fir/first-domain.com
/var/sites/s/se/sec/second-domain.com

You get the idea...

There are a number of advantages to adopt such a structure which should be obvious.

I'd like to open a discussion on the implications of such a structure and the possible implementation for ispconfig.

ispcomm
Reply With Quote
Sponsored Links
  #2  
Old 26th August 2010, 16:33
till till is offline
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 36,421
Thanks: 834
Thanked 5,501 Times in 4,330 Posts
Default

I dont see any bigger problems to use such a path with the current ispconfig system. To implement that, you would basically do the following steps.

1) Define a new placeholder for the "f/fi/fir/first-domain.com" path of the domain path that you can use in the paths and symlinks in the server config settings. e.g. name it [website_splitted_domain]
2) Implement the logic to replace the placeholder when the website path is set in the interface/web/sites/web_domain_edit.php file and the interface/lib/plugins/sites_web_domain_plugin.inc.php file.
3)
__________________
Till Brehm
--
Get ISPConfig support and the ISPConfig 3 manual from ispconfig.org.
Reply With Quote
  #3  
Old 26th August 2010, 16:54
ispcomm ispcomm is offline
Senior Member
 
Join Date: Aug 2010
Posts: 166
Thanks: 19
Thanked 11 Times in 11 Posts
Default

ok.. it's on my todo list now
Reply With Quote
  #4  
Old 31st August 2010, 12:34
ispcomm ispcomm is offline
Senior Member
 
Join Date: Aug 2010
Posts: 166
Thanks: 19
Thanked 11 Times in 11 Posts
Default Implemented: patch attached

Quote:
Originally Posted by till View Post
I dont see any bigger problems to use such a path with the current ispconfig system. To implement that, you would basically do the following steps.

1) Define a new placeholder for the "f/fi/fir/first-domain.com" path of the domain path that you can use in the paths and symlinks in the server config settings. e.g. name it [website_splitted_domain]
2) Implement the logic to replace the placeholder when the website path is set in the interface/web/sites/web_domain_edit.php file and the interface/lib/plugins/sites_web_domain_plugin.inc.php file.
3)
I did some coding this morning and implemented the "splitter" placeholder. The placeholders are 4 and are called [website_domain_1], [website_domain_2], [website_domain_3] and [website_domain_4].

They split the path in t/testsite.com trough t/te/tes/test/testsite.com.
4 levels should be enough for anybody (uhm... have a dejavu feeling here).

I discovered that there's a module interface/lib/plugins/sites_web_domain_plugin.inc.php which basically overrides all settings (but duplicates them nonetheleast). I have implemented the patch there too, duplicating some code. I added TODO notes as I don't understand (yet) the need for that module.

The code is tested in my installation and works fine. Apache is also correctly configured so I guess it's OK.

Please advice me when/if this is included in the trunk (patch is agains latest trunk as of today).

Code:
# svn diff
Index: interface/lib/plugins/sites_web_domain_plugin.inc.php
===================================================================
--- interface/lib/plugins/sites_web_domain_plugin.inc.php	(revision 1947)
+++ interface/lib/plugins/sites_web_domain_plugin.inc.php	(working copy)
@@ -10,6 +10,22 @@
 	var $plugin_name        = 'sites_web_domain_plugin';
 	var $class_name         = 'sites_web_domain_plugin';
 
+	// TODO: This function is a duplicate from the one in interface/web/sites/web_domain_edit.php
+	//       There should be a single "token replacement" function to be called from modules and
+	//	 from the main code.
+        // Returna a "w/we/web/website.ext" path from "website.ext" domain name
+        function domain_name_split($domain_name,$levels) {
+                $domain = strtok( $domain_name, "." ) ;         // get first part only
+                $path = "";
+                for ( $i = 1 ; $i <= $levels ; $i ++ ) {
+                        $path .= substr( $domain, 0, $i ) . "/"  ;
+                }
+                $path .= $domain_name ;
+                return $path ;
+        }
+
+
+
     /*
             This function is called when the plugin is loaded
     */
@@ -38,7 +54,14 @@
         // Get configuration for the web system
         $app->uses("getconf");        
         $web_config = $app->getconf->get_server_config(intval($page_form->dataRecord['server_id']),'web');            
+	// TODO: This code is a duplicate from interface/web/sites/web_site_edit.php (there should be only 1).
         $document_root = str_replace("[website_id]",$page_form->id,$web_config["website_path"]);
+print_r($web_config);
+        $document_root = str_replace("[website_domain_1]",$this->domain_name_split($page_form->dataRecord['domain'],1),$document_root);
+        $document_root = str_replace("[website_domain_2]",$this->domain_name_split($page_form->dataRecord['domain'],2),$document_root);
+        $document_root = str_replace("[website_domain_3]",$this->domain_name_split($page_form->dataRecord['domain'],3),$document_root);
+        $document_root = str_replace("[website_domain_4]",$this->domain_name_split($page_form->dataRecord['domain'],4),$document_root);
+
         // get the ID of the client
         if($_SESSION["s"]["user"]["typ"] != 'admin' && !$app->auth->has_clients($_SESSION['s']['user']['userid'])) {                    
             $client_group_id = $_SESSION["s"]["user"]["default_group"];
@@ -53,11 +76,16 @@
         // Set the values for document_root, system_user and system_group
         $system_user 				= $app->db->quote('web'.$page_form->id);
         $system_group 				= $app->db->quote('client'.$client_id);
+	// TODO: Isn't this a duplication of the code above???
         $document_root 				= $app->db->quote(str_replace("[client_id]",$client_id,$document_root));
+        $document_root = str_replace("[website_domain_1]",$this->domain_name_split($page_form->dataRecord['domain'],1),$document_root);
+        $document_root = str_replace("[website_domain_2]",$this->domain_name_split($page_form->dataRecord['domain'],2),$document_root);
+        $document_root = str_replace("[website_domain_3]",$this->domain_name_split($page_form->dataRecord['domain'],3),$document_root);
+        $document_root = str_replace("[website_domain_4]",$this->domain_name_split($page_form->dataRecord['domain'],4),$document_root);
         $php_open_basedir 			= str_replace("[website_path]",$document_root,$web_config["php_open_basedir"]);
         $php_open_basedir 			= $app->db->quote(str_replace("[website_domain]",$page_form->dataRecord['domain'],$php_open_basedir));
         $htaccess_allow_override 	= $app->db->quote($web_config["htaccess_allow_override"]);
         $sql = "UPDATE web_domain SET system_user = '$system_user', system_group = '$system_group', document_root = '$document_root', allow_override = '$htaccess_allow_override', php_open_basedir = '$php_open_basedir'  WHERE domain_id = ".$page_form->id;
 		$app->db->query($sql);
-	}
-}              	
\ No newline at end of file
+	}	
+}              	
Index: interface/web/sites/web_domain_edit.php
===================================================================
--- interface/web/sites/web_domain_edit.php	(revision 1947)
+++ interface/web/sites/web_domain_edit.php	(working copy)
@@ -251,6 +251,17 @@
 		parent::onShowEnd();
 	}
 
+	// Returna a "w/we/web/website.ext" path from "website.ext" domain name
+	function domain_name_split($domain_name,$levels) {
+		$domain = strtok( $domain_name, "." ) ;		// get first part only
+		$path = "";
+		for ( $i = 1 ; $i <= $levels ; $i ++ ) {
+			$path .= substr( $domain, 0, $i ) . "/"  ;
+		}
+		$path .= $domain_name ; 
+		return $path ;
+	}
+
 	function onSubmit() {
 		global $app, $conf;
 
@@ -345,6 +356,10 @@
 		$web_rec = $app->tform->getDataRecord($this->id);
 		$web_config = $app->getconf->get_server_config(intval($web_rec["server_id"]),'web');
 		$document_root = str_replace("[website_id]",$this->id,$web_config["website_path"]);
+		$document_root = str_replace("[website_domain_1]",$this->domain_name_split($web_rec['domain'],1),$document_root);
+		$document_root = str_replace("[website_domain_2]",$this->domain_name_split($web_rec['domain'],2),$document_root);
+		$document_root = str_replace("[website_domain_3]",$this->domain_name_split($web_rec['domain'],3),$document_root);
+		$document_root = str_replace("[website_domain_4]",$this->domain_name_split($web_rec['domain'],4),$document_root);
 
 		// get the ID of the client
 		if($_SESSION["s"]["user"]["typ"] != 'admin' && !$app->auth->has_clients($_SESSION['s']['user']['userid'])) {
@@ -426,6 +441,10 @@
 		$web_rec = $app->tform->getDataRecord($this->id);
 		$web_config = $app->getconf->get_server_config(intval($web_rec["server_id"]),'web');
 		$document_root = str_replace("[website_id]",$this->id,$web_config["website_path"]);
+		$document_root = str_replace("[website_domain_1]",$this->domain_name_split($web_rec['domain'],1),$document_root);
+		$document_root = str_replace("[website_domain_2]",$this->domain_name_split($web_rec['domain'],2),$document_root);
+		$document_root = str_replace("[website_domain_3]",$this->domain_name_split($web_rec['domain'],3),$document_root);
+		$document_root = str_replace("[website_domain_4]",$this->domain_name_split($web_rec['domain'],4),$document_root);
 
 		// get the ID of the client
 		if($_SESSION["s"]["user"]["typ"] != 'admin' && !$app->auth->has_clients($_SESSION['s']['user']['userid'])) {
@@ -516,4 +535,4 @@
 $page = new page_action;
 $page->onLoad();
 
-?>
\ No newline at end of file
+?>
ispcomm
Reply With Quote
  #5  
Old 31st August 2010, 12:48
till till is offline
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 36,421
Thanks: 834
Thanked 5,501 Times in 4,330 Posts
Default

Thanks for the patch. Have you tested the renaming of a domain as this is the only critical part that I see at the moment? E.g. when you rename the domain of a website from e.g. test.int to johndoe.org.

Quote:
I discovered that there's a module interface/lib/plugins/sites_web_domain_plugin.inc.php which basically overrides all settings (but duplicates them nonetheleast). I have implemented the patch there too, duplicating some code. I added TODO notes as I don't understand (yet) the need for that module.
This new module is needed for the remoting API and it also replaces the code in interface/web/sites/web_domain_edit.php, so the code in the web_domain_edit.php file has to be removed if the plugin is working.
__________________
Till Brehm
--
Get ISPConfig support and the ISPConfig 3 manual from ispconfig.org.

Last edited by till; 31st August 2010 at 12:53.
Reply With Quote
  #6  
Old 31st August 2010, 13:13
ispcomm ispcomm is offline
Senior Member
 
Join Date: Aug 2010
Posts: 166
Thanks: 19
Thanked 11 Times in 11 Posts
Default

Quote:
Originally Posted by till View Post
Thanks for the patch. Have you tested the renaming of a domain as this is the only critical part that I see at the moment? E.g. when you rename the domain of a website from e.g. test.int to johndoe.org.
I was not aware of this point. I tried to change the domain name from sites->edit then insert a new name in the sitename.ext.... it gets changed in the domain field of the web_domain table but the path stays the same (old path).

the apache vhost file is updated with a new "virtualhost" directive but sitting on the old path.

I cannot find where the new path should be calculated. Perhaps a little hint will save me lots of time in searches.

ADDED: Hmm... apache vhost is somewhat bad. Created with wrong document root:
Reloading web server config: apache2Warning: DocumentRoot [/var/www/newsite.com/web] does not exist

Quote:
This new module is needed for the remoting API and it also replaces the code in interface/web/sites/web_domain_edit.php, so the code in the web_domain_edit.php file has to be removed if the plugin is working.
I discovered by debugging that both places are called. First is web_domain_edit.php and then the plugin.

ispcomm.

Last edited by ispcomm; 31st August 2010 at 13:17.
Reply With Quote
  #7  
Old 31st August 2010, 13:17
till till is offline
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 36,421
Thanks: 834
Thanked 5,501 Times in 4,330 Posts
Default

The web_domain_edit.php contains a function for onAfterInsert and one for OnAfterUpdate. Have you added your code in both functions or only in the onAfterInsert part? There is no other place where the path gets changed plus in the plugin.

Quote:
I discovered by debugging that both places are called. First is web_domain_edit.php and then the plugin.
Thats how the plugins are intended to work an why the code in the dit file gets removed.
__________________
Till Brehm
--
Get ISPConfig support and the ISPConfig 3 manual from ispconfig.org.
Reply With Quote
  #8  
Old 31st August 2010, 13:35
ispcomm ispcomm is offline
Senior Member
 
Join Date: Aug 2010
Posts: 166
Thanks: 19
Thanked 11 Times in 11 Posts
Default

Quote:
Originally Posted by till View Post
The web_domain_edit.php contains a function for onAfterInsert and one for OnAfterUpdate. Have you added your code in both functions or only in the onAfterInsert part? There is no other place where the path gets changed plus in the plugin.

Thats how the plugins are intended to work an why the code in the dit file gets removed.
Yes, I have inserted the code in both web_domain_edit and in the plugin.

It also seems that web_document_root and web_document_root_www are never read from the database but generated from the domain name. If this is a case, I might have found a bug.

I'm doing a little more of debugging right now (but I'm using echo/print statements and it's taking time).

ispcomm.
Reply With Quote
  #9  
Old 31st August 2010, 13:45
till till is offline
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 36,421
Thanks: 834
Thanked 5,501 Times in 4,330 Posts
Default

Quote:
It also seems that web_document_root and web_document_root_www are never read from the database but generated from the domain name. If this is a case, I might have found a bug.
This might be the case, even if it is not a bug as it matches the current ispconfig implementation of paths. But it might have to be extended then if other path identifiers are available in future.

Please be aware that there is on "real" path and one symlink (for easier shell navigation for the admin). On current setups, /var/www/domain.com is a symlink and /var/www/clients/client1/web1/ is the real path which uses ID's to ensure that it never changes on a domain name update, otherwise it would break cms systems and scripts that are installed in the website.
__________________
Till Brehm
--
Get ISPConfig support and the ISPConfig 3 manual from ispconfig.org.
Reply With Quote
  #10  
Old 31st August 2010, 13:50
ispcomm ispcomm is offline
Senior Member
 
Join Date: Aug 2010
Posts: 166
Thanks: 19
Thanked 11 Times in 11 Posts
 
Default

some more information. The plugin is never called on domain edit (rename). It's only called on creation time.

The onAfterUpdate is called, but the flow is not really clear to me. It seems that it never reaches any of the sql update statements. The domain I'm adding is binded to a reseller and not an end client (shall this matter?).

Document root is calculated correctly in the upper side of the function, but never updated in the database.

My guess is that this has nothing to do with the patch....

Could you have a look on this issue?

ispcomm.
Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
I'm attack brute force qb7 General 6 21st July 2012 22:34
Change domain of existing web? wpwood3 Installation/Configuration 3 11th November 2008 21:03
change web-ownership to different customer? brengo General 2 18th March 2007 16:58
Mail using postfix receive but cannot send garfabian Installation/Configuration 17 2nd September 2006 14:55


All times are GMT +2. The time now is 11:00.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.