ISPConfig and Drush/Git

[rdan]

I posted this question in the installation forum. It is probably better posted here. Drupal is a nice community-based website program, but it is difficult to keep things up-to-date when you have a number of modules involved. It is also a bit difficult to update the drupal version. Drush/Git was created to solve the installation and updating procedures with drupal.

ISPConfig takes care of pretty well everything when setting up a new website/database and I am wondering whether it could be integrated with drupal. My question is whether it is possible to integrate the command line Drush program shell commands with ISPConfig3. I've come upon one project dealing with this objective (WALID - http://drupal.org/sandbox/defconjuan/1157038) but could not find any link to similar projects in howtoforge.net. Thanks.

[defconjuan]

Originally Posted by rdan

Drupal is a nice community-based website program, but it is difficult to keep things up-to-date when you have a number of modules involved. It is also a bit difficult to update the drupal version. Drush/Git was created to solve the installation and updating procedures with drupal.

Agree. Drupal's strong and gaining popularity. The application/integration of Drush/Git, as well as the contributions of the QuickStart project really beef up the development and automation side of things.

Originally Posted by rdan

ISPConfig takes care of pretty well everything when setting up a new website/database and I am wondering whether it could be integrated with drupal.

I agree. ISPConfig's a great open source control panel. Of course, it's missing a lot of the features of something like Webmin, but it's not designed to be a Webmin replacement. So as much as possible, I use ISPConfig and only use Webmin for higher-level sysadmin type things. In doing so, they co-exist nicely.

Originally Posted by rdan

My question is whether it is possible to integrate the command line Drush program shell commands with ISPConfig3. I've come upon one project dealing with this objective (WALID - http://drupal.org/sandbox/defconjuan/1157038) but could not find any link to similar projects in howtoforge.net. Thanks.

Thanks rdan, WALID's my project. ISPConfig3 plays nicely with Drush, git, and even Webmin and AEGIR so long as you make configuraton changes -- lots of them -- to integrate these mostly disparate productions.

But techincally, you shouldn't need Drush on a production box, only on your dev and staging boxes.

Why's this relevant? Well, if you start using both Drush and ISPConfig to create websites, you're going to have nothing but problems.

For example: If you must (want to) have Drush installed on a production box, every time you'd want to create a new Drupal site, you'd have to be conscious to ALWAYS first create the website and database in ISPConfig; then you'd have to create an ssh user; and only then would/should you start using Drush to manually build out a site using a make file. (Emphasis on manual)

If you don't do a manual Drush build out, you'd end up screwing up your hosts and ISPConfig generated vhost file. -- See what I mean? A pain - and you have to really know all tools involved and what they do each time you execute administrative commands.

Aside from several problems/issues like this, you shouldn't need Drush on a production box because best practices would have one developing on a development box where Drush is used during development, and to publish/sync the site to the production server where Drush wouldn' t be needed.

Sorry, I'm just prepping you for the things you're going to come accross when you start combining these tools. That's baically why I created WALID. I did what it sounds like you're about to embark up and decided that I should crystalize my ending result into a build script I can use to create slightly-hardened production/staging servers that harmonize ISPConfig, Drush, QuickStart, AEGIR, and Webmin (including live DNS and Email services) on a box with a fresh ubuntu 10.04 LTS server image.

So it really depends on what you're trying to accomplish. Are you needing a dev box or a production box? If you're needing a dev box, just use QuickStart, where you won't need ISPConfig. If you're building a production box, you can use something like WALID.

WALID's a a slightly-hardened, firewall-enabled, production server configured with:
Ubuntu Linux 10.04.2 LTS server
Apache/2.2.14 (Ubuntu)
PHP 5.3.2-1 ubuntu4.9
MySQL: 5.1.41-3 ubuntu12.10
PHPMyAdmin 3.3.2 deb1
AEGIR 1.1
Drush 4.4
ISPConfig 3.0.3.3
Webmin 1.550
SquirrelMail 1.4.20
All Drupal dependencies (upload progress, rewrite, php/mysql tweaks, etc.)

WALID also maintains (and will always maintain) parrallel architectures for both QuickStart (thanks much to Michael Cole) and the ISPConfig3 ubuntu 10.04 perfect server tutorial and follow-ups on this website (thanks to the ISPConfig and HowtoForge teams). I did this so that people can get up and going quickly with WALID by following the tons of wonderful QuickStart, ISPConfig, Drush, Drupal, and AEGIR tutorials out there.

Testing on rc3 is going well and within the next few days, I'll be releasing a release candidate for public consumption. I'll release it as a build script and VirtualBox virtual appliance.

Your use and feedback will be welcome.

[rdan]

Hello defconjuan,
Thanks for your very detailed and informative post. I've spent the last week or so attempting to learn a bit about the set of utilities you have integrated into WALID.

I am attempting to set up 4-5 production drupal systems. As you might have guessed, I have ISPConfig3 already installed and am working on one of the drupal systems. It's more or less a developmental web since I'm trying to make all of my mistakes on that web instead of more important ones to follow (e.g. ubercart).

The problem is that as I understand it, core drupal version 7 modules can only be upgraded manually. I did that in going from 7.2 to 7.4. It's not impossible, but definitely a pain. It becomes exponentially more of a burden when there are a number of extra modules involved. My last experience with drupal was with version 4.7 with gallery 2 installed as an outside module. The dependencies were so problematic that I closed down the system. That's why drush/git looks so good. From what I can understand, it is easy to re-make the module or the system without getting into trouble with dependencies.

ISPConfig3 has a pretty good secure ftp system, but it doesn't seem to be possible to use the module install/update capabilities of drupal 7 to handle modules. I keep getting in trouble with the error message involved with use of normal ftp in the update procedure. That means every module install and upgrade has to be done manually, using the pure-ftpd client for the website and then installing it manually. Of course that means also that the module upgrades will have to be manual too. That seems to take me right back to drupal 4.7.

Since my server is up already and is running with ISPConfig3, I don't really want to start again. However I could set up another machine to use as a developmental machine which could have the same number of drupal websites as are going to be on the production server. My production OS is Debian Squeeze but Ubuntu 10.04 is essentially the same thing, so it should not matter to drupal. As I understand your notes, the dev machine could be used to keep each system up-to-date and then simply copied over to the production server. That way, no ownerships issues should arise in ISPConfig and sites should be able to be updated regularly on the developmental machine.

I would certainly like to try out your system if it can do what I have described above. If it can't, and ISPConfig3 is not really suited for running drupal, I suppose I could reinstall my entire system using the combination of quickstart, Webmin, AEGIR, drush and git. This reminds me of the old carpenter's rule: If you've done it wrong the first time, there's always time to do a job twice.

[defconjuan]

The problem is that as I understand it, core drupal version 7 modules can only be upgraded manually. [...] My last experience with drupal was with version 4.7 [...] The dependencies were so problematic that I closed down the system.

You can update Drupal core or any module using Drush (or FTP and the Drupal admin interface). Once Drush is installed and configured properly on the development/staging box, you can run a simple one line command to update Drupal core, to execute the update scripts (update the database), or to update everything.

ISPConfig3 [...] doesn't seem to [handle Drupal] modules. [...] That means every module install and upgrade has to be done manually [...] That seems to take me right back to drupal 4.7.

You can only update Drupal modules/core using FTP or Drush over SSH. (I suppose you could write a Drush script that you execute with Cron but I wouldn't recommend it.) Even if you use something like Plesk, Plesk won't upgrade the modules you added after installing Drupal using the Plesk interface. At best, you'll be able to update the core. Without sounding critical, I think you're confusing the role of ISPConfig vis-a-vis what it's meant to manage and what it's not. And if your last experience with Drupal was 4.7, get ready for a ride... a lot's changed and you're going to have some growing pains.

My production OS is Debian Squeeze but Ubuntu 10.04 is essentially the same thing, so it should not matter to drupal.

You're more or less right. It's more about your Apache and PHP configuration than the distro or version.

As I understand your notes, the dev machine could be used to keep each system up-to-date and then simply copied over to the production server. That way, no ownerships issues should arise in ISPConfig and sites should be able to be updated regularly on the developmental machine.

Not really, let me clarify: You develop on a dev box that has Drush. When you're ready to update your production site, you issue a Drush sync command from the Dev box and it will sync to your remote/production server. Wether ownership is an issue or not depends on how your Apache and PHP are setup (and ISPConfig, if it's installed).

For example: Your production box's Apache may be configured to run as the user www-data (common); or as your ftp/shell user (common when you use SuExec). Now let's say you SSH (or FTP) an update up to your production site. You may have ownership problems because the files you just uploaded are written with your SSH (or FTP) user, but Apache maybe running as another user (i.e. the www-data user). This actually isn't an issue uless you start downloading/uploading files to your siteroot/sites/sitename/files folder (which Drupal needs read/write permissions to) or if a module you've added needs special permissions to files outside of that folder.

There are so many configuration combinations for the way Apache and PHP can be running on your production box that you'll always have to be aware/familiar with how your dev box does things as oppossed to how your production box does them. As well as what happens when you publish because you may have to reset permissions on certain files or folders each time you publish core or module updates to your production box. (See: http://groups.drupal.org/node/135269 for a posting and a script on this.)

[rdan]

Well, I definitely see your point about the drupal learning curve between version 4.7 and 7. I've spent the past week just getting awstats visible again.

I have managed to install modules using the drupal interface so that has solved one of the larger problems I mentioned in my first post. It remains to be seen whether I can use the interface to upgrade existing contributed modules. If so, I would only need to go back to ssh and ftp for core upgrades.

Since my websites will all be a bit different from each other, I think I will start off installing by hand. They will all be running inside of ispconfig3 so a lot of the work will be taken by ispconfig. I won't have a new development server available for a month or two anyway. When it is running I will re-join your project.

Once again, thanks very much for your timely, detailed and helpful information.