How to migrate a website area from a server to another

Discussion in 'General' started by mapreri, Oct 27, 2021.

  1. mapreri

    mapreri New Member

    Hi!

    I'm basically reviving this thread here: https://www.howtoforge.com/community/threads/single-web-site-migration-tool.77022/

    As I seem unable to find more information regarding operations such a moving a website from one host to another with ispconfig, possibly carrying over the data together with the configuration.

    From what I could see the only possibilities that I could come up (that I haven't yet actually tried) involve at least:
    1. creating a new db on the new host, same user/db name
    2. manually creating user/group like it had on the previous host
    3. copying over the files with the same structure it had on the previous host, preserving user/group
    4. messing with the ispconfig master db to move the website to the new host
    5. do something to trigger ispconfig to rebuild its things on the new host (?)
    6. change the FTP settings to switch to the new host
    Am I looking at the problem from the right angle, or am I missing something huge somewhere?

    I saw @ispcomm mention on that thread he wrote their own tool to accomplish this job, so I suspect that indeed ispconfig doesn't have such tool built-in and I'd need to create something myself (if I can't find anything already available around, I tried PMing @ispcomm a couple of weeks ago but I haven't got an answer yet) to do the above steps?

    Thank you for any insight anybody could give me! :)
     
  2. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    Are you wanting to move a site between servers in the same multi-server installation, or separate ispconfig installs?
     
  3. mapreri

    mapreri New Member

    within the same multi-server installation.

    And, amongst the thing that I'd like to see preserved is the full path to the website, since some CMSs embed the path in their configuration and I really don't want to mess with customers' websites for this. (sorry, forgot to mention in the OP)
     
  4. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    Some thoughts, if you use php-fpm in chroot mode the path to the website will always be seen as /web which would would remove that issue, though if you're keeping the same web# and client#, the paths shouldn't change. I've never tried to do this, but in general what you'd need to do is update all relevant db records in the the master ispconfig database (in web_domain, web_database, shell_user, etc., etc....) to change the old server_id to the new one, then resync everything (or at least everything involved, eg. websites, shell users, databases, etc.) - that will create things on the new server. Then copy your data over (website files and database(s)). Then you'll need to remove the db records and data from the old server, which you could do manually or probably leverage sys_datalog to do that (ie. create sys_datalog entries for the old server_id to delete those). I would create a test setup where you can try, break, debug, fix, try again until the process is working and complete. You definitely want to back up all your site files/database, and make a backup of all ispconfig databases and /etc/ on both web servers before starting the live move; having a vps snapshot you could roll back to in case of problems would probably be handy.

    One other tthing you could test to move the data is to use the website and database backup feature. Eg. setup a shared backup location (network filesystem/mount), then create a backup of the website and database(s) on the old server prior to updating db tables and resyncing - then after resyncing, when the site and database is created on the new server, restore the backup you made. It may or may not be easier than just using mysqldump and rsync to move things.

    Note when the site is created on the new server, I think the actual uid/gid numbers can/will change between the old and new server, but the user and group names will remain the same, so just set ownership based on the names rather than preserving uid/gid.

    Good luck, and if you come up with a useful and tested tool to do this, I'm sure other folks would find it useful if you wanted to post it.
     
  5. florian030

    florian030 ISPConfig Developer ISPConfig Developer

    create the website on the new server
    create a new database on the new server with the same db-user
    ex- and import the database
    use rsync to transfer the site (use a shell-user)
    adjust your website-config
     
    atle likes this.
  6. florian030

    florian030 ISPConfig Developer ISPConfig Developer

    are you sure that this will work? the restore does not create a database and the database-id is "linked" to the backup.
     
  7. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    Not at all, I'm just tossing out things to consider/try, I've not tried to make such a site move myself. (Fwiw, I would just do everything manually, and only look at scripting the move if it were something that needed done on a recurring basis.)
     
  8. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    i've done it before, moving a number of sites from one member server of a multiserver setup to another member server.
    @Jesse Norell has got it right.
    in mysql, using cli or phpmyadmin, change the server_id value for the site/user/db in (depending on what ftp/db/ssh accounts exist) ftp_user, shell_user, web_backup, web_database, web_domain, web_folder, web_folder_user, and webdav_user (assuming webdav users, or protected folders/users are configured for those last 3 columns, and assuming the db is on the same server as the website.)
    if mail is on the same server as well, then the same change is needed for the various mail and spamfilter tables if your moving that service as well.

    resync the relevant services for the server the site is being moved to.

    everything required should be created on the target server, then it's just a case of copying over the db and website files from the source server using whatever method you find easiest.
    at this point, i would suggest creating an entry for the domain in your local hosts file, pointing to the new server, and test everythings ok.
    once happy, update the domains dns to point the necessary records to the new hosting servers ip.

    i would advise not making any changes to the website itself (or inform the site owner not to), for at least 72 hours, leaving the site still running on the old server as well, in case visitors are still getting the old ip's, after this time, you can start removing the site/database from the old server.

    then it's just a case of cleaning up those records in the dbispconfig database on the old hosting server (assuming it's not also the master)
    can be done by manually removing them, or as @Jesse Norell suggests, using the sys_datalog table (never done it using this method, so suggest being very careful what you put in this table). i do know that just running a resync of those services on the old hosting server does not remove them from the database, or at least it didn't for me, but that was back around ispconfig 3.0.4....


    *you may also have to play around a bit with copying over letsencrypt certs / config or briefly delete letsencrypt and then re-apply it after changing the dns records. letsencrypt wasn't an issue/option when i last moved sites this way.
     
  9. florian030

    florian030 ISPConfig Developer ISPConfig Developer

    I think, this depends on the amount of sites running on the destination-server. It's not very funny, when you resync 100+ websites.

    Using your way (changing the server_id) you have to remove everything on the old server by hand.

    I'm pretty sure, that you can not sys_datalog when there are no records (anymore) in the database. A resync will just resync the configs and will never deleted any records.

    Personally, i see no problem with creating the website on the new server (you can use the same name) and a new database (you need a different name but you can use the same db-user.
    Most CMS do not care about the full-path (iirc, joomla does) so can use sed to replace the db-user in the cms-config (and maybe the directory). I use this to automaticly import web-sites from a remote server. You must the directory with joomla and nextcloud. For magento, shopware5, shopware6 (maybe adjust COMPOSER_HOME) and wordpress it's enough to update the db-connection.

    This way allows you, to keep both sites for a longer time. I you are find with new server, just change the dns. You can revert back at time and this leads to a much cleaner setup when you delete the site and database from the old server.[/QUOTE][/QUOTE]
     
  10. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    you're right, it's not fun when you have to resync a lot of websites. if doing that though, i would also suggest turning off the config checks and restarts first. testing every vhost config, and then restarting apache after every vhost change *really* slows down the resync, and of course, a hundred plus apache restarts will inevitably lead to calls from site owners saying their site is down.


    joomla et al might not care about the filepath, and if you're lucky wordpress won't be either, but a lot of times wordpress is a problem. wordfence for one, puts the filepath into several files, loads of plugins put the full filepath in the database. i've moved sites where there's literally 100000+ full filepaths in the wordpress database that require changing. if moving the wordpress site from one ispconfig member server to another, at least with the changing server_id method, the full filepath never changes. the whole site can be moved without worrying about having to make any changes at all.
    if moving a wordpress site from some other provider / control panel, i would use the duplicator wordpress plugin by snap-creek. makes it very easy, and handles any filepath / domain-name/url changes automatically.
     
  11. florian030

    florian030 ISPConfig Developer ISPConfig Developer

    nope. see i.e. $log_path and $tmp_path

    from wordfence: "If you are moving your site to a new server we recommend that you first remove extended protection. When your site has been migrated, you will then optimize the Firewall again."
     
  12. mapreri

    mapreri New Member

    So, I gave it a shot with a test website. I didn't try yet to move the db, but that looks trivial in comparison.

    We use php-fastcgi, so that sadly is not our case :(

    So, in our setup we only have a couple of shell_user entries, and they are not affected by this move at the moment, one less thing to care about.
    I'm also postponing trying to move the db, which looks trivial so I'll try that later.

    Regarding the actual website: I tried a single `update web_domain set server_id = 16 where domain_id=2544;` and then made a silly change on the ispconfig web interface to trigger it (in this case, changed the PHP version; after all that was also invalidated by the change above).

    This did create the users using the same username and group, different UID/GID as you figured, but indeed I don't think (hope!) that's a problem. it also created the basic directory structure, however:
    • it didn't create the "client0" group and the `/var/www/clients/client0` directory (so there were errors such as `ln: failed to create symbolic link '/var/www/clients/client0/XXXXX': No such file or directory` in the ispconfig log; I did that manually
    • this being a branch new host, apache wasn't really fully configured (missed a couple of a2enmod I had to take care of)
    • more importantly: the permissions weren't fixed, despite the user/groups having been created correctly:

    Code:
    [email protected] ~www/clients/client1320/web2544 # la
    total 40
    drwxr-xr-x 10 root    root       4096 Oct 28 13:30 .
    drwxr-xr-x  3 root    root       4096 Oct 28 13:30 ..
    drwxr-xr-x  2 root    root       4096 Oct 28 13:30 cgi-bin
    drwxr-xr-x  2 root    root       4096 Oct 28 13:30 log
    drwx--x---  2 web2544 client1320 4096 Oct 28 13:30 private
    drwx------  2 web2544 client1320 4096 Oct 28 13:30 .ssh
    drwxr-xr-x  2 root    root       4096 Oct 28 13:30 ssl
    drwxr-xr-x  2 root    root       4096 Oct 28 13:30 tmp
    drwxr-xr-x  4 root    root       4096 Oct 28 13:30 web
    drwxr-xr-x  2 root    root       4096 Oct 28 13:30 webdav
    [email protected] ~www/clients/client1320/web2544 #
    
    Now, I suppose I can go ahead and chown it myself, however it leaves me a feeling any script I'll write to automate the change will be troublesome and full of double checks.

    Is there somewhere some documentation about how sys_datalog works (that doesn't involve reading through ispconfig's code :p)?
    I also see that there is no direct reference/access to sys_datalog in the documented APIs.

    probably :)
    though replicating this whole setup would be a huge pain, so actually right now I'm daring to use a single testing web_domain for this, mostly hoping that ispconfig won't go and mess with the other unrelated things while I play with this website... /o\

    This setup is technically a bunch of VMs all hosted in 3 physical servers in the same rack, so moving files around is trivial. TBH I was hoping to leverage rsync so that it would retain permissions/mtimes/etc, which is of course also possible by tarballing with the proper options but well.

    I also just realize that I need to verify how these websites are connecting to their databases... I hope nobody thought it was a good idea to use anything that is not "localhost" :\

    Since I'm doing this as a paid job I'll have to verify with the business if/how they'd be open to sharing whatever I create (which is likely to be a bunch of scripts to be run across origin server, target server, ispconfig master).
    But otherwise, I'd happy to dump such things in the open myself :)
     
  13. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    i don't think just making a change like the php version triggers the creation of everything that's needed.
    bits like the /var/www/clients/client0 folder, or the symlink creation may only be done by a new site creation, or a resync. just a rebuild of the vhost conf file or the php conf may not trigger the folder/symlink creation. similarly, some things like the file/folder permissions, may only be on new site creation / resync, or if 'set folder permission on update' is set in the server config.

    i think running an actual website resync on that server will successfully create everything whilst avoiding all the issues you've encountered by just making a config change to the website.
    once this parts done, you should be able to go ahead and rsync everything from the old server to the new server for that domain.
     
  14. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    The wp cli tool has a search and replace which makes these changes easy; you can also use it for changing embedded http urls to https when enabling encryption, changing www.domaim.tld urls to domain.tld, etc.
     
  15. mapreri

    mapreri New Member

    That would make sense indeed.

    I actually forgot about the existence of that "resync" page, thank you for reminding me of it. I just tried, but even that didn't create the client0 folder, nor fixed the directories owner/group of /var/www/clientXXX/siteYYY/* (I tried even after removing the /var/www/clientXXX directory and let the resync recreate it, which it did, everything except private/ and .ssh/ fully owned by root).
     
  16. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter


    yep. very nice tool.. used it for that many times. good for scripting wordpress installs from scratch as well. and for automating scheduled wordpress updates. (hate that wordpress are trying to have everything auto-update at random times. i don't want sites randomly breaking in the middle of the night because some silly little plugin decided to auto-update)
     
  17. mapreri

    mapreri New Member

    Right, but there are two problems with doing that:
    • touching customer's data more than absolutely necessary is *bad*
    • even if we knew that that particular CMS stores the path in that specific way, it would likely not cover other CMS, or customer-developed sites that do whatever they want
    So we can't just go and change things like that willy-nilly. However, I do plan on perhaps moving things to fpm or such in a future stage (or at least do some research and considerations on how that affects our setups) when this particular move is not urgent anymore.
     
  18. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    i would start thinking that maybe something isn't fully configured, or some parts misconfigured. if a file/folder isn't there, a resync should recreate it as if creating the site from new. might be worth turning up the debugging level on that server and running another resync, see if any problems are getting logged.
     
  19. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    Maybe resync more things as well, eg. clients.
     
  20. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    it would only be done like this, if it's known that such changes are required, eg moving a site around your own servers so it has different filepaths, or a site is migrated from hosting elsewhere, or changing domain names, or changing from http to https, everything except for the first issue would be known to, and should have been requested by the customer. so it wouldn't be willy-nilly, or more than absolutely necessary. and the issues described, and the use of wp-cli would be specific to wordpress sites anyway.

    also if you're planning on switching sites from fastcgi to fpm, you can just do that in the ispconfig interface. nothing needs to be done on wordpress sites for such a change.
     

Share This Page