Single web site migration tool

Discussion in 'Plugins/Modules/Addons' started by ispcomm, Aug 2, 2017.

  1. ispcomm

    ispcomm Member

    I would like to open a new thread for what I think is a much needed addition to ispconfig, merely a single web site migration tool between servers. I've wanted this for a few years now and I guess I'm not alone.
    There has been some discussion on the subject, but currenly there's no evident progress.
    I think it will be good to have a dedicated thread where we can express our desire and offer help and money for getting this tool done.
    If you need this add your +1 and comments are welcome. I think Till is already working on this thing, but perhaps it's at the bottom of his todo.
    I'll summarize a few comments from the other thread so get us started:
    At this point I must say I'm getting "desperate" too for a tool to let me move a single site from one server to another in a multiserver setup.

    I'm having issues with very old debian setups that will not upgrade cleanly to recent versions, and it's necessary to move single sites from old servers to different new servers.

    It's been now over six months I've been delaying this work. Is there any hope for the next few weeks ?

    ispcomm.
     
    webguyz likes this.
  2. Jesse Norell

    Jesse Norell Well-Known Member

    Just a thought regarding ssh login from master to all slaves, I'd certainly not want to have direct root access from master to all slave servers; for websites you could have the source server create a temporary ssh account under the affected website which the destination server uses to transfer data - that wouldn't always work for the database though, as dedicated database servers don't setup client ssh accounts.

    I supposed for database servers you just have the destination server connect directly to mysql on the source server and pull the data in.

    For moving websites specifically you could even leverage http(s) to perform the data transfer - the source server creates a tarball, sends "now avaliable" status and the link to the master, which propogates to the destination node, which downloads the tarball from that link. That isn't an option for dedicated mail servers though (which isn't a requirement in this thread, but might be in the overall project), so maybe considering ssh first is better because you could reuse the "tool".
     
  3. ahrasis

    ahrasis Active Member

    Using the old method for that is just like @Jesse Norell said above (sinceI am not sure how @till gonna do it and when):
    1. Safely ssh into your new web server.
    2. Then from it, ssh to your old server and get into the old web folder via "cd /oldserver/webfolder".
    3. Afterwards, run database backup via "mysqldump -u username -p database > db_backup.sql".
    4. Further archive the web folder via: "tar -czvf backup.tar.gz /oldserver/webfolder --exclude=/exclude/path1/ --exclude=/exclude/path2/".
    5. Exit back to your new web server and copy the backup files via "scp [email protected]:/oldserver/webfolder/backup.tar.gz /newserver/webfolder/".
    6. Extract the backup file in the new web folder via "tar -xzvf backup.tar.gz".
    7. Get into mysql shell via "mysql -u username -p" and create new database via "CREATE DATABASE database;" then "exit".
    8. Finally run "mysql -u username -p database < db_backup.sql".

    Do correct me if I am wrong.
     
    chico11mbit likes this.
  4. ispcomm

    ispcomm Member

    @till seems to have a beta in the next couple of week.
    the issue with your approach is that it's too manual. you might have more than 1 db per site, different users, their passwords are unknown etc. Also, in a more typical scenario, you'll be moving a site from one server to another server in the same multiserver setup. You need to update the ispconfig database to point the "records" of the site on the new server. And to do this, you must also account from the way ispconfig propagates changes between servers (i.e. incremental records in the central db, for which the slaves look in the cron job).
    It's better done properly or not at all :)
    Good thing is @till seems to be working on this seriously now.
     
  5. webguyz

    webguyz Active Member HowtoForge Supporter

    The more gui based the better. I can see a web based tool where everything is chosen from drop down boxes, current website, which server to move to and maybe option boxes for moving db as well and the migration tool would move the website over to new host and remove it from the old server. No reason to make it cmdline based when you have a rich api such as IPConfig has.
     
  6. ahrasis

    ahrasis Active Member

    Very much true, especially if its a GUI that is needed. And if @till is not too busy, then we may see this module / plugin coming out soon.
     
  7. webguyz

    webguyz Active Member HowtoForge Supporter

    A couple of weeks ago Till said a beta was coming in 2 weeks so hopefully real soon now.
     
  8. till

    till Super Moderator Staff Member ISPConfig Developer

    You are referring to a wrong thread here, the tool I talked about in the other thread and which is nearly finished is a software to move an ISPConfig server within a multiserver setup (moving a slave to a new hardware which is connected to a master) by moving an exact copy (in difference to importing things with the remote API as done by the migration tool which requires it to have a new master). A tool to migrate a single site is something completely different, especially as it will be an ispconfig UI plugin and not a standalone software) as it requires another approach and I will be working on that next.
     
    ahrasis likes this.
  9. webguyz

    webguyz Active Member HowtoForge Supporter

    There is a need for a server move tool within a multiserver set up as your working on. When I moved to vmware I used the original migration tool and had to deal with dns issues etc, but the tool your talking about would have made the changeover so much easier.
     
    till likes this.
  10. ahrasis

    ahrasis Active Member

    Actually I was confused as well at first, but what @till confirmed, cleared it. So basically what is being done and almost finished is a software to move server in a multiserver setup and the tool for moving a single will be worked on next. Thanks for the info.
     
    till likes this.
  11. webguyz

    webguyz Active Member HowtoForge Supporter

    Just in time! I have a need for the server move tool within a multiserver setup. Any idea when it will be released?
    I have a server that everytime I try to do a "aptitude upgrade" it wipes out some of my customers databases and would like to move to a new server.

    Thanks!
     
  12. till

    till Super Moderator Staff Member ISPConfig Developer

  13. ispcomm

    ispcomm Member

    I've written a tool to help with moving sites from one server to another, on the same ispconfig multiserver setup. It can be used also to move sites between different ispconfig installations too.

    The tool doubles as archival (backup/restore) for a single site and all related settings and data, including
    • databases and database users
    • ftp users
    • protected areas and their users
    • cron jobs
    • everything else
    This is a cli-only tool and must be run from the shell (linux and cygwin under windows).
    As a bonus, it includes some bulk manipulation:
    • bulk dns records update: name, data (ip) on multiple sites at the same time
    • activator: enable/disable one or more sites at a time.
    • delete: delete all data for a site and related accounts, crons, ftp users etc.
    The tool is in internal production, and I used it to move a few hundred sites from old servers to new servers in the same ispconfig multiserver setup.
    All path to the sites are preserved, so no modification is necessary for the files of the site (like config paths, users etc).

    If there's enough demand for this, I could polish and document it and release it. It will require some additional work tough.

    If you're interested, PM me
    ispcomm
     
    Last edited: Sep 26, 2017
  14. vk3heg

    vk3heg Member

    ispcomm I like the sound of this tool.

    I've been developing a new website on a backup of my server, and would love an easy to use tool to move the dev website to the real one.
     

Share This Page