API tools faq
paste
lyfsy

Fastest way to migrate cPanel to cPanel

lyfsy
Jan 23rd, 2020
98
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 21.01 KB | None | 0 0
raw download clone embed print report
  1. Fastest way to migrate cPanel to cPanel
  2. Whats the fastest way to migrate 1 cpanel account with more than 200GB in size?
  3. ++++++++++++++
  4. list of top cheapest host http://Listfreetop.pw
  5.  
  6. Top 200 best traffic exchange sites http://Listfreetop.pw
  7.  
  8. free link exchange sites list http://Listfreetop.pw
  9. list of top ptc sites
  10. list of top ptp sites
  11. Listfreetop.pw
  12. Listfreetop.pw
  13. +++++++++++++++
  14. Both has root access..
  15. Since you have root access on both servers,
  16.  
  17. 1) You will have to generate your account backup with --skiphomedir , With this options cPanel backup scripts will create your cPanel account backup without user home directory.
  18.  
  19. 2) Migrate account backup file to new server using rsync and restore that file on your new server.
  20.  
  21. 3) Once backup file is restored, Just rsync your user home directory from old server to new server with rsync.
  22.  
  23. 4) Take a database's dump from source server and then restore them in new server.
  24. TuxAndrew
  25. Linux - RedHat|cPANEL CentOS,Ubuntu |Azure-Vm/AWS-Ec2 Administrator,
  26. Assistance, Analysis and Diagnosis| Skype: tuxandrew
  27. [email protected]
  28. Since you have root access on both servers,
  29.  
  30. 1) You will have to generate your account backup with --skiphomedir , With this options cPanel backup scripts will create your cPanel account backup without user home directory.
  31.  
  32. 2) Migrate account backup file to new server using rsync and restore that file on your new server.
  33.  
  34. 3) Once backup file is restored, Just rsync your user home directory from old server to new server with rsync.
  35.  
  36. 4) Take a database's dump from source server and then restore them in new server.
  37. step 4 only would make sense if step 1 included --skipacctdb https://documentation.cpanel.net/dis...pkgacct+Script. Otherwise, step 1 would already include the account's database backups
  38. mega-uploading.com
  39. rootlayer.net
  40. worldwidete.com
  41. faucetgame.com
  42. load-tang.blogspot.com
  43. domain model
  44. sedigo.com
  45.  
  46. Exclude the account's databases from the cpmove archive.
  47. --skiphomedir
  48. Exclude the account's home directory from the archive.
  49. As to speed of backup/transfer it would depend on where that 200GB of data lies.
  50.  
  51. If it's a large MySQL databases, then above --skiphomedir only won't help as without --skipacctdb flag, step 1 will be backing up your large MySQL databases and it will be restored when you restore the cpanel pkgacct backup file so doing step 4 as well just doubles the time taken as you do database backup/restores twice - once in step 1 and a 2nd time in step 4. Adding --skipacctdb to step 1, will mean you can then do faster MySQL database backups/restores manually (step 4 above) via multi-threaded MySQL database backup tools like mydumper instead of single threaded mysqldump and then restore using multi-threaded mydumper's myloader command tool instead of single threaded native mysql client restores. With mydumper, databases are backed up into sql files on an individual table basis into a directory you tell it to save to. So you can eventually tar compress that directory for transfer too and use faster pigz multi-threaded gzip or facebook's zstd compression for the tar'd directory.
  52.  
  53. If it's large data files in /home/username/*, then --skiphomedir MAY help as you can have faster options to backup /home/username/* - rsync is one but would be slow if you have many smaller files as opposed to using tar compressed backups. You can use tar with faster compression algorithms of which cpanel has pigz support for multi-threaded gzip. Or you can install facebook's zstd compression for even faster compression and smaller compressed tar file but it may use more memory by default (you can tune to use less memory). Smaller compressed tar file may help if you're network bandwidth/uplinks between both servers is slower i.e. <500Mbit-1Gbps
  54.  
  55. Quick test I shared with my forum users a while back with mydumper 0.95 built against MariaDB 10.3.13 with GCC 8.3.1 compiler and 261MB sized dummy InnoDB MySQL database consisting of 20 tables each with 50000 rows.
  56.  
  57. Code:
  58. mydumper -V
  59. mydumper 0.9.5, built against MySQL 10.3.13
  60. On 2 cpu thread KVM VPS with Intel Xeon Silver 4114 @ 2.20Ghz
  61.  
  62. mydumper backup speed = 103.57 MB/s 6214.20 MB/min 364.11 GB/hr = 1.9x faster than mysqldump backup.
  63. myloader restore speed = 14.27 MB/s 856.20 MB/min 50.16 GB/hr = 2.03x faster native mysql restores
  64. mysqldump backup speed = 54.26 MB/s 3255.60 MB/min 190.75 GB/hr. An optimised mysql config backup speed = 61.26 MB/s 3675.60 MB/min 215.36 GB/hr which is 12.9% faster now.
  65. mysql native restore speed = 7.03 MB/s 421.80 MB/min 24.71 GB/hr
  66.  
  67. if your server hardware and mysql configuration is optimal, it should scale with more cpu threads allocated to mydumper/myloader
  68.  
  69. My mydumper vs mysqldump benchmark script I wrote.
  70.  
  71. mydumper backup
  72. Code:
  73. ./mydumperbench.sh mydumper
  74. ----------------------------------------------------------------------------------------
  75. mydumperbench.sh 0.0.9 by George Liu (eva2000)
  76. https://centminmod.com
  77. Sun Sep 22 05:15:02 UTC 2019
  78. ----------------------------------------------------------------------------------------
  79. ----------------------------------------------------------------------------------------
  80. [sbtest total size: 261 MB]
  81. [mydumper threads: 2 ] mydumper -t 2 -B sbtest
  82. [backup stats:] real: 2.52s user: 1.28s sys: 0.75s cpu: 80% maxmem: 8124 KB cswaits: 2178
  83. [backup speed:] 103.57 MB/s 6214.20 MB/min 364.11 GB/hr
  84. Successfully backed database using mydumper using 2 threads
  85. ----------------------------------------------------------------------------------------
  86. *************************************************************************************
  87. Sun Sep 22 05:16:07 UTC 2019
  88. results logged: /root/mydumperlogs/mydumperbench_220919-051501.txt
  89. Sep 21 22:24 mydumper_extstats_210919-222320.txt
  90. Sep 21 22:24 mydumper_mysqlreport_210919-222320.txt
  91. Sep 21 22:24 mydumperbench_210919-222320.txt
  92. Sep 21 22:24 mydumperstats_210919-222320.txt
  93. Sep 21 22:24 mydumpercsv_210919-222320.txt
  94. Sep 21 22:24 mydumper_mycnf_210919-222320.txt
  95. Sep 21 22:25 mydumper_extstats_210919-222445.txt
  96. Sep 21 22:25 mydumper_mysqlreport_210919-222445.txt
  97. Sep 21 22:25 mydumperbench_210919-222445.txt
  98. Sep 21 22:25 mydumperstats_210919-222445.txt
  99. Sep 21 22:25 mydumpercsv_210919-222445.txt
  100. Sep 21 22:25 mydumper_mycnf_210919-222445.txt
  101. Sep 22 05:16 mydumper_extstats_220919-051501.txt
  102. Sep 22 05:16 mydumper_mysqlreport_220919-051501.txt
  103. Sep 22 05:16 mydumperbench_220919-051501.txt
  104. Sep 22 05:16 mydumperstats_220919-051501.txt
  105. Sep 22 05:16 mydumpercsv_220919-051501.txt
  106. Sep 22 05:16 mydumper_mycnf_220919-051501.txt
  107. *************************************************************************************
  108. CSV format:
  109. threads,real time,user time,system time,cpu %,max mem,context switch waits,speed (MB/s),speed (MB/min),speed (GB/hr)
  110. 2,2.52,1.28,0.75,80,8124,2178,103.57,6214.20,364.11
  111. myloader restore
  112. Code:
  113. ./mydumperbench.sh myloader
  114. ----------------------------------------------------------------------------------------
  115. mydumperbench.sh 0.0.9 by George Liu (eva2000)
  116. https://centminmod.com
  117. Sun Sep 22 05:16:16 UTC 2019
  118. ----------------------------------------------------------------------------------------
  119. ----------------------------------------------------------------------------------------
  120. [export-20190922-051604 total size: 190 MB]
  121. [myloader threads: 2 ] myloader -t 2 -d export-20190922-051604 -B sbtest -q 10000
  122. [restore stats:] real: 13.31s user: 0.36s sys: 0.24s cpu: 4% maxmem: 7912 KB cswaits: 1241
  123. [restore speed:] 14.27 MB/s 856.20 MB/min 50.16 GB/hr
  124. Successfully restored mydumper backup using 2 threads
  125. ----------------------------------------------------------------------------------------
  126. *************************************************************************************
  127. Sun Sep 22 05:17:32 UTC 2019
  128. results logged: /root/mydumperlogs/myloaderbench_220919-051616.txt
  129. Sep 22 05:17 myloader_extstats_220919-051616.txt
  130. Sep 22 05:17 myloader_mysqlreport_220919-051616.txt
  131. Sep 22 05:17 myloaderbench_220919-051616.txt
  132. Sep 22 05:17 myloaderstats_220919-051616.txt
  133. Sep 22 05:17 myloadercsv_220919-051616.txt
  134. Sep 22 05:17 myloader_mycnf_220919-051616.txt
  135. *************************************************************************************
  136. CSV format:
  137. threads,real time,user time,system time,cpu %,max mem,context switch waits,speed (MB/s),speed (MB/min),speed (GB/hr)
  138. 2,13.31,0.36,0.24,4,7912,1241,14.27,856.20,50.16
  139. mysqldump backup
  140. Code:
  141. ./mydumperbench.sh mysqldump
  142. ----------------------------------------------------------------------------------------
  143. mydumperbench.sh 0.0.9 by George Liu (eva2000)
  144. https://centminmod.com
  145. Sun Sep 22 05:19:09 UTC 2019
  146. ----------------------------------------------------------------------------------------
  147. ----------------------------------------------------------------------------------------
  148. [sbtest total size: 261 MB]
  149. [mysqldump threads: 1 ] mysqldump sbtest > /backup-mydumper/mysqldump/sbtest_220919-051909.sql
  150. [backup stats:] real: 4.81s user: 3.49s sys: 0.99s cpu: 93% maxmem: 5120 KB cswaits: 434
  151. [backup speed:] 54.26 MB/s 3255.60 MB/min 190.75 GB/hr
  152. Successfully backed up database using mysqldump
  153. ----------------------------------------------------------------------------------------
  154. *************************************************************************************
  155. Sun Sep 22 05:20:16 UTC 2019
  156. results logged: /root/mydumperlogs/mysqldumpbench_220919-051909.txt
  157. Sep 21 22:21 mysqldump_extstats_210919-222024.txt
  158. Sep 21 22:21 mysqldump_mysqlreport_210919-222024.txt
  159. Sep 21 22:21 mysqldumpbench_210919-222024.txt
  160. Sep 21 22:21 mysqldumpstats_210919-222024.txt
  161. Sep 21 22:21 mysqldumpcsv_210919-222024.txt
  162. Sep 21 22:21 mysqldump_mycnf_210919-222024.txt
  163. Sep 22 05:20 mysqldump_extstats_220919-051909.txt
  164. Sep 22 05:20 mysqldump_mysqlreport_220919-051909.txt
  165. Sep 22 05:20 mysqldumpbench_220919-051909.txt
  166. Sep 22 05:20 mysqldumpstats_220919-051909.txt
  167. Sep 22 05:20 mysqldumpcsv_220919-051909.txt
  168. Sep 22 05:20 mysqldump_mycnf_220919-051909.txt
  169. *************************************************************************************
  170. CSV format:
  171. threads,real time,user time,system time,cpu %,max mem,context switch waits,speed (MB/s),speed (MB/min),speed (GB/hr)
  172. 1,4.81,3.49,0.99,93,5120,434,54.26,3255.60,190.75
  173. mysql restore
  174. Code:
  175. ./mydumperbench.sh mysqlrestore
  176. ----------------------------------------------------------------------------------------
  177. mydumperbench.sh 0.0.9 by George Liu (eva2000)
  178. https://centminmod.com
  179. Sun Sep 22 05:30:12 UTC 2019
  180. ----------------------------------------------------------------------------------------
  181. ----------------------------------------------------------------------------------------
  182. [/backup-mydumper/mysqldump/sbtest_220919-051909.sql size: 188 MB]
  183. [mysql threads: 1 ] mysql sbtest < /backup-mydumper/sbtest.sql
  184. [restore stats:] real: 26.74s user: 3.48s sys: 0.21s cpu: 13% maxmem: 6480 KB cswaits: 1159
  185. [restore speed:] 7.03 MB/s 421.80 MB/min 24.71 GB/hr
  186. Successfully restored mysqldump backup
  187. ----------------------------------------------------------------------------------------
  188. *************************************************************************************
  189. Sun Sep 22 05:31:43 UTC 2019
  190. results logged: /root/mydumperlogs/mysqlrestorebench_220919-053012.txt
  191. Sep 22 05:22 mysqlrestorebench_220919-052052.txt
  192. Sep 22 05:22 mysqlrestore_extstats_220919-052052.txt
  193. Sep 22 05:22 mysqlrestore_mysqlreport_220919-052052.txt
  194. Sep 22 05:22 mysqlrestorestats_220919-052052.txt
  195. Sep 22 05:22 mysqlrestorecsv_220919-052052.txt
  196. Sep 22 05:22 mysqlrestore_mycnf_220919-052052.txt
  197. Sep 22 05:31 mysqlrestore_extstats_220919-053012.txt
  198. Sep 22 05:31 mysqlrestore_mysqlreport_220919-053012.txt
  199. Sep 22 05:31 mysqlrestorebench_220919-053012.txt
  200. Sep 22 05:31 mysqlrestorestats_220919-053012.txt
  201. Sep 22 05:31 mysqlrestorecsv_220919-053012.txt
  202. Sep 22 05:31 mysqlrestore_mycnf_220919-053012.txt
  203. *************************************************************************************
  204. CSV format:
  205. threads,real time,user time,system time,cpu %,max mem,context switch waits,speed (MB/s),speed (MB/min),speed (GB/hr)
  206. 1,26.74,3.48,0.21,13,6480,1159,7.03,421.80,24.71
  207. The more cpu threads your servers have, the faster the mysql database backup and restore via mydumper/myloader. So if majority of that 200GB is MySQL data, you'd need to use faster backup method than cpanel's native pkgacct's mysqldump single threaded tool.
  208. So if majority of that 200GB is MySQL data, you'd need to use faster backup method than cpanel's native pkgacct's mysqldump single threaded tool.
  209. I would strongly recommend skipping the DB in the cpanel backup entirely - and do not perform a mysqldump locally either.
  210.  
  211. Since OP has root on both machines, the best way to transfer the database is to allow New Server to access MySQL over the network (ie, punch a hole in your firewall on OldServer to permit NewServer to connect to it)
  212.  
  213. Then on new server, do the mysqldump command there and -h old.server.ip.address and dump it directly over the network.
  214.  
  215. Why?
  216.  
  217. Because there is almost always a difference in MySQL/MariaDB server versions. A mysqldump on OldServer might not be directly importable into NewServer's database.
  218.  
  219. However, the mysqldump program from a newer version will automatically perform any changes needed during the dump from the old DB server. That dump file is then guaranteed to import cleanly on your new server - without any issues what-so-ever.
  220.  
  221. tl;dr - unless MySQL/MariaDB are exact same versions on both servers then save yourself a potential headache, and run the mysqldump command on the new server's command line to dump the old server's DB.
  222. "I've seen spam you people wouldn't believe. Routers on fire off the OCs of AGIS. I watched MXes burning in the dark near the Cyberpromo Gateway. All those moments will be lost in time, like tears in rain. TTL=0."
  223. Indeed though cpanel to cpanel is more likely to have same MySQL fork version if setup and configured by same person. But mysqldump generally is compatible for MySQL 5.7 and MariaDB 10.3 and below - MySQL 8.0 and MariaDB 10.4 could run into issues. But you can run into issues if going from MariaDB to MySQL due to cpanel user prefix limits.
  224.  
  225. Guess the rule of thumb, ensure cpanel on source and destination servers are the same MySQL/MariaDB versions and should be fine to skip database backups via --skipacctdb and then use multi-threaded mydumper for faster database backup/restores if his 200GB data is mainly from MySQL databases. This would be faster than dumping database backups over the network via specifying -h hostname on mysqldump.
  226. CentminMod.com Nginx Installer (Nginx 1.17, PHP-FPM 5.4-5.6, 7.0-7.3 (7.4 ready), MariaDB 10 + optional ngx_pagespeed/lua) for CentOS
  227. Whats the fastest way to migrate 1 cpanel account with more than 200GB in size?
  228. Both has root access..
  229.  
  230. Thank you!
  231.  
  232. Fastest way without being notice?
  233.  
  234. Leave the old server online while doing the migration and change the DNS accordingly.
  235. Fastest way without being notice?
  236. No lol..
  237.  
  238. I need to migrate the data as fast as possible to minimize the time that the website is closed because its a high traffic and many users are waiting.
  239. If I leave the website open, there might be new contents posted on the website during the transition that might not be included
  240. I need to migrate the data as fast as possible to minimize the time that the website is closed because its a high traffic and many users are waiting.
  241. If I leave the website open, there might be new contents posted on the website during the transition that might not be included
  242. Short of having a full mysql replicated setup with distributed shared NFS share for files and load balancer like setup, having a maintenance static html page is very useful for these cases and if you're creative it can add value as you could have the static html page with messages and links to site's social media pages to keep visitors informed via social media as well as drive traffic and eye balls to social media accounts as well as the obvious - offload load from php/database requests so it speeds up file/db backup and transfer process.
  243.  
  244. You can also do dummy test backup and restore tests to time how long it would take roughly so you have an idea of maintenance duration you convey to visitors.
  245. Its depend on network speed in between your two servers. A few days ago, I tested 70 GB transfer using cPanel migration tool and it I was sucessful. Probably within 20-30 minutes.
  246. Personally would only handle such migrations via manual methods and not via cpanel migration. For example:
  247. Change site to use cloudflare or other proxy services where your origin server can be updated and visitors will hit the new server near instantly.
  248. Do an rsync of the site from one server to the other and ensure ownership of files and directories are correct on the new server
  249. Do another rsync on directories that hold content that is updated constantly to catch any that was done during the initial rsync
  250. Take site down for maintenance
  251. Dump database, import to new server, change script to use new server for database access and/or remove mysql users to avoid split brains.
  252. At proxy service, change origin to new server
  253. Bring site back online at new server only .
  254. Do another rsync of the content directories to get any content updated during the last few steps.
  255.  
  256. You could do the initial rsync via a cpanel migration and then do manual rsyncs afterward. If you go with just rsync for the initial one, do so within a screen or detach the process from the current SSH session to avoid the rsync being interrupted with any connection issues with the SSH session.
  257.  
  258. This process should limit maintenance / downtime to when you are switching the database around and the additional rsyncs should complete quickly since the bulk of the content has been moved over. Users should not see missing content and if they do the last rsync should fix that.
  259. -Steven | u2-web@Cooini, LLC - Business Shared Hosting | Isolate sites with Webspaces | Site Builder | PHP-FPM | MariaDB
  260. WHMCS Modules: Staff Knowledgebase | Custom Modules and Hooks
  261. "It is the mark of an educated mind to be able to entertain a thought without accepting it" -Aristotle
  262. My favorite is the tool built right into cPanel, I've done 50 GB transfers, no problem. Haven't tested any higher capacities yet though.
  263. Well, the million dollar question is still - where is the bulk of that 200GB at?
  264.  
  265. If it's files... rsync is the way to go... using something similar to what @tuxandrew suggested. The nice thing about rsync is... you can synch it as many times as you want. The first synch is going to take a while because it has to copy all the files over. But subsequent synchs - and the more often you run them - will be shorter as the change in files would be less and less. Then skipping the home directory with the cPanel backup package, send that to the new server, move the rsync'd home directory out of the way, restore it, and then move the user's home directory back, do some wide assortment of chown to change all of the files back to their respective owner, update IP, and you're done.
  266.  
  267. But if the bulk of that 200GB is in the database... then this gets quite a bit more complex. I suppose you could set up a MySQL master/slave duplication between the two servers. I really don't have a ton of experience with any of this, perhaps others do. You could dump the database and rsync the differences... but if the database is that large... it's probably going to take a while to both dump and then re-insert on the new server.
  268.  
  269. But the point is, you really need to know where the bulk of the 200GB is being used at and then formulate a procedure based on that.
  270. But if the bulk of that 200GB is in the database... then this gets quite a bit more complex. I suppose you could set up a MySQL master/slave duplication between the two servers. I really don't have a ton of experience with any of this, perhaps others do. You could dump the database and rsync the differences... but if the database is that large... it's probably going to take a while to both dump and then re-insert on the new server.
  271.  
  272. But the point is, you really need to know where the bulk of the 200GB is being used at and then formulate a procedure based on that.
  273. Other option if you have know-how and disk space is use MySQL binary logging so you can do snapshot mysql backup and transfer and they replay the binary logs from existing server on the destination server to get up to update MySQL transactions/data sync'd. But static html maintenance page method would be easiest and cheapest to implement.
  274. Also, as well as telling us whether the bulk of the data is in the DB or the files, I'd suggest you do a trial transfer to see how long it takes and what problems you run into. Document those problems and work out a way around them (probablyt not going to be too difficult), test it, and then repeat it for the real site. You're asking for problems if you try to move the site without testing.
  275.  
  276. Don't know if this is an option, but you might consider putting the site into read-only while you do the transfer. That way at least the site is partly available.
  277.  
  278. Lots of amazing quality advice above. You might want to hire a sysadmin to do the move - 200Gb is up at the difficult end for anyone.
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!