My settings

Hi @brankod, thanks for posting the php settings that are working for you!

From my understanding, the issue for most DreamHost users is that they have their fcgid IO Timeout set to 2 minutes. If you have a larger site that is going to take more than 2 minutes to backup, the backup process will fail.

The FastCGI suggestion is because the same timeouts are not set in that type of environment.

I think what may be helping in your case above the most is the compression level being set to 1. In our testing, the lower the compression the faster backups complete, but the larger the file size.

As an FYI of the compression level, I found a good snippet here:
https://linuxize.com/post/how-to-zip-files-and-directories-in-linux/

The zip command allows you to specify a compression level using number prefixed with a dash from 0 to 9. The default compression level is -6. When using -0, all files will be stored without compression. -9 will force the zip command to use an optimal compression for all files.

I’m glad things are working @brankod! If anything changes, please do let us know, we’re happy to help!
– Brad

Der Brad,
I made backup with compression level 9 this time. Everything went with no problems at all. Difference is only 300-400mb. Not big deal.
But now problem is restore. Memory peak.

[2020-07-09 19:32:29 UTC] Last error: Array
(
    [type] => 8
    [message] => Undefined index: td_action
    [file] => /home/dh_vaepg3/xxxxxxx.dreamhosters.com/wp-content/plugins/td-composer/css-live/css-live.php
    [line] => 30
)

[2020-07-09 19:32:29 UTC] PHP Version: 7.3.16
[2020-07-09 19:32:29 UTC] WordPress Version: 5.4.2
[2020-07-09 19:32:29 UTC] Total Upkeep version: 1.14.1
[2020-07-09 19:32:29 UTC] Restoration process initialized.
[2020-07-09 19:32:29 UTC] Memory usage - limit / current / peak memory usage: 500M / 91869008 (87.61 MB) / 92412064 (88 MB)
[2020-07-09 19:32:29 UTC] Restore info: Array
(
    [mode] => restore
    [dryrun] => 
    [filename] => boldgrid-backup-xxxxx.dreamhosters.com-8929ac2e-20200709-120908.zip
    [filepath] => /home/dh_vaepg3/boldgrid_backup/boldgrid-backup-xxxxx.dreamhosters.com-8929ac2e-20200709-120908.zip
    [filesize] => 6925962602
    [archive_key] => 1
    [restore_ok] => 1
)

[2020-07-09 19:32:36 UTC] Unzipping archive... filepath / ABSPATH: /home/dh_vaepg3/boldgrid_backup/boldgrid-backup-xxxxxx.dreamhosters.com-8929ac2e-20200709-120908.zip / /home/dh_vaepg3/xxxxxxx.dreamhosters.com/
[2020-07-09 19:32:36 UTC] Memory usage - limit / current / peak memory usage: 1073741824 / 91874280 (87.62 MB) / 92412064 (88 MB)
[2020-07-09 19:35:33 UTC] Last error: Array
(
    [type] => 1
    [message] => Allowed memory size of 1073741824 bytes exhausted (tried to allocate 1677108190 bytes)
    [file] => /home/dh_vaepg3/xxxxxx.dreamhosters.com/wp-admin/includes/file.php
    [line] => 1571
)

Hi @brankod

I do think that we need to increase the memory limit defined in the plugin. We could do it based on the size of the archive being restored, or we could just set it to a constant value.

Before we do that, can you help confirm that changing the memory limit will resolve this issue? You can do this by changing this line:

https://github.com/BoldGrid/boldgrid-backup/blob/master/admin/class-boldgrid-backup-admin-core.php#L2279

From Boldgrid_Backup_Admin_Utility::bump_memory_limit( '1G' );

To Boldgrid_Backup_Admin_Utility::bump_memory_limit( '2G' );

That will increase the memory limit for the restoration from 1G to 2G.

Thanks,
– Brad

Will do ASAP.
One more thing. I it possible to split restoring data chunks on smaller pieces? Maybe? Don’t know…
Sorry bad English.

Hi @brankod,

At this time, unfortunately we don’t have that option. Total Upkeep currently only creates 1 zip file.

However, if you’re familiar working with the files in a WordPress installation, you can download the zip file to your local computer, open the zip, and restore the files that way. There’s a .sql file there too that would need to be restored via something like phpmyadmin.

Thanks,
– Brad

No luck again. Changed to bump_memory_limit( '2G' );

[2020-07-11 04:46:19 UTC] Last error: Array
(
    [type] => 8
    [message] => Undefined index: td_action
    [file] => /home/dh_rbtdji/xxxxx.dreamhosters.com/wp-content/plugins/td-composer/css-live/css-live.php
    [line] => 30
)

[2020-07-11 04:46:19 UTC] PHP Version: 7.3.16
[2020-07-11 04:46:19 UTC] WordPress Version: 5.4.2
[2020-07-11 04:46:19 UTC] Total Upkeep version: 1.14.1
[2020-07-11 04:46:19 UTC] Restoration process initialized.
[2020-07-11 04:46:19 UTC] Memory usage - limit / current / peak memory usage: 500M / 95296696 (90.88 MB) / 95834840 (91 MB)
[2020-07-11 04:46:19 UTC] Restore info: Array
(
    [mode] => restore
    [dryrun] => 
    [filename] => boldgrid-backup-xxxxxx.dreamhosters.com-8929ac2e-20200711-040157.zip
    [filepath] => /home/dh_rbtdji/boldgrid_backup/boldgrid-backup-xxxxxx.dreamhosters.com-8929ac2e-20200711-040157.zip
    [filesize] => 6906297776
    [archive_key] => 0
    [restore_ok] => 1
)

[2020-07-11 04:46:27 UTC] Unzipping archive... filepath / ABSPATH: /home/dh_rbtdji/boldgrid_backup/boldgrid-backup-xxxxxx.dreamhosters.com-8929ac2e-20200711-040157.zip / /home/dh_rbtdji/xxxxxx.dreamhosters.com/
[2020-07-11 04:46:27 UTC] Memory usage - limit / current / peak memory usage: 10737418240 / 95301968 (90.89 MB) / 95834840 (91 MB)

Hi @brankod,

It looks like you have a 6.9 GB backup file that you’re trying to restore, that quite impressive!

If this is the complete log, it seems like the restoration script is being killed by your server for being too intensive (either memory or cpu). IE the last time it failed it gave us an “out of memory” error, but this time it says it’s “unzipping archive” – but then we don’t see anything else in the logs.

It looks like you’re with DreamHost. I’ll reach out to their support to see if they can see any additional errors.

Thanks,
– Brad

I made this changes:
memory_limit = 512M
upload_max_filesize = 100M
post_max_size = 105M
max_execution_time = 3600
max_input_time = 3600
max_input_vars = 2000

Used system zip. level 6
Backup went as dream.
After that, tried to restore. Everything seems ok. Got success message.
But….nothing was restored!
Please take look at log:

[2020-07-11 11:59:20 UTC] Last error: Array
(
    [type] => 8
    [message] => Undefined index: td_action
    [file] => /home/dh_rbtdji/xxxx.dreamhosters.com/wp-content/plugins/td-composer/css-live/css-live.php
    [line] => 30
)

[2020-07-11 11:59:20 UTC] PHP Version: 7.3.16
[2020-07-11 11:59:20 UTC] WordPress Version: 5.4.2
[2020-07-11 11:59:20 UTC] Total Upkeep version: 1.14.1
[2020-07-11 11:59:20 UTC] Restoration process initialized.
[2020-07-11 11:59:20 UTC] Memory usage - limit / current / peak memory usage: 512M / 91698120 (87.45 MB) / 92266288 (88 MB)
[2020-07-11 11:59:20 UTC] Restore info: Array
(
    [mode] => restore
    [dryrun] => 
    [filename] => boldgrid-backup-xxxxx.dreamhosters.com-8929ac2e-20200711-101603.zip
    [filepath] => /home/dh_rbtdji/boldgrid_backup/boldgrid-backup-xxxxx.dreamhosters.com-8929ac2e-20200711-101603.zip
    [filesize] => 6754025637
    [archive_key] => 0
    [restore_ok] => 1
)

[2020-07-11 11:59:32 UTC] Unzipping archive... filepath / ABSPATH: /home/dh_rbtdji/boldgrid_backup/boldgrid-backup-xxxxxx.dreamhosters.com-8929ac2e-20200711-101603.zip / /home/dh_rbtdji/xxxxxx.dreamhosters.com/
[2020-07-11 11:59:32 UTC] Memory usage - limit / current / peak memory usage: 1073741824 / 91703392 (87.46 MB) / 92266288 (88 MB)
[2020-07-11 12:02:44 UTC] Last error: Array
(
    [type] => 2
    [message] => mkdir(): File exists
    [file] => /home/dh_rbtdji/xxxxx.dreamhosters.com/wp-admin/includes/class-wp-filesystem-direct.php
    [line] => 508
)

[2020-07-11 12:02:44 UTC] Unzip complete! Status: 1
[2020-07-11 12:02:44 UTC] Memory usage - limit / current / peak memory usage: 1073741824 / 91704136 (87.46 MB) / 135589968 (129 MB)
[2020-07-11 12:02:44 UTC] Attempting database restoration... $db_dump_filepath = 
[2020-07-11 12:02:44 UTC] Memory usage - limit / current / peak memory usage: 1073741824 / 91706896 (87.46 MB) / 135589968 (129 MB)
[2020-07-11 12:02:44 UTC] Database restoration complete.
[2020-07-11 12:02:44 UTC] Memory usage - limit / current / peak memory usage: 1073741824 / 91706896 (87.46 MB) / 135589968 (129 MB)
[2020-07-11 12:02:45 UTC] Sending "restoration complete" email notification...
[2020-07-11 12:02:45 UTC] Email sent. Status: Success
[2020-07-11 12:02:45 UTC] Restoration complete!

ps: Even “Zip Archive” works, but at the begining of restore process shows error:

[2020-07-11 12:23:21 UTC] Last error: Array
(
    [type] => 8
    [message] => Undefined index: td_action
    [file] => /home/dh_rbtdji/xxxxxx.dreamhosters.com/wp-content/plugins/td-composer/css-live/css-live.php
    [line] => 30
)

[2020-07-11 12:23:21 UTC] PHP Version: 7.3.16
[2020-07-11 12:23:21 UTC] WordPress Version: 5.4.2
[2020-07-11 12:23:21 UTC] Total Upkeep version: 1.14.1
[2020-07-11 12:23:21 UTC] Restoration process initialized.
[2020-07-11 12:23:21 UTC] Memory usage - limit / current / peak memory usage: 512M / 91696704 (87.45 MB) / 92264872 (88 MB)
[2020-07-11 12:23:21 UTC] Restore info: Array
(
    [mode] => restore
    [dryrun] => 
    [filename] => boldgrid-backup-xxxxxx.dreamhosters.com-8929ac2e-20200711-120634.zip
    [filepath] => /home/dh_rbtdji/boldgrid_backup/boldgrid-backup-xxxxxx.dreamhosters.com-8929ac2e-20200711-120634.zip
    [filesize] => 6747650489
    [archive_key] => 0
    [restore_ok] => 1
)

[2020-07-11 12:23:29 UTC] Unzipping archive... filepath / ABSPATH: /home/dh_rbtdji/boldgrid_backup/boldgrid-backup-xxxxx.dreamhosters.com-8929ac2e-20200711-120634.zip / /home/dh_rbtdji/xxxxx.dreamhosters.com/
[2020-07-11 12:23:29 UTC] Memory usage - limit / current / peak memory usage: 1073741824 / 91701976 (87.45 MB) / 92264872 (88 MB)
[2020-07-11 12:23:33 UTC] Unzip complete! Status: WP_Error Object
(
    [errors] => Array
        (
            [incompatible_archive] => Array
                (
                    [0] => Incompatible Archive.
                )

        )

    [error_data] => Array
        (
            [incompatible_archive] => PCLZIP_ERR_BAD_FORMAT (-10) : Invalid archive structure
        )

)

[2020-07-11 12:23:33 UTC] Memory usage - limit / current / peak memory usage: 1073741824 / 91704392 (87.46 MB) / 92264872 (88 MB)

I think this plugin has potential, but needs improvement for bigger sites.

If this is the complete log, it seems like the restoration script is being killed by your server for being too intensive (either memory or cpu). IE the last time it failed it gave us an “out of memory” error, but this time it says it’s “unzipping archive” – but then we don’t see anything else in the logs.

It looks like you’re with DreamHost. I’ll reach out to their support to see if they can see any additional errors.

Yes, this is complete log. DreamHost is problem. I tried few plugins and all of them are kicked out from server.

Hi @brankod,

I’m sorry for taking so long to get back to you.

—

After that, tried to restore. Everything seems ok. Got success message.
But….nothing was restored!

In looking at your log, I see this line:
[2020-07-11 12:02:44 UTC] Attempting database restoration... $db_dump_filepath =
It looks like it’s missing the filepath of the zip file. It should look something like this:
[2020-06-29 14:14:38 UTC] Attempting database restoration... $db_dump_filepath = /home/user/public_html/php-54/single-site-3/domain.com.20200629-135529.sql

Are you sure the zip file you’re restoring includes a .sql file?

—

[2020-07-11 12:23:21 UTC] Last error: Array
(
[type] => 8
[message] => Undefined index: td_action
[file] => /home/dh_rbtdji/xxxxxx.dreamhosters.com/wp-content/plugins/td-composer/css-live/css-live.php
[line] => 30
)

The plugin you’re running, is triggering a notice (the type “8”). Most likely that’s not causing any real problems.

—

I tried few plugins and all of them are kicked out from server.

I was a little curious about this. I just wanted to confirm – you’ve tried other backup / restore plugins, and none of them are working?

—

I heard back from DreamHost, and it looks like based on the size of your backup, they may be able to help.

I mentioned to them that there’s a CLI command that can be used to restore:
[user@host wp-content/boldgrid-backup]$ php cli/bgbkup-cli.php restore zip=/home/user/boldgrid_backup/backup.zip

They said they may be able to run the command for you. They also said you may have access to try to run the wp-cli command on your own.

If worse comes to worse, the zip file is pretty straight forward. The zip simply needs to be unzipped, and the .sql file needs to be restore to your database.

Are you able to contact DreamHost for help with the above?

—

Thanks,
– Brad

Brad,
I appreciate your effort. Thank you.
I was wrong at the first place. Backup is not completed properly. Database is corrupted. It is in zip file but damaged.Even sql file exists in zip, when i check “More info Show”, all tables are excluded. Of course, i choose “Full backup” every time. Strange thing is that sql is inside zip and it is 1.6GB after extracting on desktop. That is correct size.
Cli option doesn’t help me. Got this:Error: Unknown database dump filename.

Hi @brankod,

—

Database is corrupted. It is in zip file but damaged.

Can you share why you think the database export is corrupted?

Does that .sql file start with
-- mysqldump-php https://github.com/ifsnop/mysqldump-php
… and end with something like this?
-- Dump completed on: Tue, 14 Jul 2020 17:49:13 +0000
If it does, it sounds like the database export completed successfully. Otherwise, it could be corrupted.

—

ps: database is 1.7GB, all files size before compression: 8.50 GB

I think this plugin has potential, but needs improvement for bigger sites.

While Total Upkeep has no limitations when it comes to large websites, when run on shared hosts with limited resources, backing up / restoring large websites can be problematic in your case.

DreamHost is problem. I tried few plugins and all of them are kicked out from server.

I agree that improvements do need to be made to Total Upkeep to help with large sites. I was curious if other plugins worked – if so we could learn something from them – but it sounds like other backup plugins are not working either with your large site.

—

I really want to see you succeed with getting your website transferred. Total Upkeep may not be the ideal choice in this scenario if you’re hitting limits with your shared host’s memory / cpu / timeouts.

1. Have you tried creating 2 backups – (1) with just your files and (2) one with just your database? Then trying to restore those?

2. Are you familiar with transferring a WordPress website without using a backup plugin? But instead, a manual transfer?

3. If the large database is the issue, you could always create backups with only a few of your database tables. With Total Upkeep you can choose which tables are included in the backup. If you just include the database, you can make a few backups with only a few of your tables until you’ve got backups of the entire database.

—

Again, my main point is that I want to see you transfer your website successfully, with our without Total Upkeep. Let me know how we can help.

Thanks!
– Brad

Hi Brad,
Still no progress with Total upkeep.
Backup Buddy won’t backup to.
Updraft, just errors. Got some backup reaults with Updraft CLI, but only with 25mb chunks.
Duplicator errors.
Got some instructions from DH support about increasing wp memory and no luck.
All in one WP Migrator did a job successful.
Wp db export and import directly on server works fine with Putty. CLI zipping and taring and vice-versa works fine.
So, cli is my destiny. :))
Total upkeep works good on smaller sites. 8.5GB is too much.

1. Have you tried creating 2 backups – (1) with just your files and (2) one with just your database? Then trying to restore those?

2. Are you familiar with transferring a WordPress website without using a backup plugin? But instead, a manual transfer?

3. If the large database is the issue, you could always create backups with only a few of your database tables. With Total Upkeep you can choose which tables are included in the backup. If you just include the database, you can make a few backups with only a few of your tables until you’ve got backups of the entire database.

1.Yes, but not sure what happened exactly. Errors. Maybe my mistake defining exclude-include parameters.
2.Yes
3.Post meta is 1GB (theme files-news portal,so many meta data).Post meta is biggest problem.

ZipArchive log

[2020-07-15 15:20:40 UTC] Last error: Array
(
    [type] => 8
    [message] => Undefined index: td_action
    [file] => /home/dh_rbtdji/xxxxx.dreamhosters.com/wp-content/plugins/td-composer/css-live/css-live.php
    [line] => 30
)

[2020-07-15 15:20:40 UTC] PHP Version: 7.4.3
[2020-07-15 15:20:40 UTC] WordPress Version: 5.4.2
[2020-07-15 15:20:40 UTC] Total Upkeep version: 1.14.1
[2020-07-15 15:20:40 UTC] Backup process initialized.
[2020-07-15 15:20:40 UTC] Starting dump of database...
[2020-07-15 15:20:40 UTC] Memory usage - limit / current / peak memory usage: 1073741824 / 85587328 (81.62 MB) / 85742944 (82 MB)
[2020-07-15 15:20:40 UTC] Dump of database complete! $status = 1
[2020-07-15 15:20:40 UTC] Memory usage - limit / current / peak memory usage: 1073741824 / 85591064 (81.63 MB) / 85742944 (82 MB)

System zip log

[2020-07-15 15:15:15 UTC] Last error: Array
(
    [type] => 8
    [message] => Undefined index: td_action
    [file] => /home/dh_rbtdji/xxxxxx.dreamhosters.com/wp-content/plugins/td-composer/css-live/css-live.php
    [line] => 30
)

[2020-07-15 15:15:15 UTC] PHP Version: 7.4.3
[2020-07-15 15:15:15 UTC] WordPress Version: 5.4.2
[2020-07-15 15:15:15 UTC] Total Upkeep version: 1.14.1
[2020-07-15 15:15:15 UTC] Backup process initialized.
[2020-07-15 15:15:15 UTC] Starting dump of database...
[2020-07-15 15:15:15 UTC] Memory usage - limit / current / peak memory usage: 1073741824 / 85223136 (81.28 MB) / 85453472 (81 MB)

• This reply was modified 4 years, 8 months ago by brankod.
• This reply was modified 4 years, 8 months ago by Steven Stern (sterndata).

Hi @brankod,

Thanks for sharing all that info ??

It looks like you got your site restored, I’m glad to hear that! Here at BoldGrid, we have some improvements to work on, and I really appreciate all the info you’ve shared and time you’ve taken to do so.

Thanks again!
– Brad