• john-munoz
    0
    I'm running CBB Ultimate on Linux Centos 6.9
    Version: 2.9.0.114

    This is running in an r5.4xlarge EC2 instance (16 vCPU 128 GiB Memory EBS Instance Storage 10 Gbps Networking Performance 3500 Mbps EBS Bandwidth)

    I keep getting failures trying to run a backup to S3 Standard. The error is a generic, "Plan process stopped with system error. Send logs to support." I've sent the logs to support and opened several tickets, but they haven't provided any insight, resolution or troubleshooting steps yet.

    This error occurs at varying times, file points and configurations. There is not particular file that kicks off this error or a particular point at which this error occurs. Sometimes 243 files get copied, sometimes 1866 files get copied. I've been able to get up to 106GB transferred one time before the job failed.

    We are backing up to Amazon S3 Standard bucket. (versioning, server access logging, static hosting, object level logging and default encryption is all disabled)

    We've tried with and without block level backup
    Using encryption, tried different algorithms, server side and client side.
    Standard Storage Class
    Not using S3 Transfer Acceleration
    No pre or post actions

    Set Backup symlinks only
    tried enabling and disabling “ignore SSL certificates”
    Tried limiting cloud storage max speed to 5000 KByte/s
    Max thread count is 5
    Chunk size is 10 Mbyte
    Memory used is 5120
  • Matt
    91
    I checked the status of the ticket and it seems to be a pretty unusual issue. The problem is being investigated with the help of our R&D and level 2 department. You should expect a reply soon.
  • john-munoz
    0
    Thanks Matt, it's been three weeks since that ticket was opened and I was wondering if someone in the community may have some troubleshooting ideas of thoughts on other places to look. ciuczdqgrgupu1me.png

    The log_worker files do not seam to indicate any red herrings. The last lines in /opt/local/CloudBerry Backup/logs (even in debug level) just show the last files and that they were successfully backed up. Are there any other log files that might contain information about the program itself or why it may be crashing?

    The localDeamon log shows this...

    2019-06-21 00:00:00,959436 [INFO ]: [ CBB ] [ 7 ] Run plan by schedule: name: Small Folder Test id: {d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3} time: Fri Jun 21 04:00:00 2019 GMT next run: Sat Jun 22 04:00:00 2019 GMT FULL
    2019-06-21 00:00:00,959477 [INFO ]: [ CBB ] [ 7 ] Plan already running
    2019-06-21 00:00:00,959503 [INFO ]: [ CBB ] [ 7 ] Update schedule info loc: QDateTime(2019-06-21 04:00:00.959 UTC Qt::TimeSpec(UTC))
    2019-06-21 00:00:00,963331 [INFO ]: [ CBB ] [ 7 ] Run plan by schedule: name: Small Folder Test id: {d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3} time: Fri Jun 21 04:00:00 2019 GMT next run: Sat Jun 22 04:00:00 2019 GMT 
    2019-06-21 00:00:00,963352 [INFO ]: [ CBB ] [ 7 ] Plan already running
    2019-06-21 00:00:00,963368 [INFO ]: [ CBB ] [ 7 ] Update schedule info loc: QDateTime(2019-06-21 04:00:00.963 UTC Qt::TimeSpec(UTC))
    2019-06-21 04:39:39,748559 [INFO ]: [ CBB ] [ 7 ] QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test : Listing Process:
    2019-06-21 04:39:39,748593 [INFO ]: [ CBB ] [ 7 ] 
    2019-06-21 04:39:39,748605 [INFO ]: [ CBB ] [ 7 ] End Listing : Status:  QProcess::ExitStatus(CrashExit) code 11
    2019-06-21 07:32:55,720801 [INFO ]: [ CBB ] [ 7 ] Request force run plan "{d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}". Request from id: {e492de20-fc76-4cb2-9776-26874e61d3be}
    2019-06-21 07:32:55,721019 [INFO ]: [ CBB ] [ 7 ] Run plan: QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test
    2019-06-21 07:32:55,721991 [INFO ]: [ CBB ] [ 7 ] Plan started: QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test 
    2019-06-21 07:51:17,926543 [INFO ]: [ CBB ] [ 7 ] QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test : Listing Process:
    2019-06-21 07:51:17,926573 [INFO ]: [ CBB ] [ 7 ] 
    2019-06-21 07:51:17,926585 [INFO ]: [ CBB ] [ 7 ] End Listing : Status:  QProcess::ExitStatus(CrashExit) code 11
    2019-06-21 07:59:27,300164 [INFO ]: [ CBB ] [ 7 ] Request force run plan "{d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}". Request from id: {f59b9010-cbf0-477e-8ffe-0a1375b668cf}
    2019-06-21 07:59:27,300376 [INFO ]: [ CBB ] [ 7 ] Run plan: QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test
    2019-06-21 07:59:27,301317 [INFO ]: [ CBB ] [ 7 ] Plan started: QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test 
    2019-06-21 15:58:36,688512 [INFO ]: [ CBB ] [ 1 ] Local managment service disconnected
    2019-06-21 15:58:36,688852 [INFO ]: [ CBB ] [ 1 ] Local managment service connected
    2019-06-21 20:01:33,335633 [INFO ]: [ CBB ] [ 7 ] QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test : Listing Process:
    2019-06-21 20:01:33,335679 [INFO ]: [ CBB ] [ 7 ] 
    2019-06-21 20:01:33,335691 [INFO ]: [ CBB ] [ 7 ] End Listing : Status:  QProcess::ExitStatus(CrashExit) code 11
    2019-06-22 00:00:00,459825 [INFO ]: [ CBB ] [ 7 ] Run plan by schedule: name: Small Folder Test id: {d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3} time: Sat Jun 22 04:00:00 2019 GMT next run: Sun Jun 23 04:00:00 2019 GMT FULL
    2019-06-22 00:00:00,459861 [INFO ]: [ CBB ] [ 7 ] Run plan: QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test
    2019-06-22 00:00:00,460872 [INFO ]: [ CBB ] [ 7 ] Plan started: QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test FULL
    2019-06-22 00:00:00,461083 [INFO ]: [ CBB ] [ 7 ] Update schedule info loc: QDateTime(2019-06-22 04:00:00.461 UTC Qt::TimeSpec(UTC))
    2019-06-22 00:00:00,464903 [INFO ]: [ CBB ] [ 7 ] Run plan by schedule: name: Small Folder Test id: {d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3} time: Sat Jun 22 04:00:00 2019 GMT next run: Sun Jun 23 04:00:00 2019 GMT 
    2019-06-22 00:00:00,464920 [INFO ]: [ CBB ] [ 7 ] Plan already running
    2019-06-22 00:00:00,464934 [INFO ]: [ CBB ] [ 7 ] Update schedule info loc: QDateTime(2019-06-22 04:00:00.464 UTC Qt::TimeSpec(UTC))
    2019-06-22 00:30:10,415567 [INFO ]: [ CBB ] [ 7 ] QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test : Listing Process:
    2019-06-22 00:30:10,415599 [INFO ]: [ CBB ] [ 7 ] 
    2019-06-22 00:30:10,415610 [INFO ]: [ CBB ] [ 7 ] End Listing : Status:  QProcess::ExitStatus(CrashExit) code 11
    2019-06-23 00:00:00,459227 [INFO ]: [ CBB ] [ 7 ] Run plan by schedule: name: Small Folder Test id: {d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3} time: Sun Jun 23 04:00:00 2019 GMT next run: Mon Jun 24 04:00:00 2019 GMT FULL
    2019-06-23 00:00:00,459270 [INFO ]: [ CBB ] [ 7 ] Run plan: QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test
    2019-06-23 00:00:00,460445 [INFO ]: [ CBB ] [ 7 ] Plan started: QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test FULL
    2019-06-23 00:00:00,460667 [INFO ]: [ CBB ] [ 7 ] Update schedule info loc: QDateTime(2019-06-23 04:00:00.460 UTC Qt::TimeSpec(UTC))
    2019-06-23 00:00:00,464735 [INFO ]: [ CBB ] [ 7 ] Run plan by schedule: name: Small Folder Test id: {d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3} time: Sun Jun 23 04:00:00 2019 GMT next run: Mon Jun 24 04:00:00 2019 GMT 
    2019-06-23 00:00:00,464751 [INFO ]: [ CBB ] [ 7 ] Plan already running
    2019-06-23 00:00:00,464766 [INFO ]: [ CBB ] [ 7 ] Update schedule info loc: QDateTime(2019-06-23 04:00:00.464 UTC Qt::TimeSpec(UTC))
    2019-06-23 04:03:43,367524 [INFO ]: [ CBB ] [ 7 ] QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test : Listing Process:
    2019-06-23 04:03:43,367559 [INFO ]: [ CBB ] [ 7 ] 
    2019-06-23 04:03:43,367571 [INFO ]: [ CBB ] [ 7 ] End Listing : Status:  QProcess::ExitStatus(CrashExit) code 11
    2019-06-24 00:00:00,459195 [INFO ]: [ CBB ] [ 7 ] Run plan by schedule: name: Small Folder Test id: {d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3} time: Mon Jun 24 04:00:00 2019 GMT next run: Tue Jun 25 04:00:00 2019 GMT FULL
    2019-06-24 00:00:00,459232 [INFO ]: [ CBB ] [ 7 ] Run plan: QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test
    2019-06-24 00:00:00,460377 [INFO ]: [ CBB ] [ 7 ] Plan started: QUuid({d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3}) Small Folder Test FULL
    2019-06-24 00:00:00,460605 [INFO ]: [ CBB ] [ 7 ] Update schedule info loc: QDateTime(2019-06-24 04:00:00.460 UTC Qt::TimeSpec(UTC))
    2019-06-24 00:00:00,464405 [INFO ]: [ CBB ] [ 7 ] Run plan by schedule: name: Small Folder Test id: {d8d5c4d1-2003-4cd6-b142-b70f5cb82ff3} time: Mon Jun 24 04:00:00 2019 GMT next run: Tue Jun 25 04:00:00 2019 GMT 
    2019-06-24 00:00:00,464421 [INFO ]: [ CBB ] [ 7 ] Plan already running
    2019-06-24 00:00:00,464436 [INFO ]: [ CBB ] [ 7 ] Update schedule info loc: QDateTime(2019-06-24 04:00:00.464 UTC Qt::TimeSpec(UTC))
    
  • john-munoz
    0
    I also noticed that these messages were showing up in /var/logs/messages

    Jun 25 10:48:11 ip-10-231-100-183 abrt[30932]: Saved core dump of pid 12462 (/opt/local/CloudBerry Backup/raw_bin/cbbWorker) to /var/spool/abrt/ccpp-2019-06-25-10:48:03-12462 (1048576000 bytes)
    Jun 25 10:48:11 ip-10-231-100-183 abrtd: Directory 'ccpp-2019-06-25-10:48:03-12462' creation detected
    Jun 25 10:48:11 ip-10-231-100-183 abrtd: Package 'cloudberry-backup' isn't signed with proper key
    Jun 25 10:48:11 ip-10-231-100-183 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2019-06-25-10:48:03-12462' exited with 1
    Jun 25 10:48:11 ip-10-231-100-183 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2019-06-25-10:48:03-12462'
    Jun 25 10:54:29 ip-10-231-100-183 abrt[29993]: Saved core dump of pid 6997 (/opt/local/CloudBerry Backup/raw_bin/cbbWebAccess) to /var/spool/abrt/ccpp-2019-06-25-10:54:27-6997 (382730240 bytes)
    Jun 25 10:54:29 ip-10-231-100-183 abrtd: Directory 'ccpp-2019-06-25-10:54:27-6997' creation detected
    Jun 25 10:54:29 ip-10-231-100-183 abrtd: Package 'cloudberry-backup' isn't signed with proper key
    Jun 25 10:54:29 ip-10-231-100-183 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2019-06-25-10:54:27-6997' exited with 1
    Jun 25 10:54:29 ip-10-231-100-183 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2019-06-25-10:54:27-6997'
  • Fred Boniface
    0
    I seem to be having the same issue with backups failing due to system error. I am currently working on trawling through logs to see what is up.
  • Fred Boniface
    0
    These are the relevant logs I can see, not too sure whether it is related to your problem or not. The only changed I made recently was to add some additional files to the plan but they don't appear to be the problem here.

    2019-06-26 09:41:36,008940 [WARN ]: [ CBC ] [ 8 ] Cloud operation error (NoContent): No Content
    2019-06-26 09:41:36,009211 [WARN ]: [ CBC ] [ 11 ] Finished with error, path: /CBB_domoticz/home/domoticz/domoticz/domoticz.db$/20190621020000/full.cbbitmap message: Operation error. Don't retry it
    2019-06-26 09:41:36,009405 [INFO ]: [ CBB ] [ 4 ] Finished download file/part, path: /opt/local/CloudBerry Backup/etc/config/dfmap/{2c5f63cb-b78c-434e-b078-8b9a8b7576c9} position: 0 size: 0
    2019-06-26 09:41:36,009499 [ERROR]: [ CBB ] [ 4 ] Error on download file/part path: /opt/local/CloudBerry Backup/etc/config/dfmap/{2c5f63cb-b78c-434e-b078-8b9a8b7576c9} code: 13 message: No Content
    2019-06-26 09:41:36,009577 [WARN ]: [ CBB ] [ 4 ] Error on download dirrerential revision map, cloud path: /CBB_domoticz/home/domoticz/domoticz/domoticz.db$/20190621020000/full.cbbitmap error code: 13 mess$2019-06-26 09:41:36,010159 [WARN ]: [ CBB ] [ 4 ] Error on open last revision differential map, id: {2c5f63cb-b78c-434e-b078-8b9a8b7576c9} message: No such file or directory
    2019-06-26 09:41:36,010260 [ERROR]: [ CBB ] [ 4 ] Backup error: /home/domoticz/domoticz/domoticz.db
    message: Invalid last revision diffirential stream
    code: 12
    2019-06-26 09:41:36,010305 [ERROR]: [ CBB ] [ 4 ] Backup finished: /home/domoticz/domoticz/domoticz.db with errror: Invalid last revision diffirential stream
    2019-06-26 09:41:36,183151 [WARN ]: QObject::startTimer: Timers cannot be started from another thread
    
  • Boris
    0

    Hi,

    The crash code 11 is a known issue, our developers are currently investigating it. There's no workarounds for it, as far as I know.


    Are you using Openstack, by any chance? We'll need the diagnostic information to troubleshoot this, please send it to us by using the Feedback button in the GUI and mention this thread in the description of the issue.
  • Jim Q
    0
    Adding that I'm having a similar issue. Just setup CBB Ultimate on Ubuntu 18.04 and have not been able to successfully complete a large backup. I've opened a ticket with Cloudberry and submitted the logs via the Feedback button. I've gotten two small backups to finish but larger ones will die at random intervals.
  • Matt
    91
    The issue usually happens sporadically without any pattern, that's just a daemon crash. It's one of our top priority issues to fix it for Linux/mac software.
  • Damian
    0
    Is there an update to this issue? I just got started and paid for Cloudberry backup and having the same issue as John. I'm running the latest 2.9.2.14 version and backing up to B2. It seems to be related to thread count, as I was able to get a normal backup exit by setting the number of threads to 1. It's just slower in this case, and overnight backups are going to take a month at this point to complete.
  • Matt
    91
    It's an ongoing problem that is being investigated by our developers. According to the latest info improvements will be made in version 2.9. It seems they were able to localize the problem with RAM and are testing things internally.
bold
italic
underline
strike
code
quote
ulist
image
url
mention
reveal
youtube
tweet
Add a Comment