Skip to content
This repository has been archived by the owner on May 9, 2023. It is now read-only.

Add support for stream=tar for streaming #194

Closed
ShahriyarR opened this issue Nov 20, 2017 · 4 comments
Closed

Add support for stream=tar for streaming #194

ShahriyarR opened this issue Nov 20, 2017 · 4 comments
Labels
enhancement

Comments

@ShahriyarR
Copy link
Owner

For now the tool has only --stream=xbstream format.
ShahriyarR added a commit that referenced this issue Nov 27, 2017
@ShahriyarR
Preliminary fix for issue #194
55b7663
@ShahriyarR
Copy link
Owner Author 

2017-11-27 09:06:41 DEBUG    The following backup command will be executed /home/shahriyar.rzaev/XB_TEST/server_dir/target/percona-xtrabackup-2.4.x-debug/bin/xtrabackup --defaults-file= --user=root --password=''  --target-dir=/home/shahriyar.rzaev/XB_TEST/backup_dir/ps_5_7_x_2_4/full/2017-11-27_09-06-41 --backup --socket=/home/shahriyar.rzaev/XB_TEST/server_dir/PS231117-5.7.20-18-linux-x86_64-debug/socket.sock --compress=quicklz --compress-chunk-size=65536 --compress-threads=4 --encrypt=AES256 --encrypt-key=VVTBwgM4UhwkTTV98fhuj+D1zyWoA89K --encrypt-threads=4 --encrypt-chunk-size=65536 --slave-info --no-version-check --core-file --parallel=10 --throttle=40 --keyring-file-data=/home/shahriyar.rzaev/XB_TEST/server_dir/PS231117-5.7.20-18-linux-x86_64-debug/mysql-keyring/keyring --stream="tar" > /home/shahriyar.rzaev/XB_TEST/backup_dir/ps_5_7_x_2_4/full/2017-11-27_09-06-41/full_backup.tar
2017-11-27 09:06:41 DEBUG    Starting /home/shahriyar.rzaev/XB_TEST/server_dir/target/percona-xtrabackup-2.4.x-debug/bin/xtrabackup
2017-11-27 09:06:41 ERROR    FAILED: FULL BACKUP
2017-11-27 09:06:41 ERROR    171127 09:06:41 Connecting to MySQL server host: localhost, user: root, password: set, port: not set, socket: /home/shahriyar.rzaev/XB_TEST/server_dir/PS231117-5.7.20-18-linux-x86_64-debug/socket.sock
Using server version 5.7.20-18-debug
/home/shahriyar.rzaev/XB_TEST/server_dir/target/percona-xtrabackup-2.4.x-debug/bin/xtrabackup version 2.4.8 based on MySQL server 5.7.13 Linux (x86_64) (revision id: )
xtrabackup: error: compressed and encrypted backups are incompatible with the 
'tar' streaming format. Use --stream=xbstream instead.
2017-11-27 09:06:41 DEBUG    <pid.PidFile object at 0x7f97b577dcc8> closing pidfile: /tmp/MySQL-AutoXtraBackup/autoxtrabackup.pid
Traceback (most recent call last):
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/bin/autoxtrabackup", line 11, in <module>
    load_entry_point('mysql-autoxtrabackup==1.5.0', 'console_scripts', 'autoxtrabackup')()
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/click-6.7-py3.5.egg/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/click-6.7-py3.5.egg/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/click-6.7-py3.5.egg/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/click-6.7-py3.5.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/click-6.7-py3.5.egg/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/mysql_autoxtrabackup-1.5.0-py3.5.egg/autoxtrabackup.py", line 239, in all_procedure
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/mysql_autoxtrabackup-1.5.0-py3.5.egg/master_backup_script/backuper.py", line 691, in all_backup
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/mysql_autoxtrabackup-1.5.0-py3.5.egg/master_backup_script/backuper.py", line 343, in full_backup
RuntimeError: FAILED: FULL BACKUP

@ShahriyarR
Copy link
Owner Author

Due to error it is clear that compression + encryption backups is not supported by tar format.

@ShahriyarR
Copy link
Owner Author

So it turned out it is impossible to create incremental backups is tar streaming.
I need only configure to take full backup with --stream=tar

2017-11-27 10:27:29 DEBUG    The following backup command will be executed /home/shahriyar.rzaev/XB_TEST/server_dir/target/percona-xtrabackup-2.4.x-debug/bin/xtrabackup --defaults-file= --user=root --password='' --target-dir=/home/shahriyar.rzaev/XB_TEST/backup_dir/ps_5_7_x_2_4/inc/2017-11-27_10-27-29 --incremental-basedir=/home/shahriyar.rzaev/XB_TEST/backup_dir/ps_5_7_x_2_4/full/2017-11-27_10-27-07 --backup --socket=/home/shahriyar.rzaev/XB_TEST/server_dir/PS231117-5.7.20-18-linux-x86_64-debug/socket.sock --slave-info --no-version-check --core-file --parallel=10 --throttle=40 --keyring-file-data=/home/shahriyar.rzaev/XB_TEST/server_dir/PS231117-5.7.20-18-linux-x86_64-debug/mysql-keyring/keyring --stream="tar" > /home/shahriyar.rzaev/XB_TEST/backup_dir/ps_5_7_x_2_4/inc/2017-11-27_10-27-29/inc_backup.tar
2017-11-27 10:27:29 ERROR    FAILED: INCREMENTAL BACKUP
2017-11-27 10:27:29 ERROR    171127 10:27:29 Connecting to MySQL server host: localhost, user: root, password: set, port: not set, socket: /home/shahriyar.rzaev/XB_TEST/server_dir/PS231117-5.7.20-18-linux-x86_64-debug/socket.sock
Using server version 5.7.20-18-debug
/home/shahriyar.rzaev/XB_TEST/server_dir/target/percona-xtrabackup-2.4.x-debug/bin/xtrabackup version 2.4.8 based on MySQL server 5.7.13 Linux (x86_64) (revision id: )
incremental backup from 2573994 is enabled.
xtrabackup: error: streaming incremental backups are incompatible with the 
'tar' streaming format. Use --stream=xbstream instead.

@ShahriyarR ShahriyarR mentioned this issue Nov 27, 2017
Merged
@ShahriyarR
Copy link
Owner Author

During tests, could not find any other issues.
So can be merged to release_v1.5.1

ShahriyarR pushed a commit that referenced this issue Nov 27, 2017
Bugfix issue194 (#203)
31385fb 
* Preliminary fix for issue #194

* Added some code to take stream=tar incremental backup

* Fixing tar command to extract to backup folder

* Doing some sanity checks for disabling taking incremental backups with --stream=tar

* Added extracting from tar archive for full backup prepare

* Trying to figure out recent fail
ShahriyarR pushed a commit that referenced this issue Dec 14, 2017
Release v1.5.1 (#227)
06aecbb 
* Bugfix issue195 (#198)

* Preliminary fix for issue #195

* Changed the user name

* Finalizing the feature

* Bugfix issue196 (#200)

* Preliminary fix for issue #196

* Fixing failed sql syntax

* Added sleep() statement

* Increasing sleep()

* Increased sleep to 30 again.

* Changed check uptime

* Added sleep again

* Raised --server-id random range

* changing random server id choice

* Changed values a bit

* Added --no-check-slave-tables for pt-table-checksum

* Finalizing the bug fix

* Bugfix issue199 (#201)

* Passing the xtra_options to copy-back action

* logging the copy-back action output

* Bugfix issue197 (#202)

* Preliminary fix for issue #197

* Added statement to alter table column collation

* Added config file generation for PS 5.5

* fixing build script for PS 5.5

* Renaming basedirs

* added new assert

* fixing basedir passing here

* Logging the failed sql statement

* Fixed alter statement

* Doing some new staff for PS 5.5

* Adding quotes

* Dropping blank users for 5.5 PS as well.

* Adding changes for generating config for PS 5.5

* Added static method for parsing xtrabackup_slave_info

* Finalizing the feature implementation

* Bugfix issue194 (#203)

* Preliminary fix for issue #194

* Added some code to take stream=tar incremental backup

* Fixing tar command to extract to backup folder

* Doing some sanity checks for disabling taking incremental backups with --stream=tar

* Added extracting from tar archive for full backup prepare

* Trying to figure out recent fail

* Preliminary fix for issue #204 (#207)

* Bugfix issue206 (#208)

* Adding general tablespace creation and altering tables

* Fixing failed thing

* Removing table encryption for general tablespace

* Bugfix issue205 (#211)

* Preliminary fix for issue #205

* Fixing sql error

* temporarily disabling the feature

* Finalizing the feature implementation

* Bugfix issue209 (#212)

* Implemented cloning pxbs  from repo for issue #209

* Added few files to build pxb locally

* Changed file permissions

* Finalizing the feature implementation for issue #209

* Trying to fix the issue with PS 5.5 cloning

* Removed unneeded test from prepare_env.bats

* Fixing config generator

* Bugfix issue213 (#214)

* Preliminary fix for issue #213

* adding BIN variable

* Added bats test also

* Bugfix issue210 (#216)

* Preliminary fix for issue #210

* Trying to fix --show_tags output

* adjusting the output

* again trying to adjust the output

* nearly done

* Added completion time

* Adding backup size to backup_tags.txt

* Showing tags + size

* Added more padding

* adjusting output

* adjusting output again

* fixing some things

* Fixing the issue with staticmethod

* Bugfix issue217 (#218)

* Preliminary fix for issue #217

* Fixing the package data to include .sh files

* Added some dependency package installation

* Bugfix issue219 (#221)

* Preliminary fix for issue #219

* Added new for loop

* removing generalt tablespace file outside the datadir each time before backup and after backup actions.

* Making slave2 to remove general tablespace then to copy-back

* creating general tablespace with relative path

* Fixing the simple code issue

* Disabled due to -> https://bugs.launchpad.net/percona-xtrabackup/+bug/1736380

* Bugfix issue220 (#224)

* Preliminary fix for issue #220

* Adding sleep queries to be executed

* fixin max_workers size

* trying to run sleep queries concurrently

* added u+x for file

* Changed to run prior taking the backup

* changed the place of runner for creating sleep queries

* killing select queries immediately

* Adding some new things for tests.

* Trying to spof the reason of the fail

* trying to spot is sqls are run properly

* Do not redirect to /dev/null for now

* added some more options

* changed the position of parallel query run

* increased the value of benchmark

* changed a bit benchmark time

* Added "--ftwrl-wait-timeout=0 " options

* added all kind of options for checking locking

* playing with option values

* Now it should work

* Added --no-backup-locks

* Changed execution a bit

* increasing query size

* Finalizing the feature implementation

* Testing backup lock killing queries

* Trying to enable backup locks

* Added options to PS 5.6 as well

* Finalizing the feature implementation

* Preliminary fix for issue #225 (#226)

* Preliminary fix for issue #229 (#230)

* Bugfix issue222 (#232)

* Preliminary fix for issue #222

* fixing syntax error N1

* Removing the folder prior creating table

* Finalizing the feature implementation for issue #222

* Bugfix issue231 (#234)

* Preliminary fix for issue #231

* Fixing test

* Missed split()

* Preliminary fix for issue #233 (#236)

* Fixed #228 (#237)

* Preliminary fix for issue #223 (#238)

* Preliminary fix for issue #215 (#239)

* Version Bump
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ShahriyarR