pt工具的使用（4） pt-osc的使用

一、参数详解

--alter = s 模式修改，不带ALTER TABLE关键字--alter-foreign-keys-method = s 如何修改外键以便它们引用新表--[no]analyze-before-swap 在新表上执行ANALYZE TABLEANALYZE TABLE与旧的交换之前（默认yes）--channel = s连接到服务器时使用的通道名称,使用复制渠道--[no] check-alter解析指定的--alter并尝试警告可能的意外行为（默认yes）--check-interval = m 检查--max-lag（默认值1）。可选后缀s =秒，m =分钟，h =小时，d =天；如果没有后缀，则使用s。--[no] check-plan 检查查询执行计划的安全性（默认yes）--[no] check-replication-filters 如果任何服务器设置了复制过滤器，则中止（默认为yes）--check-slave-lag = s 暂停数据拷贝，直到此副本的延迟小于--max-lag--[no] check-unique-key-change 避免pt-online-schema-change运行，如果--alter的指定语句试图添加唯一索引（默认为yes）--chunk-index = s对于分块表，首选此索引--chunk-index-columns = i 仅使用-块索引--chunk-size = z 为每个拷贝块选择的行数（默认为1000）--chunk-size-limit = f 不要复制比该参数还要大的块（默认为4.0）--chunk-time = f 动态调整块大小，以便每个数据复制查询需要很长时间才能执行（默认值0.5）--config = A 读取此逗号分隔的config列表文件；如果指定，则必须是第一个命令行上的选项--critical-load = A 在每次执行后检查SHOW GLOBAL STATUS块，如果负载太高则中止（默认Threads_running = 50） --data-dir = s 在另一个数据目录创建新表使用DATA DIRECTORY功能进行分区--defaults-file = s -F 仅从给定文件读取mysql选项--[no] drop-new-table 如果复制原始表失败，则删除新表（默认为yes）--[no] drop-old-table 重命名原始表后将其删除（默认yes）--[no] drop-triggers 在旧表上删除触发器。 --no-drop-triggers强制--no-drop-old-table（默认yes）--dry-run 创建和更改新表，但不要创建触发器，复制数据或替换原始表--execute 表示您已阅读文档并想要更改表格--force该选项绕过确认以防万一使用alter-foreign-keys-method = none的方法这可能会打破外键约束--max-flow-ctl = f类似于--max-lag，但用于PXC集群--max-lag = m暂停数据副本，直到所有副本的延迟小于此值（默认为1s）。可选后缀s =秒，m =分钟，h =小时，d =天；如果没有后缀，则使用s。--max-load = A 每一块执行之后检查SHOW GLOBAL STATUS，如果有任何状态变量，则暂停高于其阈值（默认线程数= 25）--new-table-name = s 交换之前的新表名。 ％T是替换为原始表名（默认为％T_new）--null-to-not-null允许修改NULL列为NOT NULL列--only-same-schema-fks 仅在与原始表具有相同的库的表上检查外键--pause-file = s 此参数指定的文件存在时暂停执行--plugin = s Perl模块文件，该文件定义了pt_online_schema_change_plugin类别--preserve-triggers 指定时保留旧触发器--print将SQL语句打印到STDOUT--progress = a 复制行时将进度报告打印到STDERR（默认时间30）--quiet -q 不将消息打印到STDOUT（禁用--progress）--recurse = i发现副本时层次结构中要递归的级别数--recursion-method = a 用于发现副本的首选递归方法（默认进程列表，主机）--remove-data-dir如果原始表是使用数据目录功能，将其删除并在MySQL默认目录中创建新表不创建新的isl文件（默认否）--set-vars = A 在此设置MySQL变量-变量=值对的列表，以逗号分隔--skip-check-slave-lag = d 在从属滞后时跳过DSN检查--sleep = f 复制每个块后的睡眠时间（以秒为单位）（默认为0）--statistics 打印有关内部计数器的统计信息--[no] swap-tables已更改表之后，交换原始表和新的表，（默认为yes）--tries = a 尝试几次关键操作

二、原理

pt-osc之工作流程：

1、检查更改表是否有主键或唯一索引，是否有触发器

2、检查修改表的表结构，创建一个临时表，在新表上执行ALTER TABLE语句

3、在源表上创建三个触发器分别对于INSERT UPDATE DELETE操作

4、从源表拷贝数据到临时表，在拷贝过程中，对源表的更新操作会写入到新建表中

5、将临时表和源表rename（需要元数据修改锁，需要短时间锁表）

6、删除源表和触发器，完成表结构的修改。

三、限制

1、源表必须有主键或唯一索引，如果没有工具将停止工作

2、如果线上的复制环境过滤器操作过于复杂，工具将无法工作

3、如果开启复制延迟检查，但主从延迟时，工具将暂停数据拷贝工作

4、如果开启主服务器负载检查，但主服务器负载较高时，工具将暂停操作

5、当表使用外键时，如果未使用--alter-foreign-keys-method参数，工具将无法执行

6、只支持Innodb存储引擎表，且要求服务器上有该表1倍以上的空闲空间。

四、实践

简单的加索引操作

pt-online-schema-change h=127.0.0.1,P=3306,u=root,D=pt_test,t=pt_osc_1 --alter 'drop index idx_age' --ask-pass --print --execute

打印出来的结果

# 尝试连接从数据库Cannot connect to h=***.***.***.***,p=...,u=rootCannot connect to h=***.***.***.***,p=...,u=rootNo slaves found. See --recursion-method if host VM_0_14_centos has slaves.# 查看主从延迟配置项Not checking slave lag because no slaves were found and --check-slave-lag was not specified.Operation, tries, wait:analyze_table, 10, 1copy_rows, 10, 0.25create_triggers, 10, 1drop_triggers, 10, 1swap_tables, 10, 1update_foreign_keys, 10, 1Altering `pt_test`.`pt_osc_1`...# 建表Creating new table...CREATE TABLE `pt_test`.`_pt_osc_1_new` (`id` int(10) unsigned NOT NULL AUTO_INCREMENT,`name` varchar(20) DEFAULT NULL,`age` tinyint(1) DEFAULT NULL,`sex` enum('man','women') DEFAULT 'man',`info` varchar(100) DEFAULT 'have a good day',PRIMARY KEY (`id`),KEY `idx_age` (`age`)) ENGINE=InnoDB AUTO_INCREMENT=1270013 DEFAULT CHARSET=utf8Created new table pt_test._pt_osc_1_new OK.# 修改表结构Altering new table...ALTER TABLE `pt_test`.`_pt_osc_1_new` drop index idx_ageAltered `pt_test`.`_pt_osc_1_new` OK.# 建触发器-07-03T10:48:46 Creating triggers...-07-03T10:48:46 Created triggers OK.# 插入数据-07-03T10:48:46 Copying approximately 1266000 rows...INSERT LOW_PRIORITY IGNORE INTO `pt_test`.`_pt_osc_1_new` (`id`, `name`, `age`, `sex`, `info`) SELECT `id`, `name`, `age`, `sex`, `info` FROM `pt_test`.`pt_osc_1` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= ?)) AND ((`id` <= ?)) LOCK IN SHARE MODE /*pt-online-schema-change 2459 copy nibble*/SELECT /*!40001 SQL_NO_CACHE */ `id` FROM `pt_test`.`pt_osc_1` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= ?)) ORDER BY `id` LIMIT ?, 2 /*next chunk boundary*/-07-03T10:49:08 Copied rows OK.# 修复索引-07-03T10:49:08 Analyzing new table...# 切换表名-07-03T10:49:08 Swapping tables...RENAME TABLE `pt_test`.`pt_osc_1` TO `pt_test`.`_pt_osc_1_old`, `pt_test`.`_pt_osc_1_new` TO `pt_test`.`pt_osc_1`-07-03T10:49:08 Swapped original and new tables OK.# 删除旧表-07-03T10:49:08 Dropping old table...DROP TABLE IF EXISTS `pt_test`.`_pt_osc_1_old`-07-03T10:49:09 Dropped old table `pt_test`.`_pt_osc_1_old` OK.-07-03T10:49:09 Dropping triggers...# 删除触发器DROP TRIGGER IF EXISTS `pt_test`.`pt_osc_pt_test_pt_osc_1_del`DROP TRIGGER IF EXISTS `pt_test`.`pt_osc_pt_test_pt_osc_1_upd`DROP TRIGGER IF EXISTS `pt_test`.`pt_osc_pt_test_pt_osc_1_ins`-07-03T10:49:09 Dropped triggers OK.Successfully altered `pt_test`.`pt_osc_1`.

通过全局日志查看详细过程

# 链接-07-03T10:48:46.466411+08:00 370 Connectroot@localhost on pt_test using TCP/IP# 检查并设置锁超时时间和链接超时时间以及sql_mode以及版本等信息-07-03T10:48:46.466838+08:00 370 QuerySHOW VARIABLES LIKE 'innodb\_lock_wait_timeout'-07-03T10:48:46.469033+08:00 370 QuerySET SESSION innodb_lock_wait_timeout=1-07-03T10:48:46.469204+08:00 370 QuerySHOW VARIABLES LIKE 'lock\_wait_timeout'-07-03T10:48:46.470890+08:00 370 QuerySET SESSION lock_wait_timeout=60-07-03T10:48:46.471045+08:00 370 QuerySHOW VARIABLES LIKE 'wait\_timeout'-07-03T10:48:46.472562+08:00 370 QuerySET SESSION wait_timeout=10000-07-03T10:48:46.483177+08:00 371 QuerySELECT @@server_id /*!50038 , @@hostname*/-07-03T10:48:46.483472+08:00 370 QuerySHOW VARIABLES LIKE 'wsrep_on'-07-03T10:48:46.485135+08:00 370 QuerySHOW VARIABLES LIKE 'version%'-07-03T10:48:46.486782+08:00 370 QuerySHOW ENGINES-07-03T10:48:46.487253+08:00 370 QuerySHOW VARIABLES LIKE 'innodb_version'-07-03T10:48:46.500382+08:00 370 QuerySHOW VARIABLES LIKE 'innodb_stats_persistent'-07-03T10:48:46.502663+08:00 370 QuerySELECT @@SERVER_ID# 查看当前用户并链接从库-07-03T10:48:46.502875+08:00 370 QuerySHOW GRANTS FOR CURRENT_USER()-07-03T10:48:46.503084+08:00 370 QuerySHOW FULL PROCESSLIST-07-03T10:48:46.504987+08:00 372 Connectroot@***.***.***.*** on using TCP/IP-07-03T10:48:46.505017+08:00 372 ConnectAccess denied for user 'root'@'152.136.115.236' (using password: YES)-07-03T10:48:46.506965+08:00 373 Connectroot@***.***.***.*** on using TCP/IP-07-03T10:48:46.506991+08:00 373 ConnectAccess denied for user 'root'@'152.136.115.236' (using password: YES)-07-03T10:48:46.508825+08:00 374 Connectroot@***.***.***.*** on using TCP/IP-07-03T10:48:46.508875+08:00 374 ConnectAccess denied for user 'root'@'152.136.115.236' (using password: YES)-07-03T10:48:46.511866+08:00 375 Connectroot@***.***.***.*** on using TCP/IP-07-03T10:48:46.511894+08:00 375 ConnectAccess denied for user 'root'@'152.136.115.236' (using password: YES)# 查看当前线程数量，观测负载是否过高-07-03T10:48:46.512786+08:00 370 QuerySHOW GLOBAL STATUS LIKE 'Threads_running'-07-03T10:48:46.514173+08:00 370 QuerySHOW GLOBAL STATUS LIKE 'Threads_running'-07-03T10:48:46.515510+08:00 370 QuerySELECT CONCAT(@@hostname, @@port)-07-03T10:48:46.515986+08:00 370 QuerySHOW TABLES FROM `pt_test` LIKE 'pt\_osc\_1'-07-03T10:48:46.516271+08:00 370 QuerySELECT VERSION()# 查看当前表是否有触发器-07-03T10:48:46.516482+08:00 370 QuerySHOW TRIGGERS FROM `pt_test` LIKE 'pt\_osc\_1'# 查看是否存在外键-07-03T10:48:46.518743+08:00 370 QuerySELECT table_schema, table_name FROM information_schema.key_column_usage WHERE referenced_table_schema='pt_test' AND referenced_table_name='pt_osc_1'# 创建表-07-03T10:48:46.548541+08:00 370 QueryCREATE TABLE `pt_test`.`_pt_osc_1_new` (`id` int(10) unsigned NOT NULL AUTO_INCREMENT,`name` varchar(20) DEFAULT NULL,`age` tinyint(1) DEFAULT NULL,`sex` enum('man','women') DEFAULT 'man',`info` varchar(100) DEFAULT 'have a good day',PRIMARY KEY (`id`),KEY `idx_age` (`age`)) ENGINE=InnoDB AUTO_INCREMENT=1270013 DEFAULT CHARSET=utf8# 修改表结构-07-03T10:48:46.588347+08:00 370 QueryALTER TABLE `pt_test`.`_pt_osc_1_new` drop index idx_age# 检查是否有触发器（共6个，对于uodate,insert,delete的before,after的触发器）-07-03T10:48:46.606298+08:00 370 QuerySELECT TRIGGER_SCHEMA, TRIGGER_NAME, DEFINER, ACTION_STATEMENT, SQL_MODE, CHARACTER_SET_CLIENT, COLLATION_CONNECTION, EVENT_MANIPULATION, ACTION_TIMING FROM INFORMATION_SCHEMA.TRIGGERS WHERE EVENT_MANIPULATION = 'DELETE' AND ACTION_TIMING = 'AFTER' AND TRIGGER_SCHEMA = 'pt_test' AND EVENT_OBJECT_TABLE = 'pt_osc_1'-07-03T10:48:46.607263+08:00 370 QuerySELECT TRIGGER_SCHEMA, TRIGGER_NAME, DEFINER, ACTION_STATEMENT, SQL_MODE, CHARACTER_SET_CLIENT, COLLATION_CONNECTION, EVENT_MANIPULATION, ACTION_TIMING FROM INFORMATION_SCHEMA.TRIGGERS WHERE EVENT_MANIPULATION = 'UPDATE' AND ACTION_TIMING = 'AFTER' AND TRIGGER_SCHEMA = 'pt_test' AND EVENT_OBJECT_TABLE = 'pt_osc_1'-07-03T10:48:46.608132+08:00 370 QuerySELECT TRIGGER_SCHEMA, TRIGGER_NAME, DEFINER, ACTION_STATEMENT, SQL_MODE, CHARACTER_SET_CLIENT, COLLATION_CONNECTION, EVENT_MANIPULATION, ACTION_TIMING FROM INFORMATION_SCHEMA.TRIGGERS WHERE EVENT_MANIPULATION = 'INSERT' AND ACTION_TIMING = 'AFTER' AND TRIGGER_SCHEMA = 'pt_test' AND EVENT_OBJECT_TABLE = 'pt_osc_1'-07-03T10:48:46.608877+08:00 370 QuerySELECT TRIGGER_SCHEMA, TRIGGER_NAME, DEFINER, ACTION_STATEMENT, SQL_MODE, CHARACTER_SET_CLIENT, COLLATION_CONNECTION, EVENT_MANIPULATION, ACTION_TIMING FROM INFORMATION_SCHEMA.TRIGGERS WHERE EVENT_MANIPULATION = 'DELETE' AND ACTION_TIMING = 'BEFORE' AND TRIGGER_SCHEMA = 'pt_test' AND EVENT_OBJECT_TABLE = 'pt_osc_1'-07-03T10:48:46.609648+08:00 370 QuerySELECT TRIGGER_SCHEMA, TRIGGER_NAME, DEFINER, ACTION_STATEMENT, SQL_MODE, CHARACTER_SET_CLIENT, COLLATION_CONNECTION, EVENT_MANIPULATION, ACTION_TIMING FROM INFORMATION_SCHEMA.TRIGGERS WHERE EVENT_MANIPULATION = 'UPDATE' AND ACTION_TIMING = 'BEFORE' AND TRIGGER_SCHEMA = 'pt_test' AND EVENT_OBJECT_TABLE = 'pt_osc_1'-07-03T10:48:46.610366+08:00 370 QuerySELECT TRIGGER_SCHEMA, TRIGGER_NAME, DEFINER, ACTION_STATEMENT, SQL_MODE, CHARACTER_SET_CLIENT, COLLATION_CONNECTION, EVENT_MANIPULATION, ACTION_TIMING FROM INFORMATION_SCHEMA.TRIGGERS WHERE EVENT_MANIPULATION = 'INSERT' AND ACTION_TIMING = 'BEFORE' AND TRIGGER_SCHEMA = 'pt_test' AND EVENT_OBJECT_TABLE = 'pt_osc_1'创建触发器-07-03T10:48:46.611152+08:00 370 QueryCREATE TRIGGER `pt_osc_pt_test_pt_osc_1_del` AFTER DELETE ON `pt_test`.`pt_osc_1` FOR EACH ROW DELETE IGNORE FROM `pt_test`.`_pt_osc_1_new` WHERE `pt_test`.`_pt_osc_1_new`.`id` <=> OLD.`id`-07-03T10:48:46.620854+08:00 370 QueryCREATE TRIGGER `pt_osc_pt_test_pt_osc_1_upd` AFTER UPDATE ON `pt_test`.`pt_osc_1` FOR EACH ROW BEGIN DELETE IGNORE FROM `pt_test`.`_pt_osc_1_new` WHERE !(OLD.`id` <=> NEW.`id`) AND `pt_test`.`_pt_osc_1_new`.`id` <=> OLD.`id`;REPLACE INTO `pt_test`.`_pt_osc_1_new` (`id`, `name`, `age`, `sex`, `info`) VALUES (NEW.`id`, NEW.`name`, NEW.`age`, NEW.`sex`, NEW.`info`);END-07-03T10:48:46.630283+08:00 370 QueryCREATE TRIGGER `pt_osc_pt_test_pt_osc_1_ins` AFTER INSERT ON `pt_test`.`pt_osc_1` FOR EACH ROW REPLACE INTO `pt_test`.`_pt_osc_1_new` (`id`, `name`, `age`, `sex`, `info`) VALUES (NEW.`id`, NEW.`name`, NEW.`age`, NEW.`sex`, NEW.`info`)检查每一个数据块的起始id-07-03T10:48:46.638632+08:00 370 QueryEXPLAIN SELECT * FROM `pt_test`.`pt_osc_1` WHERE 1=1-07-03T10:48:46.640148+08:00 370 QuerySELECT /*!40001 SQL_NO_CACHE */ `id` FROM `pt_test`.`pt_osc_1` FORCE INDEX(`PRIMARY`) ORDER BY `id` LIMIT 1 /*first lower boundary*/-07-03T10:48:46.640758+08:00 370 QuerySELECT /*!40001 SQL_NO_CACHE */ `id` FROM `pt_test`.`pt_osc_1` FORCE INDEX (`PRIMARY`) WHERE `id` IS NOT NULL ORDER BY `id` LIMIT 1 /*key_len*/-07-03T10:48:46.641088+08:00 370 QueryEXPLAIN SELECT /*!40001 SQL_NO_CACHE */ * FROM `pt_test`.`pt_osc_1` FORCE INDEX (`PRIMARY`) WHERE `id` >= '1' /*key_len*/-07-03T10:48:46.641686+08:00 370 QueryEXPLAIN SELECT /*!40001 SQL_NO_CACHE */ `id` FROM `pt_test`.`pt_osc_1` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '1')) ORDER BY `id` LIMIT 999, 2 /*next chunk boundary*/-07-03T10:48:46.64+08:00 370 QuerySELECT /*!40001 SQL_NO_CACHE */ `id` FROM `pt_test`.`pt_osc_1` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '1')) ORDER BY `id` LIMIT 999, 2 /*next chunk boundary*/# 按数据块插入数据-07-03T10:48:46.642968+08:00 370 QueryEXPLAIN SELECT `id`, `name`, `age`, `sex`, `info` FROM `pt_test`.`pt_osc_1` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '1')) AND ((`id` <= '1000')) LOCK IN SHARE MODE /*explain pt-online-schema-change 2459 copy nibble*/-07-03T10:48:46.643469+08:00 370 QueryINSERT LOW_PRIORITY IGNORE INTO `pt_test`.`_pt_osc_1_new` (`id`, `name`, `age`, `sex`, `info`) SELECT `id`, `name`, `age`, `sex`, `info` FROM `pt_test`.`pt_osc_1` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '1')) AND ((`id` <= '1000')) LOCK IN SHARE MODE /*pt-online-schema-change 2459 copy nibble*/-07-03T10:48:46.666795+08:00 370 QuerySHOW WARNINGS-07-03T10:48:46.667272+08:00 370 QuerySHOW GLOBAL STATUS LIKE 'Threads_running'-07-03T10:48:46.668973+08:00 370 QueryEXPLAIN SELECT /*!40001 SQL_NO_CACHE */ `id` FROM `pt_test`.`pt_osc_1` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '1001')) ORDER BY `id` LIMIT 21480, 2 /*next chunk boundary*/-07-03T10:48:46.670148+08:00 370 QuerySELECT /*!40001 SQL_NO_CACHE */ `id` FROM `pt_test`.`pt_osc_1` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '1001')) ORDER BY `id` LIMIT 21480, 2 /*next chunk boundary*/-07-03T10:48:46.711830+08:00 370 QueryEXPLAIN SELECT `id`, `name`, `age`, `sex`, `info` FROM `pt_test`.`pt_osc_1` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '1001')) AND ((`id` <= '22481')) LOCK IN SHARE MODE /*explain pt-online-schema-change 2459 copy nibble*/-07-03T10:48:46.712513+08:00 370 QueryINSERT LOW_PRIORITY IGNORE INTO `pt_test`.`_pt_osc_1_new` (`id`, `name`, `age`, `sex`, `info`) SELECT `id`, `name`, `age`, `sex`, `info` FROM `pt_test`.`pt_osc_1` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '1001')) AND ((`id` <= '22481')) LOCK IN SHARE MODE /*pt-online-schema-change 2459 copy nibble*/-07-03T10:49:08.847235+08:00 370 QuerySHOW WARNINGS-07-03T10:49:08.847791+08:00 370 QuerySHOW GLOBAL STATUS LIKE 'Threads_running'# 修复索引-07-03T10:49:08.849748+08:00 370 QueryANALYZE TABLE `pt_test`.`_pt_osc_1_new` /* pt-online-schema-change */# 重命名表-07-03T10:49:08.864306+08:00 370 QueryRENAME TABLE `pt_test`.`pt_osc_1` TO `pt_test`.`_pt_osc_1_old`, `pt_test`.`_pt_osc_1_new` TO `pt_test`.`pt_osc_1`# 删除表及触发器-07-03T10:49:08.939689+08:00 370 QueryDROP TABLE IF EXISTS `pt_test`.`_pt_osc_1_old`-07-03T10:49:09.114312+08:00 370 QueryDROP TRIGGER IF EXISTS `pt_test`.`pt_osc_pt_test_pt_osc_1_del`-07-03T10:49:09.114661+08:00 370 QueryDROP TRIGGER IF EXISTS `pt_test`.`pt_osc_pt_test_pt_osc_1_upd`-07-03T10:49:09.114900+08:00 370 QueryDROP TRIGGER IF EXISTS `pt_test`.`pt_osc_pt_test_pt_osc_1_ins`-07-03T10:49:09.17+08:00 370 QuerySHOW TABLES FROM `pt_test` LIKE '\_pt\_osc\_1\_new'-07-03T10:49:09.123379+08:00 371 Quit-07-03T10:49:09.123761+08:00 370 Quit

五、低版本pt-osc遇到的问题及解决方式

1.对主从云数据库做修改时会找不到从机

--noversion-check --nocheck-replication-filters --recursion-method='none' # 加上如下几个参数

2.数据量较大时候出现copy未完成就报错

sudo vim /usr/bin/pt-online-schema-change注释掉 printf STDERR "$name: %3d%% %s remain\n",（第3595行）

如果觉得《pt工具的使用（4） pt-osc的使用》对你有帮助，请点赞、收藏，并留下你的观点哦！