ITPub博客
首页 博客 专栏 学院 社区 IT新媒体评选
发博

首页 > 数据库 > MySQL > 使用Vitess构建数据管道 — PlanetScale

使用Vitess构建数据管道 — PlanetScale

MySQL 作者：ITPUB社群 时间：1970-01-01 08:33:44 0 删除 编辑


Vitess 是一个广受欢迎的 CNCF 项目，被 Slack、Square、Shopify 和 GitHub 等公司用于扩展全球最大规模的 MySQL 部署。它提供了分片、连接池以及许多其他功能，使 MySQL 的水平扩展变得简单易行。

Vitess 和 MySQL 非常适合用作在线事务处理（OLTP）系统——在这种系统中，最终用户直接与系统交互，快速响应时间至关重要，用户获取产品和服务信息，生成订单、用户档案等关键业务记录。它们并未针对在线分析处理（OLAP）工作负载以及随着产品、公司和数据需求增长而遇到的其他用例和需求进行优化。这正是变更数据捕获（CDC），也称为 ETL 或 Extract-Transform-Load，以及更广泛意义上的数据管道发挥作用的地方，它们允许你在各种服务于特定需求的系统之间维护同步的数据副本，其中 CDC 是一种用于跟踪数据库变更并将其传播到其他系统的技术。这对于多种用例都很有用，包括数据复制、数据仓库和数据集成。这使你能够例如维护用于分析和报告目的（如季度和年度销售报告）的数据仓库和/或数据湖，或者将数据与需要处理最初由 OLTP 系统创建和更新的数据的其他系统集成。

Vitess 原语

Vitess 拥有许多原语或构建块，使构建数据管道变得简单。这些是 VReplication 的功能，VReplication 是一个强大的系统，支持各种类型的数据复制和转换。对于 CDC 和类似用例，VReplication 在 VTGate（Vitess 网关）中提供了 VStream API（应用程序编程接口），允许你从 Vitess 集群实时流式传输变更。

这个底层的 VStream 原语随后被 Debezium 等流行的 CDC 工具使用，以捕获 Vitess 中的变更并将其传播到其他系统。PlanetScale 也使用 VStream API 来构建 Connect 功能，为 Airbyte（源）和 Fivetran（源）等流行的 CDC/ETL 服务使用额外的开源驱动程序。

深入了解 VStream 的内部机制

VStream 是一个底层组件，通过 gRPC 提供，被 VReplication 内部用于在 Vitess 中为各种工作流类型（如 MoveTables 和 Reshard）复制数据——通常从一个 VTTablet 到另一个。VTGate VStream RPC 利用这个底层组件从 Vitess Keyspace 中的分片流式传输数据，提供一个跨越逻辑数据库的单一统一变更流，该逻辑数据库可能包含数百甚至数千个分片。你可以在这里看到一个直接使用 VStream API 的简单示例客户端。

输出结果如下所示，如果你对底层细节感兴趣，可以自己运行这些命令（如果你打算使用现有的连接器/驱动程序，如 Debezium Connector for Vitess，则不需要）：

git
clone
git@github.com:vitessio/vitess.git
cd
vitess
git
checkout
main
make
build
cd
examples/local
./101_initial_cluster.sh
;
mysql
<
../common/insert_commerce_data.sql
;
./201_customer_tablets.sh
;
./202_move_tables.sh
;
./203_switch_reads.sh
;
./204_switch_writes.sh
;
./205_clean_commerce.sh
;
./301_customer_sharded.sh
;
./302_new_shards.sh
;
./303_reshard.sh
;
./304_switch_reads.sh
;
./305_switch_writes.sh
;
./306_down_shard_0.sh
;
./307_delete_shard_0.sh
go
run
vstream_client.go
# In another terminal, connecting to the VTGate that was started
for
i
in
{
1..10}
;
do
command
mysql
-
-no-defaults
-
h
127.0.0.1
-
P
15306
customer
-
e
"insert into customer (email) values ('
${
i
}
@foo.com')"
done
# Cleanup whenever you're done testing
./401_teardown.sh

VStream 客户端将输出从 VTGate 流式传输的变更，如下所示——首先对分片 customer keyspace 中 customer 表的当前状态进行快照，然后实时流式传输该表的后续变更：

go
run
vstream_client.go
[
type:BEGIN keyspace:
"customer"
shard:
"80-"
type:FIELD field_event:{table_name:
"customer.customer"
fields:{name:
"customer_id"
type:INT64 table:
"customer"
org_table:
"customer"
database:
"vt_customer"
org_name:
"customer_id"
column_length:20 charset:63 flags:53251 column_type:
"bigint"
} fields:{name:
"email"
type:VARBINARY table:
"customer"
org_table:
"customer"
database:
"vt_customer"
org_name:
"email"
column_length:128 charset:63 flags:128 column_type:
"varbinary(128)"
} keyspace:
"customer"
shard:
"80-"
enum_set_string_values:
true
} keyspace:
"customer"
shard:
"80-"
]
[
type:VGTID vgtid:{shard_gtids:{keyspace:
"customer"
shard:
"-80"
} shard_gtids:{keyspace:
"customer"
shard:
"80-"
gtid:
"MySQL56/97257a80-3e14-11ef-b24b-2eaacc019d10:1-58"
}} keyspace:
"customer"
shard:
"80-"
]
[
type:ROW row_event:{table_name:
"customer.customer"
row_changes:{after:{lengths:1 lengths:14 values:
"4dan@domain.com"
}} keyspace:
"customer"
shard:
"80-"
} keyspace:
"customer"
shard:
"80-"
type:VGTID vgtid:{shard_gtids:{keyspace:
"customer"
shard:
"-80"
} shard_gtids:{keyspace:
"customer"
shard:
"80-"
gtid:
"MySQL56/97257a80-3e14-11ef-b24b-2eaacc019d10:1-58"
table_p_ks:{table_name:
"customer"
lastpk:{fields:{name:
"customer_id"
type:INT64 charset:63 flags:53251} rows:{lengths:1 values:
"4"
}}}}} keyspace:
"customer"
shard:
"80-"
type:COMMIT keyspace:
"customer"
shard:
"80-"
]
[
type:BEGIN keyspace:
"customer"
shard:
"80-"
type:VGTID vgtid:{shard_gtids:{keyspace:
"customer"
shard:
"-80"
} shard_gtids:{keyspace:
"customer"
shard:
"80-"
gtid:
"MySQL56/97257a80-3e14-11ef-b24b-2eaacc019d10:1-58"
}} keyspace:
"customer"
shard:
"80-"
type:COMMIT keyspace:
"customer"
shard:
"80-"
]
[
type:COPY_COMPLETED keyspace:
"customer"
shard:
"80-"
]
[
type:BEGIN keyspace:
"customer"
shard:
"-80"
type:FIELD field_event:{table_name:
"customer.customer"
fields:{name:
"customer_id"
type:INT64 table:
"customer"
org_table:
"customer"
database:
"vt_customer"
org_name:
"customer_id"
column_length:20 charset:63 flags:53251 column_type:
"bigint"
} fields:{name:
"email"
type:VARBINARY table:
"customer"
org_table:
"customer"
database:
"vt_customer"
org_name:
"email"
column_length:128 charset:63 flags:128 column_type:
"varbinary(128)"
} keyspace:
"customer"
shard:
"-80"
enum_set_string_values:
true
} keyspace:
"customer"
shard:
"-80"
]
[
type:VGTID vgtid:{shard_gtids:{keyspace:
"customer"
shard:
"-80"
gtid:
"MySQL56/90a3e2d2-3e14-11ef-bb33-30b3ef9417b6:1-58"
} shard_gtids:{keyspace:
"customer"
shard:
"80-"
gtid:
"MySQL56/97257a80-3e14-11ef-b24b-2eaacc019d10:1-58"
}} keyspace:
"customer"
shard:
"-80"
]
[
type:ROW row_event:{table_name:
"customer.customer"
row_changes:{after:{lengths:1 lengths:16 values:
"1alice@domain.com"
}} keyspace:
"customer"
shard:
"-80"
} keyspace:
"customer"
shard:
"-80"
type:ROW row_event:{table_name:
"customer.customer"
row_changes:{after:{lengths:1 lengths:14 values:
"2bob@domain.com"
}} keyspace:
"customer"
shard:
"-80"
} keyspace:
"customer"
shard:
"-80"
type:ROW row_event:{table_name:
"customer.customer"
row_changes:{after:{lengths:1 lengths:18 values:
"3charlie@domain.com"
}} keyspace:
"customer"
shard:
"-80"
} keyspace:
"customer"
shard:
"-80"
type:ROW row_event:{table_name:
"customer.customer"
row_changes:{after:{lengths:1 lengths:14 values:
"5eve@domain.com"
}} keyspace:
"customer"
shard:
"-80"
} keyspace:
"customer"
shard:
"-80"
type:VGTID vgtid:{shard_gtids:{keyspace:
"customer"
shard:
"-80"
gtid:
"MySQL56/90a3e2d2-3e14-11ef-bb33-30b3ef9417b6:1-58"
table_p_ks:{table_name:
"customer"
lastpk:{fields:{name:
"customer_id"
type:INT64 charset:63 flags:53251} rows:{lengths:1 values:
"5"
}}}} shard_gtids:{keyspace:
"customer"
shard:
"80-"
gtid:
"MySQL56/97257a80-3e14-11ef-b24b-2eaacc019d10:1-58"
}} keyspace:
"customer"
shard:
"-80"
type:COMMIT keyspace:
"customer"
shard:
"-80"
]
[
type:BEGIN keyspace:
"customer"
shard:
"-80"
type:VGTID vgtid:{shard_gtids:{keyspace:
"customer"
shard:
"-80"
gtid:
"MySQL56/90a3e2d2-3e14-11ef-bb33-30b3ef9417b6:1-58"
} shard_gtids:{keyspace:
"customer"
shard:
"80-"
gtid:
"MySQL56/97257a80-3e14-11ef-b24b-2eaacc019d10:1-58"
}} keyspace:
"customer"
shard:
"-80"
type:COMMIT keyspace:
"customer"
shard:
"-80"
]
[
type:COPY_COMPLETED keyspace:
"customer"
shard:
"-80"
type:COPY_COMPLETED
]
[
type:BEGIN timestamp:1720544456 current_time:1720544456166536000 keyspace:
"customer"
shard:
"-80"
type:FIELD timestamp:1720544456 field_event:{table_name:
"customer.customer"
fields:{name:
"customer_id"
type:INT64 table:
"customer"
org_table:
"customer"
database:
"vt_customer"
org_name:
"customer_id"
column_length:20 charset:63 flags:53251 column_type:
"bigint"
} fields:{name:
"email"
type:VARBINARY table:
"customer"
org_table:
"customer"
database:
"vt_customer"
org_name:
"email"
column_length:128 charset:63 flags:128 column_type:
"varbinary(128)"
} keyspace:
"customer"
shard:
"-80"
} current_time:1720544456168488000 keyspace:
"customer"
shard:
"-80"
type:ROW timestamp:1720544456 row_event:{table_name:
"customer.customer"
row_changes:{after:{lengths:4 lengths:9 values:
"10001@foo.com"
}} keyspace:
"customer"
shard:
"-80"
flags:1} current_time:1720544456168646000 keyspace:
"customer"
shard:
"-80"
type:VGTID vgtid:{shard_gtids:{keyspace:
"customer"
shard:
"-80"
gtid:
"MySQL56/90a3e2d2-3e14-11ef-bb33-30b3ef9417b6:1-59"
} shard_gtids:{keyspace:
"customer"
shard:
"80-"
gtid:
"MySQL56/97257a80-3e14-11ef-b24b-2eaacc019d10:1-58"
}} keyspace:
"customer"
shard:
"-80"
type:COMMIT timestamp:1720544456 current_time:1720544456168652000 keyspace:
"customer"
shard:
"-80"
]
[
type:BEGIN timestamp:1720544456 current_time:1720544456182035000 keyspace:
"customer"
shard:
"80-"
type:FIELD timestamp:1720544456 field_event:{table_name:
"customer.customer"
fields:{name:
"customer_id"
type:INT64 table:
"customer"
org_table:
"customer"
database:
"vt_customer"
org_name:
"customer_id"
column_length:20 charset:63 flags:53251 column_type:
"bigint"
} fields:{name:
"email"
type:VARBINARY table:
"customer"
org_table:
"customer"
database:
"vt_customer"
org_name:
"email"
column_length:128 charset:63 flags:128 column_type:
"varbinary(128)"
} keyspace:
"customer"
shard:
"80-"
} current_time:1720544456183630000 keyspace:
"customer"
shard:
"80-"
type:ROW timestamp:1720544456 row_event:{table_name:
"customer.customer"
row_changes:{after:{lengths:4 lengths:9 values:
"10012@foo.com"
}} keyspace:
"customer"
shard:
"80-"
flags:1} current_time:1720544456183642000 keyspace:
"customer"
shard:
"80-"
type:VGTID vgtid:{shard_gtids:{keyspace:
"customer"
shard:
"-80"
gtid:
"MySQL56/90a3e2d2-3e14-11ef-bb33-30b3ef9417b6:1-59"
} shard_gtids:{keyspace:
"customer"
shard:
"80-"
gtid:
"MySQL56/97257a80-3e14-11ef-b24b-2eaacc019d10:1-59"
}} keyspace:
"customer"
shard:
"80-"
type:COMMIT timestamp:1720544456 current_time:1720544456183649000 keyspace:
"customer"
shard:
"80-"
]
[
type:BEGIN timestamp:1720544456 current_time:1720544456197796000 keyspace:
"customer"
shard:
"-80"
type:ROW timestamp:1720544456 row_event:{table_name:
"customer.customer"
row_changes:{after:{lengths:4 lengths:9 values:
"10023@foo.com"
}} keyspace:
"customer"
shard:
"-80"
flags:1} current_time:1720544456197810000 keyspace:
"customer"
shard:
"-80"
type:VGTID vgtid:{shard_gtids:{keyspace:
"customer"
shard:
"-80"
gtid:
"MySQL56/90a3e2d2-3e14-11ef-bb33-30b3ef9417b6:1-60"
} shard_gtids:{keyspace:
"customer"
shard:
"80-"
gtid:
"MySQL56/97257a80-3e14-11ef-b24b-2eaacc019d10:1-59"
}} keyspace:
"customer"
shard:
"-80"
type:COMMIT timestamp:1720544456 current_time:1720544456197814000 keyspace:
"customer"
shard:
"-80"
]
[
type:BEGIN timestamp:1720544456 current_time:1720544456211383000 keyspace:
"customer"
shard:
"80-"
type:ROW timestamp:1720544456 row_event:{table_name:
"customer.customer"
row_changes:{after:{lengths:4 lengths:9 values:
"10034@foo.com"
}} keyspace:
"customer"
shard:
"80-"
flags:1} current_time:1720544456211392000 keyspace:
"customer"
shard:
"80-"
type:VGTID vgtid:{shard_gtids:{keyspace:
"customer"
shard:
"-80"
gtid:
"MySQL56/90a3e2d2-3e14-11ef-bb33-30b3ef9417b6:1-60"
} shard_gtids:{keyspace:
"customer"
shard:
"80-"
gtid:
"MySQL56/97257a80-3e14-11ef-b24b-2eaacc019d10:1-60"
}} keyspace:
"customer"
shard:
"80-"
type:COMMIT timestamp:1720544456 current_time:1720544456211398000 keyspace:
"customer"
shard:
"80-"
]
[
type:BEGIN timestamp:1720544456 current_time:1720544456224248000 keyspace:
"customer"
shard:
"80-"
type:ROW timestamp:1720544456 row_event:{table_name:
"customer.customer"
row_changes:{after:{lengths:4 lengths:9 values:
"10045@foo.com"
}} keyspace:
"customer"
shard:
"80-"
flags:1} current_time:1720544456224258000 keyspace:
"customer"
shard:
"80-"
type:VGTID vgtid:{shard_gtids:{keyspace:
"customer"
shard:
"-80"
gtid:
"MySQL56/90a3e2d2-3e14-11ef-bb33-30b3ef9417b6:1-60"
} shard_gtids:{keyspace:
"customer"
shard:
"80-"
gtid:
"MySQL56/97257a80-3e14-11ef-b24b-2eaacc019d10:1-61"
}} keyspace:
"customer"
shard:
"80-"
type:COMMIT timestamp:1720544456 current_time:1720544456224261000 keyspace:
"customer"
shard:
"80-"
]
[
type:BEGIN timestamp:1720544456 current_time:1720544456237018000 keyspace:
"customer"
shard:
"80-"
type:ROW timestamp:1720544456 row_event:{table_name:
"customer.customer"
row_changes:{after:{lengths:4 lengths:9 values:
"10056@foo.com"
}} keyspace:
"customer"
shard:
"80-"
flags:1} current_time:1720544456237029000 keyspace:
"customer"
shard:
"80-"
type:VGTID vgtid:{shard_gtids:{keyspace:
"customer"
shard:
"-80"
gtid:
"MySQL56/90a3e2d2-3e14-11ef-bb33-30b3ef9417b6:1-60"
} shard_gtids:{keyspace:
"customer"
shard:
"80-"
gtid:
"MySQL56/97257a80-3e14-11ef-b24b-2eaacc019d10:1-62"
}} keyspace:
"customer"
shard:
"80-"
type:COMMIT timestamp:1720544456 current_time:1720544456237031000 keyspace:
"customer"
shard:
"80-"
]
[
type:BEGIN timestamp:1720544456 current_time:1720544456249777000 keyspace:
"customer"
shard:
"80-"
type:ROW timestamp:1720544456 row_event:{table_name:
"customer.customer"
row_changes:{after:{lengths:4 lengths:9 values:
"10067@foo.com"
}} keyspace:
"customer"
shard:
"80-"
flags:1} current_time:1720544456250142000 keyspace:
"customer"
shard:
"80-"
type:VGTID vgtid:{shard_gtids:{keyspace:
"customer"
shard:
"-80"
gtid:
"MySQL56/90a3e2d2-3e14-11ef-bb33-30b3ef9417b6:1-60"
} shard_gtids:{keyspace:
"customer"
shard:
"80-"
gtid:
"MySQL56/97257a80-3e14-11ef-b24b-2eaacc019d10:1-63"
}} keyspace:
"customer"
shard:
"80-"
type:COMMIT timestamp:1720544456 current_time:1720544456250150000 keyspace:
"customer"
shard:
"80-"
]
[
type:BEGIN timestamp:1720544456 current_time:1720544456263391000 keyspace:
"customer"
shard:
"80-"
type:ROW timestamp:1720544456 row_event:{table_name:
"customer.customer"
row_changes:{after:{lengths:4 lengths:9 values:
"10078@foo.com"
}} keyspace:
"customer"
shard:
"80-"
flags:1} current_time:1720544456263407000 keyspace:
"customer"
shard:
"80-"
type:VGTID vgtid:{shard_gtids:{keyspace:
"customer"
shard:
"-80"
gtid:
"MySQL56/90a3e2d2-3e14-11ef-bb33-30b3ef9417b6:1-60"
} shard_gtids:{keyspace:
"customer"
shard:
"80-"
gtid:
"MySQL56/97257a80-3e14-11ef-b24b-2eaacc019d10:1-64"
}} keyspace:
"customer"
shard:
"80-"
type:COMMIT timestamp:1720544456 current_time:1720544456263411000 keyspace:
"customer"
shard:
"80-"
]
[
type:BEGIN timestamp:1720544456 current_time:1720544456276388000 keyspace:
"customer"
shard:
"-80"
type:ROW timestamp:1720544456 row_event:{table_name:
"customer.customer"
row_changes:{after:{lengths:4 lengths:9 values:
"10089@foo.com"
}} keyspace:
"customer"
shard:
"-80"
flags:1} current_time:1720544456276398000 keyspace:
"customer"
shard:
"-80"
type:VGTID vgtid:{shard_gtids:{keyspace:
"customer"
shard:
"-80"
gtid:
"MySQL56/90a3e2d2-3e14-11ef-bb33-30b3ef9417b6:1-61"
} shard_gtids:{keyspace:
"customer"
shard:
"80-"
gtid:
"MySQL56/97257a80-3e14-11ef-b24b-2eaacc019d10:1-64"
}} keyspace:
"customer"
shard:
"-80"
type:COMMIT timestamp:1720544456 current_time:1720544456276402000 keyspace:
"customer"
shard:
"-80"
]
[
type:BEGIN timestamp:1720544456 current_time:1720544456289697000 keyspace:
"customer"
shard:
"-80"
type:ROW timestamp:1720544456 row_event:{table_name:
"customer.customer"
row_changes:{after:{lengths:4 lengths:10 values:
"100910@foo.com"
}} keyspace:
"customer"
shard:
"-80"
flags:1} current_time:1720544456289711000 keyspace:
"customer"
shard:
"-80"
type:VGTID vgtid:{shard_gtids:{keyspace:
"customer"
shard:
"-80"
gtid:
"MySQL56/90a3e2d2-3e14-11ef-bb33-30b3ef9417b6:1-62"
} shard_gtids:{keyspace:
"customer"
shard:
"80-"
gtid:
"MySQL56/97257a80-3e14-11ef-b24b-2eaacc019d10:1-64"
}} keyspace:
"customer"
shard:
"-80"
type:COMMIT timestamp:1720544456 current_time:1720544456289714000 keyspace:
"customer"
shard:
"-80"
]

如果你对更多底层细节感兴趣，可以查看 VStream API 文档。

一个示例设置

你可以使用与这里描述的类似设置，但使用 Debezium Connector for Vitess 而不是 Debezium Connector for MySQL，并使用 AWS RedShift 实例而不是 PostgreSQL 作为目标（RedShift 基于 PostgreSQL）。这也展示了建立这类系统时的一般规则：你应该使用连接器/驱动的 Vitess 变体，而不是 MySQL 变体——其他方面则相同。

转载声明：本文为转载整理，原文版权归原作者及原站点所有。

原文来源：PlanetScale Blog

原创地址：https://planetscale.com/blog/building-data-pipelines-with-vitess

原文作者：PlanetScale Blog；如有侵权，请联系管理员删除。

0
0
上一篇： PB级零停机迁移 — PlanetScale
下一篇： PlanetScale 如何进行架构变更 — PlanetScale
请登录后发表评论 登录
全部评论
ITPUB社群

注册时间：2022-11-21

  • 博文量
    1090
  • 访问量
    546623

最新文章

支持我们 作者招募 用户协议 FAQ Contact Us

北京盛拓优讯信息技术有限公司. 版权所有  京ICP备16024965号-8 北京市公安局海淀分局网监中心备案编号：11010802021510 niuxiaotong@pcpop.com 17352615567 未成年人举报专区

广播电视节目制作经营许可证(京) 字第1234号 中国互联网协会会员

违法和不良信息举报电话:18101376593