从 Oracle 迁移到 Greenplum,不是简单的“建表、导数据”,而是一次 架构升级 。
Oracle :行存、OLTP 优化、强一致性
Greenplum :列存、MPP 并行、OLAP 优化
📌 迁移的核心策略: 数据模型重构 + SQL 语法转换 + 性能重新设计 。直接“搬过去”大概率跑得慢。
|
|
|
|
|
|---|---|---|---|
| 架构 |
|
|
|
| 存储 |
|
|
|
| 索引 |
|
|
|
| 并行能力 |
|
|
|
| 分区 |
|
|
|
| 统计信息 |
|
|
|
| 序列 |
|
|
|
|
|
|
|
|---|---|---|
| 表结构 |
|
|
| 存储估算 |
|
|
| SQL 清单 |
|
|
| 依赖关系 |
|
|
-- Oracle 端:统计表和大小
SELECT owner, table_name, num_rows, avg_row_len,
(avg_row_len * num_rows / 1024/1024/1024) as size_gb
FROM dba_tables
WHERE owner = 'ORDERS'
ORDER BY size_gb DESC;
Oracle → Greenplum 数据类型映射表:
|
|
|
|
|---|---|---|
NUMBER(p,s) |
numeric(p,s) |
|
VARCHAR2(n) |
varchar(n) |
|
DATE |
timestamp |
|
TIMESTAMP |
timestamp |
|
CLOB |
text |
|
BLOB |
bytea |
|
ROWID |
|
|
-- Oracle 原表
CREATE TABLE orders_oracle (
order_id NUMBER(12) PRIMARY KEY,
user_id NUMBER(10),
amount NUMBER(10,2),
status VARCHAR2(20),
order_date DATE
);
-- Greenplum 目标表(优化后)
CREATE TABLE orders_gp (
order_id bigint,
user_id int,
amount numeric(10,2),
status varchar(20),
order_date timestamp
) WITH (appendonly=true, orientation=column, compresstype=zstd)
DISTRIBUTED BY (user_id) -- 按 user_id 分布
PARTITION BY RANGE (order_date); -- 按日期分区
常用转换对照表:
|
|
|
|---|---|
ROWNUM <= N |
LIMIT N |
SYSDATE |
now()
current_timestamp |
TO_DATE('2025-01-01','YYYY-MM-DD') |
'2025-01-01'::date |
NVL(col, default) |
coalesce(col, default) |
DECODE(col, v1, r1, v2, r2, default) |
CASE WHEN ... |
CONNECT BY
|
|
MERGE INTO |
INSERT ... ON CONFLICT |
SELECT * FROM DUAL |
|
迁移方案对比:
|
|
|
|
|
|---|---|---|---|
| gpfdist + 外部表 |
|
|
|
| COPY TO/FROM |
|
|
|
| Oracle 网关 + INSERT |
|
|
|
| 第三方 ETL 工具 |
|
|
|
# 推荐方案:Oracle → 导出 CSV → gpfdist 加载
# 1. Oracle 导出 CSV(使用 spool 或 Oracle SQL Developer)
sqlplus orders/xxx @export_orders.sql
# 2. 启动 gpfdist
gpfdist -d /data/export -p 8081 &
# 3. Greenplum 创建外部表
CREATE READABLE EXTERNAL TABLE ext_orders (LIKE orders_gp)
LOCATION ('gpfdist://gp-master:8081/orders_*.csv')
FORMAT 'CSV';
# 4. 并行加载
INSERT INTO orders_gp SELECT * FROM ext_orders;
-- 1. 行数对比
SELECT 'Oracle' as source, count(*) FROM oracle_orders
UNION ALL
SELECT 'Greenplum' as source, count(*) FROM orders_gp;
-- 2. 关键字段汇总对比
SELECT 'Oracle' as source, sum(amount), avg(amount), min(order_date), max(order_date)
FROM oracle_orders
UNION ALL
SELECT 'Greenplum' as source, sum(amount), avg(amount), min(order_date), max(order_date)
FROM orders_gp;
-- 3. 抽样数据对比
SELECT * FROM oracle_orders WHERE order_id IN (1,100,1000,10000,100000)
UNION ALL
SELECT * FROM orders_gp WHERE order_id IN (1,100,1000,10000,100000)
ORDER BY order_id, source;
迁移后常见性能问题及解决方案:
|
|
|
|
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
背景 :
Oracle 两节点 RAC,核心订单表 50 亿行
每日 ETL 跑 8 小时,报表查询 P99 > 30 秒
迁移步骤 :
|
|
|
|
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
迁移效果 :
|
|
|
|
|
|---|---|---|---|
|
|
|
|
70% ↓ |
|
|
|
|
81% ↓ |
|
|
|
|
96% ↓ |
|
|
|
|
98% ↓ |
|
|
|
|
10 倍 ↑ |
Oracle 迁移到 Greenplum,核心原则:
|
|
|
|---|---|
| 模型重构 |
|
| 分布键设计 |
|
| SQL 重写 |
|
| 增量并行 |
|
| 验证充分 |
|
迁移不是搬运,是升级。改好了性能翻倍,改坏了不如不迁。
来自 “ ITPUB博客 ” ,链接:https://blog.itpub.net/70051295/viewspace-3185894/ ,如需转载,请注明出处,否则将追究法律责任。