首页 > 数据库 > 数据库开发技术 > 关于烤面包的方方面面
PostgreSQL 中的 TOAST 是一项十分有趣的技术,又名"烤面包",但是网上把 TOAST 讲解地明明白白的文章少之又少,都是互相摘录,赶巧上周五也出了一个 TOAST 相关的问题,所以借此机会,好好与各位唠唠这个烤面包的细枝末节。
先容我插播一条重要广告😊 17号我会联合分会再次在成都举办一场 PG 沙龙 ~ 欢迎各位老铁的到来,我又要上台当蹩脚主持人了

每一条元组都有元组头(tuple header),用于存储关于这一条元组的一些信息,比如我们所熟知的 xmin/xmax,以及 infomask 和 t_bits 空值位图,再加上 padding,一般是 24 个字节起步,因此要比 Oracle 要大得多

那么一个字段会消耗多大的空间呢?这块在官网上有所说明 https://www.postgresql.org/docs/current/datatype-numeric.html#DATATYPE-INT。以常见的数字类型为例,smallint 占 2 字节,bigint 占 8 个字节

我们可以使用 row() 函数查看字段占据多大的空间,什么都不带的话就是 TupleHeader 的长度:
postgres=# select pg_column_size (row()); ---TupleHeader长度
pg_column_size
----------------
24
(1 row)
postgres=# select pg_column_size (row(0::smallint)); ---smallint的长度
pg_column_size
----------------
26
(1 row)
但是有些类型其并没有固定的长度,比如 text 变长类型,插入 'hello' 这五个字符在 UTF8 编码下就是 5 个字节
postgres=# select pg_column_size (row('hello'::text)) - 24 as size;
size
------
6
(1 row)
postgres=# select pg_column_size (row('hello'::char(10))) - 24 as size;
size
------
11
(1 row)
postgres=# select pg_column_size ( row(1.1::numeric) ) - 24 as size;
size
------
7
(1 row)
那为何第一条数据返回的是 6 个字节呢?先卖个关子。假如这个时候插入了一个巨长的字段该怎么办?比如 10KB 的数据,PostgreSQL 会如何处理?
不同于 Oracle 的 row chaining,PostgreSQL 采用了名为 TOAST 的技术——直译过来就是烤面包,TOAST 是 "The Oversized-Attribute Storage Technique"(超尺寸属性存储技术)的缩写,PostgreSQL 不允许一行数据跨页存储,那么对于超长的行数据就会启动 TOAST,将大的字段压缩或切片成多个物理行存到另一张系统表中(TOAST表)。当然只有特定的数据类型支持 TOAST,因为那些整数、浮点数等不太长的数据类型是没有必要使用 TOAST 的。
每个表字段支持四种 TOAST 策略:

我们可以通过 alter table xxx 按需调整 TOAST 策略,比如你提前已经知晓该数据不能被压缩,比如 img、pdf 等,那么你可以设置为 external:
postgres=# alter table t alter column a set storage plain;
ALTER TABLE
postgres=# alter table t alter column b set storage extended ;
ALTER TABLE
postgres=# select attname, atttypid::regtype,
case attstorage when 'p' then 'plain'
when 'e' then 'external'
when 'm' then 'main'
when 'x' then 'extended'
end AS strategy
from pg_attribute
where attrelid = 't'::regclass and attnum > 0;
attname | atttypid | strategy
---------+----------+----------
a | text | plain
b | text | extended
(2 rows)
关于压缩,在 14 引入了 lz4 新的压缩算法(Good tradeoff between compression speed and compression ratio),在以前仅支持 pglz。
那么具体多大的阈值会触发 TOAST 呢?代码在 heaptoast.h 中
/*
* These symbols control toaster activation. If a tuple is larger than
* TOAST_TUPLE_THRESHOLD, we will try to toast it down to no more than
* TOAST_TUPLE_TARGET bytes through compressing compressible fields and
* moving EXTENDED and EXTERNAL data out-of-line.
*
如果一条元组大小超过了TOAST_TUPLE_THRESHOLD,那么会通过压缩以及行外存储来控制这条
元组大小不超过TOAST_TUPLE_TARGET
* The numbers need not be the same, though they currently are. It doesn't
* make sense for TARGET to exceed THRESHOLD, but it could be useful to make
* it be smaller.
这些数字不需要相同,尽管它们目前是相同的。TARGET超过THRESHOLD是没有意义的,但让它变得
更小可能会有帮助。
*
* Currently we choose both values to match the largest tuple size for which
* TOAST_TUPLES_PER_PAGE tuples can fit on a heap page.
目前,我们选择这两个值来匹配TOAST_TUPLES_PER_PAGE元组在一个堆页上可以容纳的最大元组大小。
*
* XXX while these can be modified without initdb, some thought needs to be
* given to needs_toast_table() in toasting.c before unleashing random
* changes. Also see LOBLKSIZE in large_object.h, which can *not* be
* changed without initdb.
*/
#define TOAST_TUPLES_PER_PAGE 4
#define TOAST_TUPLE_THRESHOLD MaximumBytesPerTuple(TOAST_TUPLES_PER_PAGE)
#define TOAST_TUPLE_TARGET TOAST_TUPLE_THRESHOLD
/*
* Find the maximum size of a tuple if there are to be N tuples per page.
*/
如果每页有N个元组,找出元组的最大尺寸。
#define MaximumBytesPerTuple(tuplesPerPage) \
MAXALIGN_DOWN((BLCKSZ - \
MAXALIGN(SizeOfPageHeaderData + (tuplesPerPage) * sizeof(ItemIdData))) \
/ (tuplesPerPage))
MAXALIGN_DOWN((8192 - MAXALIGN(24 + 4 * 4))) / 4 ) ≈ 2KB
也就意味着默认 8 KB 的情况下,加上对齐,大小约等于 2 KB。并且可以看到,当前 toast_tuple_target 和 toast_tuple_threshold 值是一样的,我们可以表级调整,以控制元组大小不超过 toast_tuple_target。
postgres=# alter table t set (toast_tuple_target = 1800);
ALTER TABLE
postgres=# \d+ t
Table "public.t"
Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
a | text | | | | plain | | |
b | text | | | | extended | lz4 | |
Access method: heap
Options: toast_tuple_target=1800
前面介绍完了 TOAST 的诞生背景,接着再看下大概算法。代码在 toasting.c 中,由于逻辑有点冗长,这里只简述大概逻辑:
所以至此各位应该就了解了为什么 MAIN 需要理解为尽量不使用行外存储更贴切,因为迫不得己的时候还是得需要,就好比 "disable cost",设置了 enable_nestloop = off,假如优化器只能使用 nestloop,那么它还是得采用(就好比你一个人百米赛跑,无论你跑多久都是第一名)。
以上是压缩的逻辑,让我们再看下读取 TOAST 的逻辑,代码流程在 heap_tuple_untoast_attr 中
所以总结来说
让我们看个例子:
postgres=# create table toast_demo ( id int primary key, content text );
CREATE TABLE
postgres=# insert into toast_demo values (1, repeat('x',10000));
INSERT 0 1
postgres=# SELECT oid::regclass,
reltoastrelid::regclass,
pg_relation_size(reltoastrelid) AS toast_size
FROM pg_class
WHERE relkind = 'r'
AND reltoastrelid <> 0 and relname = 'toast_demo'
ORDER BY 3 DESC;
oid | reltoastrelid | toast_size
------------+-------------------------+------------
toast_demo | pg_toast.pg_toast_16453 | 0
(1 row)
可以看到这个操作并没有触发 TOAST,因为经过压缩之后,数据满足了,无需行外存储。
再插入一条长的数据试试,这次就可以看到 TOAST 表里有数据了
postgres=# WITH dummy_string AS (
postgres(# SELECT
postgres(# string_agg(md5(random()::text), '') AS dummy
postgres(# FROM
postgres(# generate_series(1, 5000))
postgres-# INSERT INTO toast_demo
postgres-# SELECT
postgres-# 2,
postgres-# dummy_string.dummy
postgres-# FROM
postgres-# dummy_string;
INSERT 0 1
postgres=# SELECT oid::regclass,
reltoastrelid::regclass,
pg_relation_size(reltoastrelid) AS toast_size
FROM pg_class
WHERE relkind = 'r'
AND reltoastrelid <> 0 and relname = 'toast_demo'
ORDER BY 3 DESC;
oid | reltoastrelid | toast_size
------------+-------------------------+------------
toast_demo | pg_toast.pg_toast_16453 | 335872
(1 row)
postgres=# select count(*) from pg_toast.pg_toast_16453;
count
-------
81
(1 row)
postgres=# select chunk_id,chunk_seq,left(chunk_data::text,10) from pg_toast.pg_toast_16453 limit 2;
chunk_id | chunk_seq | left
----------+-----------+------------
16466 | 0 | \x36633964
16466 | 1 | \x61656565
(2 rows)
前面也提及了,假如请求不需要访问 TOAST,不难想到,速度自然就快得多。此处导入一个 pdf
postgres=# create table toast_demo2 ( id int, doc bytea );
CREATE TABLE
postgres=# insert into toast_demo2 select i, :'file' from generate_series(1,50) i;
INSERT 0 50
postgres=# select pg_size_pretty(pg_relation_size('toast_demo2')); ---不包含TOAST
pg_size_pretty
----------------
8192 bytes
(1 row)
postgres=# select pg_size_pretty(pg_total_relation_size('toast_demo2')); ---包括TOAST
pg_size_pretty
----------------
478 MB
(1 row)

没错,全是 TOAST 占据的大小 👆🏻
postgres=# select id from toast_demo2;
id
----
...
49
50
(50 rows)
Time: 0.298 ms
postgres=# select * from toast_demo2; ---极其之慢
^CCancel request sent
ERROR: canceling statement due to user request
Time: 3266.191 ms (00:03.266)
这个不难想到,不查询 TOAST 当然很快,所以这也是为什么要在开发规范里写:禁止 select *,只检索需要的字段,不仅仅是节省什么资源的问题,更加深层次的原因就是这个可能的 TOAST,损失大把性能而不知。
除此之外,还要注意一个大坑——explain analyze 会误导你!
postgres=# explain analyze select * from toast_demo2; ---同样的查询,analyze却特别快
QUERY PLAN
-----------------------------------------------------------------------------------------------------------
Seq Scan on toast_demo2 (cost=0.00..22.70 rows=1270 width=36) (actual time=0.006..0.010 rows=50 loops=1)
Planning Time: 0.036 ms
Execution Time: 0.026 ms
(3 rows)
Time: 0.419 ms
DBA 表示惊呆了!😳 同样的查询,一个是几十秒返回不了,一个是 0.4 毫秒,仅仅加了一个 explain analyze,效率 N 个数量级的差异,切记。
除此之外,还有一个鲜为人知的坑——oid,感兴趣的老铁可以复现一下。

现在让我们深入到底层细节,在讲 TOAST 底层数据之前,需要先了解一个特殊的数据结构——varlena,所有的变长数据都有这样一个元组头。
All variable-length data types share the common header structure struct varlena, which includes the total length of the stored value and some flag bits. Depending on the flags, the data can be either inline or in a TOAST table; it might be compressed, too
struct varlena是一个通用的结构体,根据字节再转化为具体的对应格式。
struct varlena
{
char vl_len_[4]; /* Do not touch this field directly! */
char vl_dat[FLEXIBLE_ARRAY_MEMBER]; /* Data content is here */
};
根据它的第一个字节,转换为响应格式:
在 postgres.h 的头文件里,可以看到对于变长数据的这一块定义,以小端序为例
xxxxxx00 4-byte length word, aligned, uncompressed data (up to 1G) xxxxxx10 4-byte length word, aligned, compressed data (up to 1G) 00000001 1-byte length word, unaligned, TOAST pointer xxxxxxx1 1-byte length word, unaligned, uncompressed data (up to 126b)
比如 varattrib_1b类型 用于存储不超过 127 个字节的数据,如下的 header 就是 0x00001001,满足第四条规则

varattrib_4b 类型会分为是否被压缩,最高第二位为 1,则表示存储的数据是未压缩的。为 0,则表示存储的数据是压缩过的。

varattrib_1b_e 并不存储数据,只是指向了外部数据的地址。前 4 个字节是 Original data size (includes header),后四个字节是 External saved size (doesn't),再是 TOAST 值的 OID 和TOAST 表的 OID。
typedef struct varatt_external
{
int32 va_rawsize; /* Original data size (includes header) */
int32 va_extsize; /* External saved size (doesn't) */
Oid va_valueid; /* Unique ID of value within TOAST table */
Oid va_toastrelid; /* RelID of TOAST table containing it */
} varatt_external;

至于 TOAST 的组成,想必各位已经十分熟悉了
postgres=# \d pg_toast.pg_toast_16453
TOAST table "pg_toast.pg_toast_16453"
Column | Type
------------+---------
chunk_id | oid
chunk_seq | integer
chunk_data | bytea
Owning table: "public.toast_demo"
Indexes:
"pg_toast_16453_index" PRIMARY KEY, btree (chunk_id, chunk_seq)
既然相同 chunk_id 组成原表字段的一行数据,那么有没有办法可以快速根据 chunk_id 定位出是哪一行数据呢?比如 ctid?说来也巧,上周有位 DSG 的朋友也问了我这个问题,能否快速定位

当然可以,还是以上方的 toast_demo2 表为例
postgres=# select r.relname,t.relname as toast,i.relname as toast_index from pg_class r, pg_class i, pg_index d, pg_class t where r.relname = 'toast_demo2' and d.indrelid = r.reltoastrelid and i.oid = d.indexrelid and t.oid = r.reltoastrelid;
relname | toast | toast_index
-------------+----------------+----------------------
toast_demo2 | pg_toast_16468 | pg_toast_16468_index
(1 row)
postgres=# select count(*) from toast_demo2;
count
-------
50
(1 row)
toastinfo 插件可以用来观测 toast,https://github.com/credativ/toastinfo
null for NULLsordinary for non-varlena datatypesshort inline varlena for varlena values up to 126 bytes (1 byte header)long inline varlena, (un)compressed for varlena values up to 1GiB (4 bytes header)toasted varlena, (un)compressed for varlena values up to 1GiB stored in TOAST tablespg_toastpointer returns a varlena's chunk_id oid in the corresponding TOAST table. It returns NULL on non-varlena input.而 pg_toastpointer 正是我们需要的!看下效果:
postgres=# SELECT id, length(doc), pg_column_size(doc), pg_toastinfo(doc), pg_toastpointer(doc) FROM toast_demo2;
id | length | pg_column_size | pg_toastinfo | pg_toastpointer
----+----------+----------------+------------------------------------+-----------------
1 | 13988940 | 9664585 | toasted varlena, compressed (pglz) | 16488
2 | 13988940 | 9664585 | toasted varlena, compressed (pglz) | 16489
3 | 13988940 | 9664585 | toasted varlena, compressed (pglz) | 16490
4 | 13988940 | 9664585 | toasted varlena, compressed (pglz) | 16491
5 | 13988940 | 9664585 | toasted varlena, compressed (pglz) | 16492
6 | 13988940 | 9664585 | toasted varlena, compressed (pglz) | 16493
7 | 13988940 | 9664585 | toasted varlena, compressed (pglz) | 16494
8 | 13988940 | 9664585 | toasted varlena, compressed (pglz) | 16495
9 | 13988940 | 9664585 | toasted varlena, compressed (pglz) | 16496
10 | 13988940 | 9664585 | toasted varlena, compressed (pglz) | 16497
11 | 13988940 | 9664585 | toasted varlena, compressed (pglz) | 16498
12 | 13988940 | 9664585 | toasted varlena, compressed (pglz) | 16499
13 | 13988940 | 9664585 | toasted varlena, compressed (pglz) | 16500
14 | 13988940 | 9664585 | toasted varlena, compressed (pglz) | 16501
15 | 13988940 | 9664585 | toasted varlena, compressed (pglz) | 16502
16 | 13988940 | 9664585 | toasted varlena, compressed (pglz) | 16503
17 | 13988940 | 9664585 | toasted varlena, compressed (pglz) | 16504
18 | 13988940 | 9664585 | toasted varlena, compressed (pglz) | 16505
19 | 13988940 | 9664585 | toasted varlena, compressed (pglz) | 16506
...
postgres=# select page_item_attrs.t_ctid,
postgres-# page_item_attrs.t_attrs[2],
postgres-# substr(substr(page_item_attrs.t_attrs[2],octet_length(page_item_attrs.t_attrs[2])-7,4)::text,3) as substr_for_chunk_id,
postgres-# ('x'||regexp_replace(substr(substr(page_item_attrs.t_attrs[2],octet_length(page_item_attrs.t_attrs[2])-7,4)::text,3),'(\w\w)(\w\w)(\w\w)(\w\w)','\4\3\2\1'))::bit(32)::int as chunk_id,
postgres-# substr(substr(page_item_attrs.t_attrs[2],octet_length(page_item_attrs.t_attrs[2])-3,4)::text,3) as substr_for_toast_relid,
postgres-# ('x'||regexp_replace(substr(substr(page_item_attrs.t_attrs[2],octet_length(page_item_attrs.t_attrs[2])-3,4)::text,3),'(\w\w)(\w\w)(\w\w)(\w\w)','\4\3\2\1'))::bit(32)::int as toast_relid
postgres-# FROM
postgres-# generate_series(0, pg_relation_size('toast_demo2'::regclass::text) / 8192 - 1) blkno ,
postgres-# heap_page_item_attrs(get_raw_page('toast_demo2', blkno::int), 'toast_demo2'::regclass) as page_item_attrs
postgres-# where
postgres-# substr(page_item_attrs.t_attrs[2]::text,3,2)='01';
t_ctid | t_attrs | substr_for_chunk_id | chunk_id | substr_for_toast_relid | toast_relid
--------+----------------------------------------+---------------------+----------+------------------------+-------------
(0,1) | \x01125074d500497893006840000057400000 | 68400000 | 16488 | 57400000 | 16471
(0,2) | \x01125074d500497893006940000057400000 | 69400000 | 16489 | 57400000 | 16471
(0,3) | \x01125074d500497893006a40000057400000 | 6a400000 | 16490 | 57400000 | 16471
(0,4) | \x01125074d500497893006b40000057400000 | 6b400000 | 16491 | 57400000 | 16471
(0,5) | \x01125074d500497893006c40000057400000 | 6c400000 | 16492 | 57400000 | 16471
(0,6) | \x01125074d500497893006d40000057400000 | 6d400000 | 16493 | 57400000 | 16471
(0,7) | \x01125074d500497893006e40000057400000 | 6e400000 | 16494 | 57400000 | 16471
(0,8) | \x01125074d500497893006f40000057400000 | 6f400000 | 16495 | 57400000 | 16471
(0,9) | \x01125074d500497893007040000057400000 | 70400000 | 16496 | 57400000 | 16471
(0,10) | \x01125074d500497893007140000057400000 | 71400000 | 16497 | 57400000 | 16471
(0,11) | \x01125074d500497893007240000057400000 | 72400000 | 16498 | 57400000 | 16471
(0,12) | \x01125074d500497893007340000057400000 | 73400000 | 16499 | 57400000 | 16471
(0,13) | \x01125074d500497893007440000057400000 | 74400000 | 16500 | 57400000 | 16471
(0,14) | \x01125074d500497893007540000057400000 | 75400000 | 16501 | 57400000 | 16471
(0,15) | \x01125074d500497893007640000057400000 | 76400000 | 16502 | 57400000 | 16471
(0,16) | \x01125074d500497893007740000057400000 | 77400000 | 16503 | 57400000 | 16471
(0,17) | \x01125074d500497893007840000057400000 | 78400000 | 16504 | 57400000 | 16471
(0,18) | \x01125074d500497893007940000057400000 | 79400000 | 16505 | 57400000 | 16471
(0,19) | \x01125074d500497893007a40000057400000 | 7a400000 | 16506 | 57400000 | 16471
(0,20) | \x01125074d500497893007b40000057400000 | 7b400000 | 16507 | 57400000 | 16471
(0,21) | \x01125074d500497893007c40000057400000 | 7c400000 | 16508 | 57400000 | 16471
(0,22) | \x01125074d500497893007d40000057400000 | 7d400000 | 16509 | 57400000 | 16471
(0,23) | \x01125074d500497893007e40000057400000 | 7e400000 | 16510 | 57400000 | 16471
(0,24) | \x01125074d500497893007f40000057400000 | 7f400000 | 16511 | 57400000 | 16471
(0,25) | \x01125074d500497893008040000057400000 | 80400000 | 16512 | 57400000 | 16471
(0,26) | \x01125074d500497893008140000057400000 | 81400000 | 16513 | 57400000 | 16471
(0,27) | \x01125074d500497893008240000057400000 | 82400000 | 16514 | 57400000 | 16471
(0,28) | \x01125074d500497893008340000057400000 | 83400000 | 16515 | 57400000 | 16471
(0,29) | \x01125074d500497893008440000057400000 | 84400000 | 16516 | 57400000 | 16471
(0,30) | \x01125074d500497893008540000057400000 | 85400000 | 16517 | 57400000 | 16471
(0,31) | \x01125074d500497893008640000057400000 | 86400000 | 16518 | 57400000 | 16471
(0,32) | \x01125074d500497893008740000057400000 | 87400000 | 16519 | 57400000 | 16471
(0,33) | \x01125074d500497893008840000057400000 | 88400000 | 16520 | 57400000 | 16471
(0,34) | \x01125074d500497893008940000057400000 | 89400000 | 16521 | 57400000 | 16471
(0,35) | \x01125074d500497893008a40000057400000 | 8a400000 | 16522 | 57400000 | 16471
(0,36) | \x01125074d500497893008b40000057400000 | 8b400000 | 16523 | 57400000 | 16471
(0,37) | \x01125074d500497893008c40000057400000 | 8c400000 | 16524 | 57400000 | 16471
(0,38) | \x01125074d500497893008d40000057400000 | 8d400000 | 16525 | 57400000 | 16471
(0,39) | \x01125074d500497893008e40000057400000 | 8e400000 | 16526 | 57400000 | 16471
(0,40) | \x01125074d500497893008f40000057400000 | 8f400000 | 16527 | 57400000 | 16471
(0,41) | \x01125074d500497893009040000057400000 | 90400000 | 16528 | 57400000 | 16471
(0,42) | \x01125074d500497893009140000057400000 | 91400000 | 16529 | 57400000 | 16471
(0,43) | \x01125074d500497893009240000057400000 | 92400000 | 16530 | 57400000 | 16471
(0,44) | \x01125074d500497893009340000057400000 | 93400000 | 16531 | 57400000 | 16471
(0,45) | \x01125074d500497893009440000057400000 | 94400000 | 16532 | 57400000 | 16471
(0,46) | \x01125074d500497893009540000057400000 | 95400000 | 16533 | 57400000 | 16471
(0,47) | \x01125074d500497893009640000057400000 | 96400000 | 16534 | 57400000 | 16471
(0,48) | \x01125074d500497893009740000057400000 | 97400000 | 16535 | 57400000 | 16471
(0,49) | \x01125074d500497893009840000057400000 | 98400000 | 16536 | 57400000 | 16471
(0,50) | \x01125074d500497893009940000057400000 | 99400000 | 16537 | 57400000 | 16471
(50 rows)
postgres=# select relname from pg_class where oid = 16471;
relname
----------------
pg_toast_16468
(1 row)
爽歪歪,对应关系十分清晰,所以假如各位碰到了 unexpected chunk number 0 (expected 1) for toast value nnnnn 这种错误,知道如何快速定位了吧?
以上就是关于 TOAST 的方方面面了,that's all.
原文作者:PostgreSQL学徒;如有侵权,请联系管理员删除。