[20201104]关于稀疏文件(sparse files).txt

--//前几天看链接：https://savvinov.com/2020/10/14/memory-fragmentation-via-buffered-file-i-o/
--//里面提到/var/log/lastlog文件是稀疏文件。对方大小达到273G，拷贝时导致文件系统缓存紧张。
--//家里没有这个环境，上班看看。

--//问题1：如何知道文件是稀疏文件.

# cat /etc/issue
Oracle Linux Server release 5.9
Kernel \r on an \m

# ls -lh /var/log/lastlog
-rw-r--r-- 1 root root 144K 2020-11-04 15:38:11 /var/log/lastlog

# du -sk /var/log/lastlog
44      /var/log/lastlog

--//很明显两者看到的大小不一致。我仅仅知道oracle 的临时表空间文件是稀疏文件。但是如何确定linux文件系统里面那个文件是稀疏文件
--//呢？ 找到如下链接：https://www.thegeekdiary.com/how-to-find-all-the-sparse-file-in-linux/

#  find /var/log/lastlog  -type f -printf "%S\t%p\n"
find: warning: unrecognized format directive `%S'
S       /var/log/lastlog

--//很明显我使用版本不支持这个参数%S。看了man find文档发现：

%b     The amount of disk space used for this file in 512-byte blocks. Since disk space is allocated in multiples of the
       filesystem block size this is usually greater than %s/1024, but it can also be smaller  if  the  file  is  a
       sparse file.

%k     The  amount  of  disk  space used for this file in 1K blocks. Since disk space is allocated in multiples of the
       filesystem block size this is usually greater than %s/1024, but it can also be smaller if the file is a sparse
       file.

#  find /var/log/lastlog  -type f -printf "%k %s\t%p\n"
44 146876       /var/log/lastlog

--//换另外的机器：

# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.8 (Maipo)

# find /var/log/lastlog  -type f -printf "%k %s %S\t%p\n"
52 15862316 0.00335689  /var/log/lastlog
--//52*1024/15862316 = .00335688685057087502,也就是高版本的find才支持%S参数。
--//旧版本改写如下：

# find / -type f -printf "%k %s\t%p\n" | awk '{ if ( $2 >0 && $1*1024/$2 < 1.0 && $2>0 ) {print } }'

--//问题2:如何创建稀疏文件：
https://www.thegeekdiary.com/how-to-create-sparse-files-in-linux-using-dd-command/

# dd if=/dev/zero of=sparse_file bs=1 count=0 seek=512M
0+0 records in
0+0 records out
0 bytes (0 B) copied, 1.6717e-05 seconds, 0.0 kB/s

# find /tmp/sparse_file -type f -printf "%k %s\t%p\n" | awk '{ if ( $2 >0 && $1*1024/$2 < 1.0 && $2>0 ) {print } }'
0 536870912     /tmp/sparse_file

# ls -lh sparse_file
-rw-r--r-- 1 root root 512M 2020-11-04 16:50:45 sparse_file

# du -sk sparse_file
0       sparse_file

# du -h --apparent-size sparse_file
512M    sparse_file

--//链接还提到拷贝问题，我记忆里以前也遇到类似问题，使用lftp客户端传输临时文件到另外的机器卡死。
http://blog.itpub.net/267265/viewspace-752384/

How to copy a sparse file

Copying a sparse file with a program that does not explicitly support them may copy the entire, uncompressed size of the
file, including the sparse. So, when you copy a sparse file using cp command, the destination file gets changed to a
fully allocated file. To copy a sparse file by keeping the destination copy as a sparse file, use any of the below
commands.

# cp --sparse=always source_file new_file
# rsync --sparse source_file new_file
# cpio --sparse
# tar --sparse

--//我看了cp的man文档，发现缺省是--sparse=auto。
# man cp
...
By  default,  sparse  SOURCE files are detected by a crude heuristic and the corresponding DEST file is made sparse as
well.  That is the behavior selected by --sparse=auto.  Specify --sparse=always to create a sparse DEST file whenever
the SOURCE file contains a long enough sequence of zero bytes.  Use --sparse=never to inhibit creation of sparse files.

# cp --sparse=never sparse_file aa
/bin/cp: overwrite `aa'? y

# du  sparse_file aa
0       sparse_file
524804  aa

--//很明显这样拷贝过来的文件就不再是稀疏文件。