Cache – Why does L2ARC provide more acceleration for directory traversal?

# zpool list -v
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
geek1 2.72T 1.18T 1.54T-53% 43 % 4.64x ONLINE -
 sdc 1.36T 606G 786G-54% 43%
 sdd 1.36T 606G 786G-53% 43%
log------
 zil 3.97 G 0 3.97G-0% 0%
cache------
 l2arc 32.0G 4.24G 27.8G-0% 13%

# while true; do time find /geek1/ -type f|wc -l; done
46774149

real 1103m18.661s
user 2m15.143s
sys 5m52.096s
46774149

real 243m0.289s
user 2m3.063s
sys 5m7.107s
46774149

real 205m23.922s
user 2m7.566s
sys 6m32.638s

NAME PROPERTY VALUE SOURCE
geek1 size 2.72T -
geek1 capacity 43% -
geek1 altroot-default
geek1 health ONLINE -
geek1 guid default
geek1 version-default
geek1 bootfs-default
geek1 delegation on default
geek1 autoreplace off default
geek1 cachefile-default
geek1 failmode wait default
geek1 listsnapshots off de fault
geek1 autoexpand off default
geek1 dedupditto 0 default
geek1 dedupratio 4.64x -
geek1 free 1.54T -
geek1 allocated 1.18T -
geek1 readonly off -
geek1 ashift 0 default
geek1 comment-default
geek1 expandsize--
geek1 freeing 0 default
geek1 fragmentation 53% -
geek1 leaked 0 default
geek1 feature@async_destroy enabled local
geek1 feature@empty_bpobj acti ve local
geek1 feature@lz4_compress active local
geek1 feature@spacemap_histogram active local
geek1 feature@enabled_txg active local
geek1 feature@hole_birth active local
geek1 feature@extensible_dataset enabled local
geek1 feature@embedded_data active local
geek1 feature@bookmarks enabled local
geek1 feature@filesystem_limits enabled local
geek1 feature@large_blocks enabled local

< /p>

L2A rc is usually configured to cache random rotating disk reads on SSD. I have set this configuration, hoping to speed up directory traversal.

This is the setting:

# zpool list -v
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
geek1 2.72T 1.18T 1.54T-53% 43% 4.64x ONLINE -
 sdc 1.36 T 606G 786G-54% 43%
 sdd 1.36T 606G 786G-53% 43%
log------
 zil 3.97G 0 3.97G-0% 0%
cache------
 l2arc 32.0G 4.24G 27.8G-0% 13%

This machine has 20GB of memory: suitable for Arc.

I hope that the lookup call through the file system can speed up a lot. It speeds up 5 times, but in an absolute sense it is still very slow. It takes 3.4 hours to list 46M files.

# while true; do time find /geek1/ -type f|wc -l; done
46774149

real 1103m18.661s
user 2m15.143s
sys 5m52.096s
46774149

real 243m0.289s
user 2m3.063s
sys 5m7.107s
46774149

real 205m23.922s
user 2m7.566s
sys 6m32.638s

How to improve the efficiency of directory traversal?

zpool gets all printed materials:

NAME PROPERTY VALUE SOURCE
geek1 size 2.72T -
geek1 capacity 43% -
geek1 altroot-default
geek1 health ONLINE -
geek1 guid default
geek1 version-default
geek1 bootfs-default
geek1 delegation on default
geek1 autoreplace off default
geek1 cachefile-default
geek1 failmode wait default
geek1 listsnapshots off defau lt
geek1 autoexpand off default
geek1 dedupditto 0 default
geek1 dedupratio 4.64x -
geek1 free 1.54T -
geek1 allocated 1.18T -
geek1 readonly off -
geek1 ashift 0 default
geek1 comment-default
geek1 expandsize--
geek1 freeing 0 default
geek1 fragmentation 53% -
geek1 leaked 0 default
geek1 feature@async_destroy enabled local
geek1 feature@empty_bpobj active local
geek1 feature@lz4_compress active local
geek1 feature@spacemap_histogram active local
geek1 feature@enabled_txg active local
geek1 feature@hole_birth active local
geek1 feature@extensible_dataset enabled local
geek1 feature@embedded_data active local
geek1 feature@bookmarks enabled local
geek1 feature@filesystem_limits enabled local
geek1 feature@large_blocks enabled local

< p>

If you are using ZFS on Linux, the slowness may depend on the xattr default setting (xattr=on), which stores extended attributes in hidden directories/files.

Try to set chaning xattr to sa and issue something like zfs set xattr = sa. However, it only affects newly created files; so you need to delete/reload all files to get significant Speed ​​up.

In other words, before reloading all files, please make a small attempt.