devmapper: Thin Pool has 162394 free data blocks which is less than minimum required 163840 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior

问题：

制作镜像的时候报错

devmapper: Thin Pool has 162394 free data blocks which is less than minimum required 163840 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior

 

解决方案：

 

运行下面三个命令： 
// 注意，以下三个命令执行时可能出错是正常的。

清理exited进程：

docker rm $(docker ps -q -f status=exited)

清理dangling volumes：

docker volume rm $(docker volume ls -qf dangling=true)

 

清理dangling image：

docker rmi $(docker images --filter "dangling=true" -q --no-trunc)

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

修改device mapper为direct-lvm

1.停止Docker服务

首先运行docker info 查看当前系统的docker配置：

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57

[root@localhost ~]# docker info Containers: 0  Running: 0  Paused: 0  Stopped: 0 Images: 0 Server Version: 1.12.6 Storage Driver: devicemapper  Pool Name: docker-8:3-135618250-pool  Pool Blocksize: 65.54 kB  Base Device Size: 10.74 GB  Backing Filesystem: xfs  Data file: /dev/loop0  Metadata file: /dev/loop1  Data Space Used: 11.8 MB  Data Space Total: 107.4 GB  Data Space Available: 51.43 GB  Metadata Space Used: 581.6 kB  Metadata Space Total: 2.147 GB  Metadata Space Available: 2.147 GB  Thin Pool Minimum Free Space: 10.74 GB  Udev Sync Supported: true  Deferred Removal Enabled: true  Deferred Deletion Enabled: true  Deferred Deleted Device Count: 0  Data loop file: /var/lib/docker/devicemapper/devicemapper/data  WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.  Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata  Library Version: 1.02.140-RHEL7 (2017-05-03) Logging Driver: journald Cgroup Driver: systemd Plugins:  Volume: local  Network: host bridge null overlay Swarm: inactive Runtimes: docker-runc runc Default Runtime: docker-runc Security Options: seccomp selinux Kernel Version: 3.10.0-327.el7.x86_64 Operating System: CentOS Linux 7 (Core) OSType: linux Architecture: x86_64 Number of Docker Hooks: 3 CPUs: 2 Total Memory: 1.954 GiB Name: localhost.localdomain ID: 5PPL:6TWG:X2JA:VSVV:YYFD:X7BG:EBP5:GYWB:S3FJ:QET6:T3ED:5HTM Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ WARNING: bridge-nf-call-iptables is disabled WARNING: bridge-nf-call-ip6tables is disabled Insecure Registries:  127.0.0.0/8 Registries: docker.io (secure) [root@localhost ~]#

 

发现Storage Driver是Devicemapper，Data File和Metadata File都是loop设备，下面我们将docker停掉：

1	# systemctl stop docker

 

2. 添加磁盘并创建thin-pool lv

 

查看新添加的磁盘

1 2 3 4 5 6 7 8

[root@localhost ~]# fdisk -l /dev/sdb   Disk /dev/sdb: 21.5 GB, 21474836480 bytes, 41943040 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes   [root@localhost ~]#

 

创建pv

1 2 3

[root@localhost ~]# pvcreate /dev/sdb   Physical volume "/dev/sdb" successfully created. [root@localhost ~]#

 

创建vg

1 2 3

[root@localhost ~]# vgcreate docker /dev/sdb    Volume group "docker" successfully created [root@localhost ~]#

 

创建data lv

1 2 3

[root@localhost ~]# lvcreate --wipesignatures y -n thinpool docker -l 95%VG    Logical volume "thinpool" created. [root@localhost ~]#

 

创建metadata lv

1 2 3

[root@localhost ~]# lvcreate --wipesignatures y -n thinpoolmeta docker -l 1%VG    Logical volume "thinpoolmeta" created. [root@localhost ~]#

[root@docker ~]# 

数据LV大小为VG的95%,元数据LV大小为VG的1%,剩余的空间用来自动扩展。

 

注意:作为meta的pool大小不能超过16GB！！！

1 2 3 4 5

[root@localhost ~]# lvs   LV           VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert   thinpool     docker -wi-a----- <19.00g                                                       thinpoolmeta docker -wi-a----- 204.00m                                                     [root@localhost ~]#

 

将pool 转换为thin-pool

 

将 thinpool lv 的 chunksize 改为 512KB,并且将前 4KB 字节清零。

1 2 3 4 5 6

[root@localhost ~]# lvconvert -y --zero n -c 512k --thinpool docker/thinpool --poolmetadata docker/thinpoolmeta   Thin pool volume with chunk size 512.00 KiB can address at most 126.50 TiB of data.   WARNING: Converting logical volume docker/thinpool and docker/thinpoolmeta to thin pool's data and metadata volumes with metadata wiping.   THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)   Converted docker/thinpool_tdata to thin pool. [root@localhost ~]#

 

创建一个thinpool的profile

1 2 3 4 5 6 7 8 9 10 11 12

[root@localhost ~]# vim /etc/lvm/profile/docker-thinpool.profile  [root@localhost ~]# cat /etc/lvm/profile/docker-thinpool.profile  activation {       thin_pool_autoextend_threshold=80       thin_pool_autoextend_percent=20    } [root@localhost ~]#    定义一个百分比的阈值，表明触发 lvm 自动扩容前，已用空间占比。 thin_pool_autoextend_threshold = 80 每次扩容 thin pool 空间的比例 thin_pool_autoextend_percent = 20

 

应用配置

1 2 3

[root@localhost ~]# lvchange --metadataprofile docker-thinpool docker/thinpool    Logical volume docker/thinpool changed. [root@localhost ~]#

注意： docker-thinpool 即刚才创建的 profile 文件名的前缀，不需要加.profile,而且要在etc/lvm/profile 目录下运行此命令。 执行完毕后不要mount，不要格式化 lv。

 

3. 查看lv状态

1 2 3 4 5

[root@localhost ~]# lvs -o+seg_monitor   LV       VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Monitor     thinpool docker twi-a-t--- <19.00g             0.00   0.03                             monitored [root@localhost ~]# 看到有Meta 和Data下面都有数字，代表刚才创建的thinpool创建成功。

 

4. 查看磁盘状态

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38

[root@localhost ~]# fdisk -l   Disk /dev/sda: 53.7 GB, 53687091200 bytes, 104857600 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x00075587      Device Boot      Start         End      Blocks   Id  System /dev/sda1   *        2048      411647      204800   83  Linux /dev/sda2          411648     1460223      524288   82  Linux swap / Solaris /dev/sda3         1460224   104857599    51698688   83  Linux   Disk /dev/sdb: 21.5 GB, 21474836480 bytes, 41943040 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes     Disk /dev/mapper/docker-thinpool_tmeta: 213 MB, 213909504 bytes, 417792 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes     Disk /dev/mapper/docker-thinpool_tdata: 20.4 GB, 20396900352 bytes, 39837696 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes     Disk /dev/mapper/docker-thinpool: 20.4 GB, 20396900352 bytes, 39837696 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 524288 bytes / 524288 bytes   [root@localhost ~]#

 

5. 配置docker

 

方式一： 调整 docker.service 的配置参数

 

添加如下参数到docker服务的启动项里:

1 2 3 4 5

--storage-driver=devicemapper --storage-opt dm.thinpooldev=/dev/mapper/docker-thinpool --storage-opt dm.use_deferred_removal=true --storage-opt dm.use_deferred_del etion=true 或者： --storage-driver=devicemapper --storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool --storage-opt=dm.use_deferred_removal=true --storage-opt=dm.use_deferred_del etion=true

 

完整的配置文件内容如下： 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43

[root@localhost ~]# cat  /usr/lib/systemd/system/docker.service [Unit] Description=Docker Application Container Engine Documentation=http://docs.docker.com After=network.target Wants=docker-storage-setup.service Requires=docker-cleanup.timer   [Service] Type=notify NotifyAccess=all EnvironmentFile=-/run/containers/registries.conf EnvironmentFile=-/etc/sysconfig/docker EnvironmentFile=-/etc/sysconfig/docker-storage EnvironmentFile=-/etc/sysconfig/docker-network Environment=GOTRACEBACK=crash Environment=DOCKER_HTTP_HOST_COMPAT=1 Environment=PATH=/usr/libexec/docker:/usr/bin:/usr/sbin ExecStart=/usr/bin/dockerd-current \           --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current \           --default-runtime=docker-runc \           --exec-opt native.cgroupdriver=systemd \           --userland-proxy-path=/usr/libexec/docker/docker-proxy-current \           --storage-driver=devicemapper --storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool --storage-opt=dm.use_deferred_removal=true --storage-opt=dm.use_deferred_deletion=true \           $OPTIONS \           $DOCKER_STORAGE_OPTIONS \           $DOCKER_NETWORK_OPTIONS \           $ADD_REGISTRY \           $BLOCK_REGISTRY \           $INSECURE_REGISTRY\           $REGISTRIES ExecReload=/bin/kill -s HUP $MAINPID LimitNOFILE=1048576 LimitNPROC=1048576 LimitCORE=infinity TimeoutStartSec=0 Restart=on-abnormal MountFlags=slave KillMode=process   [Install] WantedBy=multi-user.target [root@localhost ~]#

 

或者：

1	[root@docker ~]# sed -i '/^ExecStart=/c\ExecStart=/usr/bin/dockerd --storage-driver=devicemapper --storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool --storage-opt=dm.use_deferred_removal=true --storage-opt=dm.use_deferred_deletion=true \\' /lib/systemd/system/docker.service

完整的配置文件内容如下： 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

[root@docker ~]# cat /usr/lib/systemd/system/docker.service [Unit] Description=Docker Application Container Engine Documentation=http://docs.docker.com After=network.target Wants=docker-storage-setup.service Requires=docker-cleanup.timer   [Service] Type=notify NotifyAccess=all EnvironmentFile=-/run/containers/registries.conf EnvironmentFile=-/etc/sysconfig/docker EnvironmentFile=-/etc/sysconfig/docker-storage EnvironmentFile=-/etc/sysconfig/docker-network Environment=GOTRACEBACK=crash Environment=DOCKER_HTTP_HOST_COMPAT=1 Environment=PATH=/usr/libexec/docker:/usr/bin:/usr/sbin ExecStart=/usr/bin/dockerd --storage-driver=devicemapper --storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool --storage-opt=dm.use_deferred_removal=true --storage-opt=dm.use_deferred_deletion=true \           --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current \           --default-runtime=docker-runc \           --exec-opt native.cgroupdriver=systemd \           --userland-proxy-path=/usr/libexec/docker/docker-proxy-current \           $OPTIONS \           $DOCKER_STORAGE_OPTIONS \           $DOCKER_NETWORK_OPTIONS \           $ADD_REGISTRY \           $BLOCK_REGISTRY \           $INSECURE_REGISTRY\           $REGISTRIES ExecReload=/bin/kill -s HUP $MAINPID LimitNOFILE=1048576 LimitNPROC=1048576 LimitCORE=infinity TimeoutStartSec=0 Restart=on-abnormal MountFlags=slave KillMode=process   [Install] WantedBy=multi-user.target [root@docker ~]#

 

 

方式二： 在 daemon.json 中配置参数

1 2 3 4 5 6 7 8 9 10 11

[root@localhost ~]# cat /etc/docker/daemon.json  {  "storage-driver": "devicemapper",     "storage-opts": [         "dm.thinpooldev=/dev/mapper/docker-thinpool",         "dm.use_deferred_removal=true",         "dm.use_deferred_deletion=true"     ] } [root@localhost ~]# systemctl daemon-reload [root@localhost ~]# systemctl restart docker

 

6. 清除Graphdriver

 

在启动docker之前，需要将之前残留的docker文件删除掉，要不然会有以下报错：

1 2 3 4

Error starting daemon: error initializing graphdriver: devmapper: Base Device UUID and Filesystem   verification failed: devicemapper: Error running deviceCreate (ActivateDevice) dm_task_run failed     rm -rf /var/lib/docker/*

 

7. 启动docker，并查看docker详情

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56

[root@localhost ~]# systemctl daemon-reload                     [root@localhost ~]# systemctl start docker                      [root@localhost ~]# docker info Containers: 0  Running: 0  Paused: 0  Stopped: 0 Images: 0 Server Version: 1.12.6 Storage Driver: devicemapper  Pool Name: docker-thinpool  Pool Blocksize: 524.3 kB  Base Device Size: 10.74 GB  Backing Filesystem: xfs  Data file:   Metadata file:   Data Space Used: 20.45 MB  Data Space Total: 20.4 GB  Data Space Available: 20.38 GB  Metadata Space Used: 61.44 kB  Metadata Space Total: 213.9 MB  Metadata Space Available: 213.8 MB  Thin Pool Minimum Free Space: 2.039 GB  Udev Sync Supported: true  Deferred Removal Enabled: true  Deferred Deletion Enabled: true  Deferred Deleted Device Count: 0  Library Version: 1.02.140-RHEL7 (2017-05-03) Logging Driver: journald Cgroup Driver: systemd Plugins:  Volume: local  Network: host bridge overlay null Swarm: inactive Runtimes: docker-runc runc Default Runtime: docker-runc Security Options: seccomp selinux Kernel Version: 3.10.0-327.el7.x86_64 Operating System: CentOS Linux 7 (Core) OSType: linux Architecture: x86_64 Number of Docker Hooks: 3 CPUs: 2 Total Memory: 1.954 GiB Name: localhost.localdomain ID: 5PPL:6TWG:X2JA:VSVV:YYFD:X7BG:EBP5:GYWB:S3FJ:QET6:T3ED:5HTM Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ WARNING: bridge-nf-call-iptables is disabled WARNING: bridge-nf-call-ip6tables is disabled Insecure Registries:  127.0.0.0/8 Registries: docker.io (secure) [root@localhost ~]#

 

查看 devicemapper 的资源,发现 docker-thinpool 与 docker info 显示的 Pool Name 一致,代表启用direct-lvm 成功。

1 2 3 4 5

[root@localhost ~]# dmsetup ls   docker-thinpool_tdata   (253:1) docker-thinpool_tmeta   (253:0) docker-thinpool (253:2) [root@localhost ~]#

 

8.查看对应的对应的设备

1 2 3 4 5 6 7 8

[root@localhost ~]# lsblk /dev/sdb NAME                    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT sdb                       8:16   0   20G  0 disk  ├─docker-thinpool_tmeta 253:0    0  204M  0 lvm   │ └─docker-thinpool     253:2    0   19G  0 lvm   └─docker-thinpool_tdata 253:1    0   19G  0 lvm     └─docker-thinpool     253:2    0   19G  0 lvm   [root@localhost ~]#

 

9.docker info 中其它的警告取消

默认禁用网桥上的netfilter，所以出现以下警告

1 2	WARNING: bridge-nf-call-iptables is disabled WARNING: bridge-nf-call-ip6tables is disabled

修改配置文件 /usr/lib/sysctl.d/00-system.conf，将net.bridge.bridge参数值设定为1，然后重启系统；

1 2 3

net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-arptables = 1

 

10.再次查看docker信息

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52

[root@localhost ~]# docker info Containers: 0  Running: 0  Paused: 0  Stopped: 0 Images: 0 Server Version: 1.12.6 Storage Driver: devicemapper  Pool Name: docker-thinpool  Pool Blocksize: 524.3 kB  Base Device Size: 10.74 GB  Backing Filesystem: xfs  Data file:   Metadata file:   Data Space Used: 20.45 MB  Data Space Total: 20.4 GB  Data Space Available: 20.38 GB  Metadata Space Used: 61.44 kB  Metadata Space Total: 213.9 MB  Metadata Space Available: 213.8 MB  Thin Pool Minimum Free Space: 2.039 GB  Udev Sync Supported: true  Deferred Removal Enabled: true  Deferred Deletion Enabled: true  Deferred Deleted Device Count: 0  Library Version: 1.02.140-RHEL7 (2017-05-03) Logging Driver: journald Cgroup Driver: systemd Plugins:  Volume: local  Network: overlay null bridge host Swarm: inactive Runtimes: docker-runc runc Default Runtime: docker-runc Security Options: seccomp selinux Kernel Version: 3.10.0-327.el7.x86_64 Operating System: CentOS Linux 7 (Core) OSType: linux Architecture: x86_64 Number of Docker Hooks: 3 CPUs: 2 Total Memory: 1.954 GiB Name: localhost.localdomain ID: 5PPL:6TWG:X2JA:VSVV:YYFD:X7BG:EBP5:GYWB:S3FJ:QET6:T3ED:5HTM Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Insecure Registries:  127.0.0.0/8 Registries: docker.io (secure) [root@localhost ~]#