环境是oracle服务器,两个光纤卡,分别连接主备两个存储。备存储平时不会响应主机的请求。
本来应该只有连接到主存储的通道才会在/dev下生成设备文件,但是有一定几率出现:备存储的通道被linux识别到了,但是无法使用,于是在oracle服务器上报通道io错误。
这个错误不会对生产造成实质影响,如果要干掉鬼盘,需要存储那边进行排查,并且客户端服务器需要重启。
messages日志报错:
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdap, sector 0
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdap, sector 8
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdap, sector 0
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdaq, sector 0
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdaq, sector 0
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdaq, sector 2097024
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdaq, sector 2097136
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdaq, sector 0
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdaq, sector 8
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdaq, sector 0
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdcf, sector 0
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdcg, sector 0
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdap, sector 0
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdaq, sector 0
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdcf, sector 0
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdcg, sector 0
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdap, sector 0
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdaq, sector 0
Oct 31 14:10:59 p2ccdbbj01 kernel: end_request: I/O error, dev sdcf, sector 0
Oct 31 14:10:59 p2ccdbbj01 kernel: end_request: I/O error, dev sdcg, sector 0
/proc/partitions可以看到sdcf等
69 16 153600 sdcd
69 17 153570 sdcd1
69 32 153600 sdce
69 33 153570 sdce1
69 48 1048576 sdcf
69 64 1048576 sdcg
253 0 208666624 dm-0
253 1 208666624 dm-1
253 2 418381824 dm-2
但是用fdisk -l | grep sdcf看不到报错的设备
最后在/dev/disks/by-path下发现,出问题的设备的确是fc设备,只是后面的地址和其它通道出入很大
lrwxrwxrwx 1 root root 11 Oct 30 18:20 pci-0000:42:00.0-fc-0x5006016446e037ac:0x0012000000000000-part1 -> ../../sdbj1
lrwxrwxrwx 1 root root 10 Oct 30 18:20 pci-0000:42:00.0-fc-0x5006016446e037ac:0x0013000000000000 -> ../../sdbk
lrwxrwxrwx 1 root root 11 Oct 30 18:20 pci-0000:42:00.0-fc-0x5006016446e037ac:0x0013000000000000-part1 -> ../../sdbk1
lrwxrwxrwx 1 root root 10 Oct 30 18:20 pci-0000:42:00.0-fc-0x5006016446e03829:0x0000000000000000 -> ../../sdcf
lrwxrwxrwx 1 root root 10 Oct 30 18:20 pci-0000:42:00.0-fc-0x5006016b46e037ac:0x0000000000000000 -> ../../sdbl
lrwxrwxrwx 1 root root 11 Oct 30 18:21 pci-0000:42:00.0-fc-0x5006016b46e037ac:0x0000000000000000-part1 -> ../../sdbl1
lrwxrwxrwx 1 root root 10 Oct 30 18:20 pci-0000:42:00.0-fc-0x5006016b46e037ac:0x0001000000000000 -> ../../sdbm
lrwxrwxrwx 1 root root 11 Oct 30 18:20 pci-0000:42:00.0-fc-0x5006016b46e037ac:0x0001000000000000-part1 -> ../../sdbm1
lrwxrwxrwx 1 root root 10 Oct 30 18:20 pci-0000:42:00.0-fc-0x5006016b46e03829:0x0000000000000000 -> ../../sdcg