ITPub博客
首页 博客 专栏 学院 社区 IT新媒体评选
发博

首页 > Linux操作系统 > Linux操作系统 > RHEL5 multipath多路径聚合到RAC的RAW设备绑定中不能用raw raw1 /dev/dm-*来实现

RHEL5 multipath多路径聚合到RAC的RAW设备绑定中不能用raw raw1 /dev/dm-*来实现

Linux操作系统 作者：luashin 时间：2016-02-12 09:17:44 0 删除 编辑

Name不会变化,但dm-*会重新识别,在RAC的RAW设备绑定中,不能用raw raw1 /dev/dm-*来实现。

两个节点,分别连接两台SAN交换,接到同一个存储上生成48条通道,为了能让48条通道冗余,用了linux下的multipath。

配置multipath.conf
cat /etc/multipath.conf
.........
# The defaults section
defaults {
        udev_dir                /dev
        polling_interval        10
        selector                "round-robin 0"
        path_grouping_policy    failover
        getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
        prio                    const
        path_checker            tur
        rr_min_io               100
        rr_weight               uniform
        failback                immediate
        no_path_retry           12
        user_friendly_names     yes
}
.........

blacklist {
#       wwid            26353900f02796769
        devnode         "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
        devnode         "^hd[a-z][[0-9]*]"
        devnode         "^cciss!c[0-9]d[0-9]*"
.......

multipaths {
# based on its wwid

        multipath {
                wwid                    36001438005de9b800000600000140000
                alias                   PSIFile
        }
        multipath {
                wwid                    36001438005de9b800000600000280000                            
                alias                   PSIDataRaw01           //给光纤的WWID起个别名
        }
        multipath {
                wwid                    36001438005de9b8000006000002e0000
                alias                   PSIDataRaw02
        }
}

device {
        vendor                 "HP"
        product                "HSV2[01]0|HSV300|HSV4[05]0"
        getuid_callout         "/sbin/scsi_id -g -u -s /block/%n"
        prio                   alua
        hardware_handler       "0"
        path_selector          "round-robin 0"
        path_grouping_policy   group_by_prio
        failback               immediate
        rr_weight              uniform
        no_path_retry          18
        rr_min_io              100
        path_checker           tur
}

重启服务： 
/etc/init.d/multipath start
/etc/init.d/boot.multipathd start 
这样会在/dev下生成/dev/dm-*，/dev/disk/by-name和/dev/mapper的设备名称。
ls -l /dev/disk/
total 0
drwxr-xr-x 2 root root  820 Dec  4  2009 by-id
drwxr-xr-x 2 root root  240 Dec  4  2009 by-name
drwxr-xr-x 2 root root 3460 Dec  4  2009 by-path
drwxr-xr-x 2 root root  180 Dec  4  2009 by-uuid

ls -l /dev/mapper/
total 0
brw------- 1 root   root     253, 2 Dec  4  2009 PSIDataRaw01
brw------- 1 root   root     253, 3 Dec  4  2009 PSIDataRaw01-part1
brw------- 1 root   root     253, 4 Dec  4  2009 PSIDataRaw01-part2
brw------- 1 root   root     253, 7 Dec  4  2009 PSIDataRaw01-part3
brw------- 1 root   root     253, 0 Dec  4  2009 PSIDataRaw02
brw------- 1 root   root     253, 5 Dec  4  2009 PSIDataRaw02-part1
brw------- 1 root   root     253, 6 Dec  4  2009 PSIDataRaw02-part2
brw-r----- 1 oracle oinstall 253, 8 Dec  3 22:18 PSIDataRaw02-part3
brw------- 1 root   root     253, 1 Dec  4  2009 PSIFile
brw------- 1 root   root     253, 9 Dec  4  2009 PSIFile-part1
lrwxrwxrwx 1 root   root         16 Dec  4  2009 control -> ../device-mapper

ls -l /dev/dm*
brw-r----- 1 root disk 253, 0 Dec  4  2009 /dev/dm-0
brw-r----- 1 root disk 253, 1 Dec  4  2009 /dev/dm-1
brw-r----- 1 root disk 253, 2 Dec  4  2009 /dev/dm-2
brw-r----- 1 root disk 253, 3 Dec  4  2009 /dev/dm-3      -----做multipath时,重启后这些设备名称会重新识别
brw-r----- 1 root disk 253, 4 Dec  4  2009 /dev/dm-4
brw-r----- 1 root disk 253, 5 Dec  4  2009 /dev/dm-5
brw-r----- 1 root disk 253, 6 Dec  4  2009 /dev/dm-6
brw-r----- 1 root disk 253, 7 Dec  4  2009 /dev/dm-7
brw-r----- 1 root disk 253, 8 Dec  4  2009 /dev/dm-8
brw-r----- 1 root disk 253, 9 Dec  4  2009 /dev/dm-9

ls -l /dev/disk/by-name/
total 0
lrwxrwxrwx 1 root root 10 Dec  4  2009 PSIDataRaw01 -> ../../dm-2
lrwxrwxrwx 1 root root 10 Dec  4  2009 PSIDataRaw01-part1 -> ../../dm-3
lrwxrwxrwx 1 root root 10 Dec  4  2009 PSIDataRaw01-part2 -> ../../dm-4
lrwxrwxrwx 1 root root 10 Dec  4  2009 PSIDataRaw01-part3 -> ../../dm-7   ----重启后,前面的alias name不会变化,但dm-*会重新识别.
lrwxrwxrwx 1 root root 10 Dec  4  2009 PSIDataRaw02 -> ../../dm-0
lrwxrwxrwx 1 root root 10 Dec  4  2009 PSIDataRaw02-part1 -> ../../dm-5
lrwxrwxrwx 1 root root 10 Dec  4  2009 PSIDataRaw02-part2 -> ../../dm-6
lrwxrwxrwx 1 root root 10 Dec  4  2009 PSIDataRaw02-part3 -> ../../dm-8
lrwxrwxrwx 1 root root 10 Dec  4  2009 PSIFile -> ../../dm-1
lrwxrwxrwx 1 root root 10 Dec  4  2009 PSIFile-part1 -> ../../dm-9

在RAC的RAW设备绑定中,不能用raw raw1 /dev/dm-*来实现. 
cat /etc/raw
raw1:mapper/PSIDataRaw02-part1   --OCR
raw2:mapper/PSIDataRaw02-part2   --OCR
raw3:mapper/PSIDataRaw01-part1   --VOTEDISK
raw4:mapper/PSIDataRaw01-part2   --VOTEDISK
raw5:mapper/PSIDataRaw01-part3   --VOTEDISK

在SUSE 10中.RAW设备分区属主和权限不能写在/etc/rc.d/boot.local中.要用udev的rules来单独定义,单独在/etc/udev/rules.d下建个rules来定义这些.
[PSIDB01:~ ]# cat /etc/udev/rules.d/99-raw.rules
KERNEL=="raw[1-2]*", OWNER="root", GROUP="oinstall", MODE="640"
KERNEL=="raw[3-5]*", OWNER="oracle", GROUP="oinstall", MODE="660"
# /etc/init.d/boot.udev stop  
# /etc/init.d/boot.udev start    --重启udev
 

用作ASM磁盘的分区属主和权限在/etc/rc.d/boot.local下添加:

/bin/chown oracle:oinstall /dev/mapper/PSIDataRaw02-part3
/bin/chmod 640 /dev/mapper/PSIDataRaw02-part3


# chkconfig raw on

# chkconfig multipathd on

# chkconfig boot.multipath boot.

重启节点后,要重新执行

# /oracle/product/crs/root.sh

(今天早上ORACLE回复,可以运行/etc/init.d/init.cssd startcheck 去查看相关问题,root.sh可以fix一些配置问题.印象中我也运行了这个命令,好象是报对crsctl没有权限.问题虽然解决,错过一次学习的机会)

查看crs进程

# ps -ef |grep init
root         1     0  0 17:02 ?        00:00:09 init [5]
root      6888 25389  0 22:38 pts/2    00:00:00 grep init
root     10113     1  0 17:03 ?        00:00:00 /usr/sbin/sshd -o PidFile=/var/run/sshd.init.pid
root     31640     1  0 22:26 ?        00:00:01 /bin/sh /etc/init.d/init.cssd fatal
root     31837     1  0 22:26 ?        00:00:00 /bin/sh /etc/init.d/init.evmd run
root     31946     1  0 22:26 ?        00:00:00 /bin/sh /etc/init.d/init.crsd run
root     32317 31640  0 22:27 ?        00:00:00 /bin/sh /etc/init.d/init.cssd oprocd
root     32324 31640  0 22:27 ?        00:00:00 /bin/sh /etc/init.d/init.cssd oclsomon
root     32340 31640  0 22:27 ?        00:00:00 /bin/sh /etc/init.d/init.cssd daemon

# ps -ef |grep crs
oracle     586 32340  0 22:27 ?        00:00:00 /oracle/product/crs/bin/ocssd.bin
oracle     644 31837  0 22:27 ?        00:00:00 /bin/su -l oracle -c sh -c 'ulimit -c unlimited; cd /oracle/product/crs/log/psidb02/evmd; exec /oracle/product/crs/bin/evmd '
oracle     645   644  0 22:27 ?        00:00:00 /oracle/product/crs/bin/evmd.bin
root       736 31946  0 22:27 ?        00:00:01 /oracle/product/crs/bin/crsd.bin reboot
oracle     841   645  0 22:27 ?        00:00:00 /oracle/product/crs/bin/evmlogger.bin -o /oracle/product/crs/evm/log/evmlogger.info -l /oracle/product/crs/evm/log/evmlogger.log
oracle    1638     1  0 22:28 ?        00:00:00 /oracle/product/crs/opmn/bin/ons -d
oracle    1639  1638  0 22:28 ?        00:00:00 /oracle/product/crs/opmn/bin/ons -d
root      7161 25389  0 22:39 pts/2    00:00:00 grep crs
root     31946     1  0 22:26 ?        00:00:00 /bin/sh /etc/init.d/init.crsd run
root     32630 32317  0 22:27 ?        00:00:00 /oracle/product/crs/bin/oprocd.bin run -t 1000 -m 500 -f
oracle   32641 32324  0 22:27 ?        00:00:00 /bin/su -l oracle -c /bin/sh -c 'cd /oracle/product/crs/log/psidb02/cssd/oclsomon; ulimit -c unlimited; /oracle/product/crs/bin/oclsomon  || exit $?'
oracle   32642 32641  0 22:27 ?        00:00:00 /bin/sh -c cd /oracle/product/crs/log/psidb02/cssd/oclsomon; ulimit -c unlimited; /oracle/product/crs/bin/oclsomon  || exit $?
oracle   32660 32642  0 22:27 ?        00:00:00 /oracle/product/crs/bin/oclsomon.bin


CRS在启动时出现问题,会在tmp下生成ocrctl.*的文件,查看这些文件信息,参考crs/log/nodename/cssd和crsd的日志可以进行分析.

# ll -l /tmp/crs*
-rw-r--r-- 1 oracle oinstall 0 Dec  3 20:35 /tmp/crsctl.14126
CRS正常,在节点服务启动时出现的问题,可以到下面路径里查看相关信息
PSIDB02:/oracle/product/db/log/psidb02/racg
# ll
total 268
-rw-r--r-- 1 oracle users    784 Nov 30 09:58 imon.log
-rw-r--r-- 1 oracle users 123401 Dec  3 22:28 imon_psiccic.log
-rw-r--r-- 1 oracle users  50897 Dec  3 22:28 imonpsiccic.log
-rw-r--r-- 1 oracle users  10560 Dec  3 22:28 mdb.log
-rw-r--r-- 1 oracle users  16855 Dec  3 22:28 ora.psiccic.psiccic2.inst.log
-rw-r--r-- 1 oracle users  14744 Dec  3 22:28 ora.psidb02.ASM2.asm.log
-rw-r--r-- 1 oracle users  12790 Dec  3 22:28 ora.psidb02.LISTENER_PSIDB02.lsnr.log
drwxr-xr-t 2 oracle users   4096 Nov 24 17:01 racgeut
drwxr-xr-t 2 oracle users   4096 Nov 24 17:03 racgimon
drwxr-xr-t 2 oracle users   4096 Nov 24 17:01 racgmain
drwxr-xr-t 2 oracle users   4096 Nov 24 17:03 racgmdb


    先装的ASM,磁盘组指向了/dev/dm-*,发现multipath的dem-*变化的现象后,ASM磁盘组也需要进行对应的更改,将原来磁盘组对应r的/dev/dm-5改为/dev/mapper/PSIDataRaw02-part3' 
oracle@PSIDB01:~> export ORACLE_SID=+ASM1
oracle@PSIDB01:~> sqlplus /nolog
SQL*Plus: Release 10.2.0.4.0 - Production on Thu Dec 3 21:59:55 2009
Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.
SQL> connect /as sysdba
Connected.
SQL> show parameter asm;
NAME                                 TYPE                   VALUE
------------------------------------ ---------------------- ------------------------------
asm_diskgroups                       string                 DATA
asm_diskstring                       string                 /dev/dm*
asm_power_limit                      integer                1

SQL> select * from v$asm_diskgroup;
no rows selected

SQL> select * from v$asm_disk;
no rows selected

SQL> alter system set asm_diskstring='/dev/mapper/PSIDataRaw02-part3';
System altered.

SQL> select path,mount_status from v$asm_disk;
PATH                                                                             MOUNT_STATUS
-------------------------------------------------------------------------------- --------------
/dev/mapper/PSIDataRaw02-part3                                                   CLOSED

SQL> alter diskgroup DATA mount;
Diskgroup altered.

SQL> select path,mount_status from v$asm_disk;
PATH                                                                             MOUNT_STATUS
-------------------------------------------------------------------------------- --------------
/dev/mapper/PSIDataRaw02-part3                                                   CACHED
 
SQL> create pfile from spfile;
File created.

SQL> shutdown immediate          ---关闭ASM实例

$ ./srvctl start asm -n psidb01  ---重启实例
 
SQL> show parameter asm;
NAME                                 TYPE                   VALUE
------------------------------------ ---------------------- ------------------------------
asm_diskgroups                       string                 DATA
asm_diskstring                       string                 /dev/mapper/PSIDataRaw02-part3
asm_power_limit                      integer
 
SQL> select group_number,name,allocation_unit_size alloc_unit_size,state,type, total_mb, usable_file_mb from v$asm_diskgroup; 
GROUP_NUMBER NAME ALLOC_UNIT_SIZE STATE   TYPE   TOTAL_MB USABLE_FILE_MB
------------ ---- --------------- ------- ------ -------- --------------
           1 DATA 1048576         MOUNTED EXTERN 1330210  930221
 
SQL> select name,path,header_status,total_mb free_mb,trunc(bytes_read/1024/1024) read_mb, trunc(bytes_written/1024/1024) write_mb from v$asm_disk;
NAME      PATH                           HEADER_STATUS FREE_MB READ_MB WRITE_MB
--------- ------------------------------ ------------- ------- ------- --------
DATA_0000 /dev/mapper/PSIDataRaw02-part3 MEMBER        1330210 0       0
 
$ ./crs_stat -t
Name           Type           Target    State     Host      
------------------------------------------------------------
ora.psiccic.db application    ONLINE    ONLINE    psidb02   
ora....c1.inst application    OFFLINE   OFFLINE             
ora....c2.inst application    ONLINE    ONLINE    psidb02   
ora....SM1.asm application    ONLINE    ONLINE    psidb01   
ora....01.lsnr application    ONLINE    ONLINE    psidb01   
ora....b01.gsd application    ONLINE    ONLINE    psidb01   
ora....b01.ons application    ONLINE    ONLINE    psidb01   
ora....b01.vip application    ONLINE    ONLINE    psidb01   
ora....SM2.asm application    ONLINE    ONLINE    psidb02   
ora....02.lsnr application    ONLINE    ONLINE    psidb02   
ora....b02.gsd application    ONLINE    ONLINE    psidb02   
ora....b02.ons application    ONLINE    ONLINE    psidb02   
ora....b02.vip application    ONLINE    ONLINE    psidb02  
 
PSIDB01:/oracle/product/crs/bin # ./srvctl start instance -d psiccic -i psiccic1
PSIDB01:/oracle/product/crs/bin # ./crs_stat -t
Name           Type           Target    State     Host      
------------------------------------------------------------
ora.psiccic.db application    ONLINE    ONLINE    psidb02   
ora....c1.inst application    ONLINE    ONLINE    psidb01   
ora....c2.inst application    ONLINE    ONLINE    psidb02   
ora....SM1.asm application    ONLINE    ONLINE    psidb01   
ora....01.lsnr application    ONLINE    ONLINE    psidb01   
ora....b01.gsd application    ONLINE    ONLINE    psidb01   
ora....b01.ons application    ONLINE    ONLINE    psidb01   
ora....b01.vip application    ONLINE    ONLINE    psidb01   
ora....SM2.asm application    ONLINE    ONLINE    psidb02   
ora....02.lsnr application    ONLINE    ONLINE    psidb02   
ora....b02.gsd application    ONLINE    ONLINE    psidb02   
ora....b02.ons application    ONLINE    ONLINE    psidb02   
ora....b02.vip application    ONLINE    ONLINE    psidb02

来自 “ ITPUB博客 ” ，链接：https://blog.itpub.net/9034054/viewspace-1987939/ ，如需转载，请注明出处，否则将追究法律责任。

0
0
上一篇： DM Multipath Configuration IN RHEL 6
下一篇： RedHat 5.4上iscsi+multipath多路径存储配置获取设备的wwid
请登录后发表评论 登录
全部评论
luashin

注册时间：2015-02-18

  • 博文量
    290
  • 访问量
    1151050

最新文章

支持我们 作者招募 用户协议 FAQ Contact Us

北京盛拓优讯信息技术有限公司. 版权所有  京ICP备16024965号-8 北京市公安局海淀分局网监中心备案编号：11010802021510 niuxiaotong@pcpop.com 17352615567 未成年人举报专区

广播电视节目制作经营许可证(京) 字第1234号 中国互联网协会会员

违法和不良信息举报电话:18101376593