Redhat 6.1 配置Linux multipath安装oracle 11g rac
一、安装配置存储节点（略）
二、数据库节点连接存储节点
2.1、数据库节点安装ISCSI启动器
yum install iscsi*
2.2、配置启动器
vim /etc/iscsi/initiatorname.iscsi
2.3、发起连接
# iscsiadm -m discovery -t st -p IP
# iscsiadm -m node -l
断开连接：
# iscsiadm -m node -u
注销掉某一个特定的Target的连接
iscsiadm -m node -T (目标器) -p (IP+端口) -u
从操作系统中删除target信息：
# iscsiadm -m node -o delete -T (目标器) -p (IP+端口)
查看当前连接：
# iscsiadm -m node session
2.4、数据库节点安装多路径软件并设置开启启动
# yum install iscsi-initiator-utils
# yum install device-mapper-multipath
# chkconfig --list
# chkconfig multipathd on
# chkconfig iscsi on
# chkconfig iscsid on
2.5、配置多路径软件配置文件
# multipath -ll -----这句执行时可能会遇到DM模块未加载
# cp /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf /etc
[[email protected] ~]# cat /etc/multipath.conf
# This is a basic configuration file with some examples, for device mapper
# multipath.
# For a complete list of the default configuration values, see
# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.defaults
# For a list of configuration options with descriptions, see
# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.annotated
#
# REMEMBER: After updating multipath.conf, you must run
#
# service multipathd reload
#
# for the changes to take effect in multipathd
## By default, devices with vendor = "IBM" and product = "S/390.*" are
## blacklisted. To enable mulitpathing on these devies, uncomment the
## following lines.
#blacklist_exceptions {
# device {
# vendor "IBM"
# product "S/390.*"
# }
#}
## Use user friendly names, instead of using WWIDs as names.
defaults {
user_friendly_names yes ------multipath在Linux上的默认配置
path_grouping_policy multibus
failback immediate
no_path_retry fail
}
##
## Here is an example of how to configure some standard options.
##
#
#defaults {
# udev_dir /dev
# polling_interval 10
# path_selector "round-robin 0"
# path_grouping_policy multibus
# getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
# prio alua
# path_checker readsector0
# rr_min_io 100
# max_fds 8192
# rr_weight priorities
# failback immediate
# no_path_retry fail
# user_friendly_names yes
#}
##
## The wwid line in the following blacklist section is shown as an example
## of how to blacklist devices by wwid. The 2 devnode lines are the
## compiled in default blacklist. If you want to blacklist entire types
## of devices, such as all scsi devices, you should use a devnode line.
## However, if you want to blacklist specific devices, you should use
## a wwid line. Since there is no guarantee that a specific device will
## not change names on reboot (from /dev/sda to /dev/sdb for example)
## devnode lines are not recommended for blacklisting specific devices.
##
blacklist {
# wwid 26353900f02796769
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" ----定义过滤黑名单的正则
devnode "^hd[a-z]"
devnode "^sda"
}
#multipaths {
# multipath {
# wwid 3600508b4000156d700012000000b0000
# alias yellow
# path_grouping_policy multibus
# path_checker readsector0
# path_selector "round-robin 0"
# failback manual
# rr_weight priorities
# no_path_retry 5
# }
# multipath {
# wwid 1DEC_____321816758474
# alias red
# }
#}
}
#devices {
# device {
# vendor "COMPAQ "
# product "HSV110 (C)COMPAQ"
# path_grouping_policy multibus
# getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
# path_checker readsector0
# path_selector "round-robin 0"
# hardware_handler "0"
# failback 15
# rr_weight priorities
# no_path_retry queue
# }
# device {
# vendor "COMPAQ "
# product "MSA1000 "
# path_grouping_policy multibus
# }
#}
[[email protected] ~]#
2.6、查看存储端映射磁盘
重启系统或者重启multipath软件后查看
[[email protected] ~]# multipath -ll
mpathe (36001405a44bed8fa92243ebbffdf2a4a) dm-1 LIO-ORG,DATA2
size=7.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 33:0:0:4 sdf 8:80 active ready running
mpathd (36001405f2777afe35334413a428597dd) dm-0 LIO-ORG,FRA2
size=4.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 33:0:0:7 sdi 8:128 active ready running
mpathi (360014051ec24687cd2f47e9a5462075b) dm-8 LIO-ORG,FRA3
size=4.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 33:0:0:8 sdj 8:144 active ready running
mpathh (36001405c65d4d5bb52a4dbf9ca03b44b) dm-7 LIO-ORG,DATA3
size=7.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 33:0:0:5 sdg 8:96 active ready running
OCRVoting3 (36001405a2d3d62e53274d7ba2ae0c7b4) dm-3 LIO-ORG,OCRVoting3
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 33:0:0:2 sdd 8:48 active ready running
mpathg (3600140524c968d2a27441fea594158c9) dm-6 LIO-ORG,DATA1
size=7.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 33:0:0:3 sde 8:64 active ready running
OCRVoting2 (360014050499205fee3d4229be9a41359) dm-4 LIO-ORG,OCRVoting2
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 33:0:0:1 sdc 8:32 active ready running
OCRVoting1 (3600140591641cefc96b430e9658a2cb8) dm-5 LIO-ORG,OCRVoting1
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 33:0:0:0 sdb 8:16 active ready running
mpathf (360014054166a939d4154d5799ad38df6) dm-2 LIO-ORG,FRA1
size=4.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 33:0:0:6 sdh 8:112 active ready running
[[email protected] ~]#
2.7、修改多路径软件配置文件：
[[email protected] ~]# cat /etc/multipath.conf
# This is a basic configuration file with some examples, for device mapper
# multipath.
# For a complete list of the default configuration values, see
# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.defaults
# For a list of configuration options with descriptions, see
# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.annotated
#
# REMEMBER: After updating multipath.conf, you must run
#
# service multipathd reload
#
# for the changes to take effect in multipathd
## By default, devices with vendor = "IBM" and product = "S/390.*" are
## blacklisted. To enable mulitpathing on these devies, uncomment the
## following lines.
#blacklist_exceptions {
# device {
# vendor "IBM"
# product "S/390.*"
# }
#}
## Use user friendly names, instead of using WWIDs as names.
defaults {
user_friendly_names yes ---这里定义的Linux的Multipath默认规则
path_grouping_policy multibus
failback immediate
no_path_retry fail
}
##
## Here is an example of how to configure some standard options.
##
#
#defaults {
# udev_dir /dev
# polling_interval 10
# path_selector "round-robin 0"
# path_grouping_policy multibus
# getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
# prio alua
# path_checker readsector0
# rr_min_io 100
# max_fds 8192
# rr_weight priorities
# failback immediate
# no_path_retry fail
# user_friendly_names yes
#}
##
## The wwid line in the following blacklist section is shown as an example
## of how to blacklist devices by wwid. The 2 devnode lines are the
## compiled in default blacklist. If you want to blacklist entire types
## of devices, such as all scsi devices, you should use a devnode line.
## However, if you want to blacklist specific devices, you should use
## a wwid line. Since there is no guarantee that a specific device will
## not change names on reboot (from /dev/sda to /dev/sdb for example)
## devnode lines are not recommended for blacklisting specific devices.
##
blacklist {
# wwid 26353900f02796769
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" ----这里是定义黑名单规则，这些正则定义了过滤条件
devnode "^hd[a-z]"
devnode "^sda"
}
#multipaths {
# multipath {
# wwid 3600508b4000156d700012000000b0000
# alias yellow
# path_grouping_policy multibus
# path_checker readsector0
# path_selector "round-robin 0"
# failback manual
# rr_weight priorities
# no_path_retry 5
# }
# multipath {
# wwid 1DEC_____321816758474
# alias red
# }
#}
multipaths {
multipath {
wwid 3600140591641cefc96b430e9658a2cb8 ----这里定义的是存储端映射过来的LUN的UUID和别名
alias OCRVoting1
}
multipath {
wwid 360014050499205fee3d4229be9a41359
alias OCRVoting2
}
multipath {
wwid 36001405a2d3d62e53274d7ba2ae0c7b4
alias OCRVoting3
}
multipath {
wwid 3600140524c968d2a27441fea594158c9
alias DATA1
}
multipath {
wwid 36001405a44bed8fa92243ebbffdf2a4a
alias DATA2
}
multipath {
wwid 36001405c65d4d5bb52a4dbf9ca03b44b
alias DATA3
}
multipath {
wwid 360014054166a939d4154d5799ad38df6
alias FRA1
}
multipath {
wwid 36001405f2777afe35334413a428597dd
alias FRA2
}
multipath {
wwid 360014051ec24687cd2f47e9a5462075b
alias FRA3
}
}
#devices {
# device {
# vendor "COMPAQ "
# product "HSV110 (C)COMPAQ"
# path_grouping_policy multibus
# getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
# path_checker readsector0
# path_selector "round-robin 0"
# hardware_handler "0"
# failback 15
# rr_weight priorities
# no_path_retry queue
# }
# device {
# vendor "COMPAQ "
# product "MSA1000 "
# path_grouping_policy multibus
# }
#}
[[email protected] ~]#
2.8查看multipath映射路径：
[[email protected] ~]# ls -lrt /dev/mapper/*
crw-rw----. 1 root root 10, 58 Sep 28 11:17 /dev/mapper/control
lrwxrwxrwx. 1 root root 7 Sep 28 11:17 /dev/mapper/OCRVoting3 -> ../dm-0
lrwxrwxrwx. 1 root root 7 Sep 28 11:17 /dev/mapper/DATA1 -> ../dm-1
lrwxrwxrwx. 1 root root 7 Sep 28 11:17 /dev/mapper/FRA3 -> ../dm-2
lrwxrwxrwx. 1 root root 7 Sep 28 11:17 /dev/mapper/DATA3 -> ../dm-3
lrwxrwxrwx. 1 root root 7 Sep 28 11:17 /dev/mapper/DATA2 -> ../dm-4
lrwxrwxrwx. 1 root root 7 Sep 28 11:17 /dev/mapper/FRA1 -> ../dm-5
lrwxrwxrwx. 1 root root 7 Sep 28 11:17 /dev/mapper/OCRVoting1 -> ../dm-6
lrwxrwxrwx. 1 root root 7 Sep 28 11:17 /dev/mapper/OCRVoting2 -> ../dm-7
lrwxrwxrwx. 1 root root 7 Sep 28 11:17 /dev/mapper/FRA2 -> ../dm-8
[[email protected] ~]#
三、安装oracleasm驱动包，配置ASM磁盘
[[email protected] mapper]# /etc/init.d/oracleasm createdisk VOL1 /dev/mapper/OCRVoting1
Marking disk "VOL1" as an ASM disk: [ OK ]
[[email protected] mapper]#
[[email protected] mapper]# /etc/init.d/oracleasm createdisk VOL2 /dev/mapper/OCRVoting2
Marking disk "VOL2" as an ASM disk: [ OK ]
[[email protected] mapper]#
[[email protected] mapper]# /etc/init.d/oracleasm createdisk VOL3 /dev/mapper/OCRVoting3
Marking disk "VOL3" as an ASM disk: [ OK ]
[[email protected] mapper]#
[[email protected] mapper]#
[[email protected] mapper]# /etc/init.d/oracleasm createdisk VOL4 /dev/mapper/DATA1
Marking disk "VOL4" as an ASM disk: [ OK ]
You have new mail in /var/spool/mail/root
[[email protected] mapper]#
[[email protected] mapper]# /etc/init.d/oracleasm createdisk VOL5 /dev/mapper/DATA2
Marking disk "VOL5" as an ASM disk: [ OK ]
[[email protected] mapper]#
[[email protected] mapper]# /etc/init.d/oracleasm createdisk VOL6 /dev/mapper/DATA3
Marking disk "VOL6" as an ASM disk: [ OK ]
[[email protected] mapper]#
[[email protected] mapper]# /etc/init.d/oracleasm createdisk VOL7 /dev/mapper/FRA1
Marking disk "VOL7" as an ASM disk: [ OK ]
[[email protected] mapper]#
[[email protected] mapper]# /etc/init.d/oracleasm createdisk VOL8 /dev/mapper/FRA2
Marking disk "VOL8" as an ASM disk: [ OK ]
[[email protected] mapper]#
[[email protected] mapper]# /etc/init.d/oracleasm createdisk VOL9 /dev/mapper/FRA3
Marking disk "VOL9" as an ASM disk: [ OK ]
[[email protected] mapper]#
查看ASM对应磁盘信息：
[[email protected] ~]# /etc/init.d/oracleasm querydisk /dev/mapper/*
Unable to access device "/dev/mapper/control"
Device "/dev/mapper/DATA1" is marked an ASM disk with the label "VOL4"
Device "/dev/mapper/DATA2" is marked an ASM disk with the label "VOL5"
Device "/dev/mapper/DATA3" is marked an ASM disk with the label "VOL6"
Device "/dev/mapper/FRA1" is marked an ASM disk with the label "VOL7"
Device "/dev/mapper/FRA2" is marked an ASM disk with the label "VOL8"
Device "/dev/mapper/FRA3" is marked an ASM disk with the label "VOL9"
Device "/dev/mapper/OCRVoting1" is marked an ASM disk with the label "VOL1"
Device "/dev/mapper/OCRVoting2" is marked an ASM disk with the label "VOL2"
Device "/dev/mapper/OCRVoting3" is marked an ASM disk with the label "VOL3"
[[email protected] ~]#
四、安装cluster与database软件
在安装grid时找不到ASM磁盘组
可能的原因：
1、检查oracleasm lib驱动包是否安装正确
2、检查ASM磁盘组的属性，最后发现是ASM磁盘的所有者是root
[[email protected] ~]# cd /dev/mapper/
[[email protected] mapper]#
[[email protected] mapper]# ll
total 0
crw-rw----. 1 root root 10, 58 Sep 28 16:47 control
lrwxrwxrwx. 1 root root 7 Sep 28 16:47 DATA1 -> ../dm-6
lrwxrwxrwx. 1 root root 7 Sep 28 16:47 DATA2 -> ../dm-2
lrwxrwxrwx. 1 root root 7 Sep 28 16:47 DATA3 -> ../dm-7
lrwxrwxrwx. 1 root root 7 Sep 28 16:47 FRA1 -> ../dm-3
lrwxrwxrwx. 1 root root 7 Sep 28 16:47 FRA2 -> ../dm-0
lrwxrwxrwx. 1 root root 7 Sep 28 16:47 FRA3 -> ../dm-1
lrwxrwxrwx. 1 root root 7 Sep 28 16:47 OCRVoting1 -> ../dm-4
lrwxrwxrwx. 1 root root 7 Sep 28 16:47 OCRVoting2 -> ../dm-5
lrwxrwxrwx. 1 root root 7 Sep 28 16:47 OCRVoting3 -> ../dm-8
[[email protected] mapper]#
手动修改后正常了。
# chown -R grid:oinstall DATA*
# chown -R grid:oinstall FRA*
# chown -R grid:oinstall OCRVoting*
但是，每当操作系统重启后，这个ASM磁盘又恢复为root了
仔细查看就会发现，这些ASM磁盘实际上都是link，它们的真实位置是在上级目录下的dm-*文件：
现在来尝试修改：/dev/dm-*这些文件的属性：
在/etc/udev/rules.d/下添加99-asm.rules文件，文件中添加以下内容：
完成后重启操作系统即可。
其他节点重复以上操作即可。