ITPub博客

首页 > Linux操作系统 > Linux操作系统 > IB_Switch交换机SB7890配置及Band网卡

IB_Switch交换机SB7890配置及Band网卡

原创 Linux操作系统 作者:Davis_itpub 时间:2020-09-28 15:36:48 0 删除 编辑

IB_Switch交换机SB7890配置及Band网卡


1.创建软件存放目录

[root@localhost ~]#  mkdir -p /soft


2.上传OS镜像,本案例以RHEL7.7为例

mount -o loop /soft/rhel-server-7.7-x86_64-dvd.iso /mnt/


3.配置yum源

[root@localhost ~]# vi /etc/yum.conf 

[rheldvd]

name=rhel7.7

baseurl=file:///mnt

enabled=1

gpgcheck=0


[main]

………………………………



4.IB交接机驱动下载或上传已有驱动包至soft目录

https://www.mellanox.com/  


驱动包:

MLNX_OFED_LINUX-5.1-2.3.7.1-rhel7.7-x86_64.tgz


5.解压驱动并安装

cd /soft

tar -xvzf MLNX_OFED_LINUX-5.1-2.3.7.1-rhel7.7-x86_64.tgz


6.安装驱动

yum install pciutils perl gtk2 atk cairo gcc-gfortran tcsh openssl-libs libusbx lsof tcl fuse-libs tk -y

cd /soft/MLNX_OFED_LINUX-5.1-2.3.7.1-rhel7.7-x86_64

./mlnxofedinstall 


7.Load HCA驱动

/etc/init.d/openibd restart


8.启动子网管理并设置开机启动

/etc/init.d/opensmd start   ——对于当前SB7890交换不自子网管理,因此只能在OS端开启子网管理服务,若多个节点建议开1——2个节点即可,不建议全部开启;

/sbin/chkconfig opensmd on

/sbin/chkconfig opensmd --list


9.重启OS

reboot


10.查看IB是否正常识别

ifconfig -a


正常识别如下:

ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 2044

Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8).

        infiniband 00:00:10:87:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00  txqueuelen 256  (InfiniBand)

        RX packets 2  bytes 232 (232.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 0  bytes 0 (0.0 B)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


ib1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 4092

Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8).

        infiniband 00:00:18:87:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00  txqueuelen 256  (InfiniBand)

        RX packets 0  bytes 0 (0.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 0  bytes 0 (0.0 B)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

        

11.Band IB配置网络

01. Create bond0 Interface on the server. Update the following file:

touch /etc/sysconfig/network-scripts/ifcfg-bond0


Note: the only meaningful bonding policy in IPoIB is High-Availability (bonding mode number 1,

or "active-backup").

To learn more about the Linux bonding options click here .


Here is an example:

DEVICE=bond0

IPADDR=11.11.11.1

NETWORK=11.11.11.0

NETMASK=255.255.255.0

BROADCAST=11.11.11.255

USERCTL=no

BOOTPROTO=none

NM_CONTROLLED=yes

BONDING_OPTS="mode=active-backup primary=ib0 miimon=100 updelay=100 downdelay=100"

MTU=2044



02.Update/Create ib0 Interface on the server. Update the following file:

touch /etc/sysconfig/network-scripts/ifcfg-ib0

Here is an example:

DEVICE=ib0

TYPE=InfiniBand

NM_CONTROLLED=yes

USERCTL=no

MASTER=bond0

SLAVE=yes

BOOTPROTO=none

PRIMARY=yes



03.Update/Create ib1 Interface on the server. Update the following file:

touch /etc/sysconfig/network-scripts/ifcfg-ib1

Here is an example:

DEVICE=ib1

TYPE=InfiniBand

NM_CONTROLLED=yes

USERCTL=no

MASTER=bond0

SLAVE=yes

BOOTPROTO=none

PRIMARY=no



04.Create/Update the bond.conf file.

touch /etc/modprobe.d/bond.conf

Configure this file as follows:

alias bond0 bonding # Adapter bonding driver

options bond0 max_bonds=2 miimon=100 mode=1 # 100ms fail-over timer. Mode 1 = Active/Backup


05. Restart the network services and the driver.

# /etc/init.d/network restart

# /etc/init.d/openibd restart


06.Verify the bond configuration:

# cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)

Bonding Mode: fault-tolerance (active-backup) (fail_over_mac active)

Primary Slave: ib0 (primary_reselect always)

Currently Active Slave: ib0

MII Status: up

MII Polling Interval (ms): 100

Up Delay (ms): 100

Down Delay (ms): 100

Slave Interface: ib1

MII Status: up

Speed: 40000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: a0:04:03:00:fe:80

Slave queue ID: 0

Slave Interface: ib0

MII Status: up

Speed: 40000 Mbps

Duplex: full

Link Failure Count: 254

Permanent HW addr: a0:04:02:20:fe:80

Slave queue ID: 0



07.Fail Over MAC

Since rdma_cm does not support MAC enslavement, fail_over_mac=1 (or active) should be added

to the bond interface configuration in case of Ethernet interface.

The value active(1) for fail_over_mac policy indicates that the MAC address of the bond should

always be the MAC address of the currently active slave. The MAC address of the slaves is not

changed; instead, the MAC address of the bond changes during a failover.

Changing to fail_over_mac=1 solves this problem as if this not active, the bond MAC jumps from

one interface to another according to the active interface.

Add the bonding options (BONDING_OPT)? "fail_over_mac=1" to /etc/sysconfig/network- scripts/ifcfg-bond0. Note: the default of this parameter is disabled (0).

Here is an example:

DEVICE=bond0

IPADDR=11.11.11.1

NETWORK=11.11.11.0

NETMASK=255.255.255.0

BROADCAST=11.11.11.255

USERCTL=no

BOOTPROTO=none

NM_CONTROLLED=yes

BONDING_OPTS="mode=active-backup primary=ib0 miimon=100 updelay=100 downdelay=100 fail_over_mac=1"

MTU=2044


来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/31520497/viewspace-2724810/,如需转载,请注明出处,否则将追究法律责任。

请登录后发表评论 登录
全部评论

注册时间:2018-02-07

  • 博文量
    171
  • 访问量
    70769