Infiniband買ったは良いが、
まったく分かってないのがよく分かるｗｗｗ
まずはIPoverIBが出来るようにしてみようと思います。
うちの環境です。
# uname -r 2.6.18-348.12.1.el5 # cat /etc/redhat-release CentOS release 5.9 (Final)
正直、Cent6系を使って、
MLNX_OFED_LINUXを入れたほうがよっぽど簡単だと思います。
http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers
また、
https://www.openfabrics.org/linux-sources.html
こちらを使って頑張って見ましたが、
Kernelが合わないのか入らず断念。。。
なので、yumを駆使して頑張ってみますｗ
参考URL
http://www.altima.co.jp/products/mellanoxtechnologies/ofed.html
http://people.redhat.com/dledford/infiniband_get_started.html
http://docs.oracle.com/cd/E19436-01/820-3522-10/ch4-linux.html
さて頑張っていきましょー
必要なもの（RDMAとかやるなら他にも必要なものがあると思う。。。）
# yum install infiniband-diags ibutils opensm libmlx4 libibumad libibcommon
（RDMAとかやるなら他にも必要なものがあると思う。。。）
サブネットマネージャーを起動
これは必ず1つは必要
# chkconfig opensmd on # /etc/init.d/opensmd start Starting IB Subnet Manager: 1 instances [ OK ]
上がりました。
ステータスとか見てみる
# ibstat ibwarn: [2979] umad_init: can't read ABI version from /sys/class/infiniband_mad/abi_version (No such file or directory): is ib_umad module loaded? ibpanic: [2979] main: can't init UMAD library: No such file or directory
なので、モジュールを読み込ませます
# modprobe ib_umad # ibstat CA 'mlx4_0' CA type: MT26428 Number of ports: 1 Firmware version: 2.7.9110 Hardware version: b0 Node GUID: 0x0002c903000e793c System image GUID: 0x0002c903000e793f Port 1: State: Down Physical state: Polling Rate: 10 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x02510868 Port GUID: 0x0002c903000e793d Link layer: InfiniBand #
いやっほー
見えました。
QDRですが、Rate:10なのは気に入らない。
本当にInfinibandなの？
# cat /sys/class/infiniband/mlx4_0/ports/1/link_layer InfiniBand
うん、Infinibandです。
もう少し詳しくファームなど見てみる。
入れ方は
MellanoxのIBのファームウェアを上げる
さて見てみよう
# mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
#
# mst status
MST modules:
------------
MST PCI module loaded
MST PCI configuration module loaded
MST devices:
------------
/dev/mst/mt26428_pci_cr0 - PCI direct access.
domain:bus:dev.fn=0000:01:00.0 bar=0xfea00000 size=0x100000
Chip revision is: B0
/dev/mst/mt26428_pciconf0 - PCI configuration cycles access.
domain:bus:dev.fn=0000:01:00.0 addr.reg=88 data.reg=92
Chip revision is: B0
#
# flint -d /dev/mst/mt26428_pci_cr0 query
Image type: ConnectX
FW Version: 2.7.9110
Device ID: 26428
Description: Node Port1 Port2 Sys image
GUIDs: 0002c903000e793c 0002c903000e793d 0002c903000e793e 0002c903000e793f
MACs: 0002c90e793c 0002c90e793d
VSD:
PSID: FJT0D90200009
Fujitsuっぽい。。。。
ファームウェア古いｗ
さて、モジュールのバージョン
# modinfo mlx4_core filename: /lib/modules/2.6.18-348.12.1.el5/kernel/drivers/net/mlx4/mlx4_core.ko version: 1.0-ofed1.5.4 license: Dual BSD/GPL description: Mellanox ConnectX HCA low-level driver author: Roland Dreier srcversion: 8E3D6D36D8914BA86B302AC alias: pci:v000015B3d0000100Fsv*sd*bc*sc*i* alias: pci:v000015B3d0000100Esv*sd*bc*sc*i* alias: pci:v000015B3d0000100Dsv*sd*bc*sc*i* alias: pci:v000015B3d0000100Csv*sd*bc*sc*i* alias: pci:v000015B3d0000100Bsv*sd*bc*sc*i* alias: pci:v000015B3d0000100Asv*sd*bc*sc*i* alias: pci:v000015B3d00001009sv*sd*bc*sc*i* alias: pci:v000015B3d00001008sv*sd*bc*sc*i* alias: pci:v000015B3d00001007sv*sd*bc*sc*i* alias: pci:v000015B3d00001006sv*sd*bc*sc*i* alias: pci:v000015B3d00001005sv*sd*bc*sc*i* alias: pci:v000015B3d00001004sv*sd*bc*sc*i* alias: pci:v000015B3d00001003sv*sd*bc*sc*i* alias: pci:v000015B3d00001002sv*sd*bc*sc*i* alias: pci:v000015B3d00001001sv*sd*bc*sc*i* alias: pci:v000015B3d00001000sv*sd*bc*sc*i* alias: pci:v000015B3d00006778sv*sd*bc*sc*i* alias: pci:v000015B3d0000676Esv*sd*bc*sc*i* alias: pci:v000015B3d00006746sv*sd*bc*sc*i* alias: pci:v000015B3d00006764sv*sd*bc*sc*i* alias: pci:v000015B3d0000675Asv*sd*bc*sc*i* alias: pci:v000015B3d00006372sv*sd*bc*sc*i* alias: pci:v000015B3d00006750sv*sd*bc*sc*i* alias: pci:v000015B3d00006368sv*sd*bc*sc*i* alias: pci:v000015B3d0000673Csv*sd*bc*sc*i* alias: pci:v000015B3d00006732sv*sd*bc*sc*i* alias: pci:v000015B3d00006354sv*sd*bc*sc*i* alias: pci:v000015B3d0000634Asv*sd*bc*sc*i* alias: pci:v000015B3d00006340sv*sd*bc*sc*i* depends: vermagic: 2.6.18-348.12.1.el5 SMP mod_unload gcc-4.1 parm: set_4k_mtu:attempt to set 4K MTU to all ConnectX ports (int) parm: debug_level:Enable debug tracing if > 0 (int) parm: block_loopback:Block multicast loopback packets if > 0 (int) parm: msi_x:attempt to use MSI-X if nonzero (int) parm: log_num_mac:Log2 max number of MACs per ETH port (1-7) (int) parm: use_prio:Enable steering by VLAN priority on ETH ports (0/1, default 0) (bool) parm: log_num_qp:log maximum number of QPs per HCA (int) parm: log_num_srq:log maximum number of SRQs per HCA (int) parm: log_rdmarc_per_qp:log number of RDMARC buffers per QP (int) parm: log_num_cq:log maximum number of CQs per HCA (int) parm: log_num_mcg:log maximum number of multicast groups per HCA (int) parm: log_num_mpt:log maximum number of memory protection table entries per HCA (int) parm: log_num_mtt:log maximum number of memory translation table segments per HCA (int) parm: log_mtts_per_seg:Log2 number of MTT entries per segment (1-7) (int) parm: enable_qos:Enable Quality of Service support in the HCA (default: off) (bool) parm: enable_pre_t11_mode:For FCoXX, enable pre-t11 mode if non-zero (default: 0) (int) parm: internal_err_reset:Reset device on internal errors if non-zero (default 1) (int) module_sig: 883f35051dd31a2eba51423e0a33a5c11214d409e22bc88cd3234ff4a60749a31fa37703c4209de40a0e1bfa46573881137ce381813361dca3695ed87
ofed1.5.4
ですね。
もう一個みてみましょ
# modinfo mlx4_ib filename: /lib/modules/2.6.18-348.12.1.el5/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko version: 1.0-ofed1.5.4 license: Dual BSD/GPL description: Mellanox ConnectX HCA InfiniBand driver author: Roland Dreier srcversion: A9F29238DA1556B371B8157 depends: mlx4_core,ib_core,ib_mad vermagic: 2.6.18-348.12.1.el5 SMP mod_unload gcc-4.1 parm: debug_level:Enable debug tracing if > 0 (int) module_sig: 883f35051dd3191eba51423e0a33a5c1129c600a0ae8b36a038656e7be1428b1aa728b797cb0a44f09f487eb57194c26c29f10634280fd136e587671e4
こちらも1.5.4ですね
IPoverIBするのに必要なモジュール
# modinfo ib_ipoib filename: /lib/modules/2.6.18-348.12.1.el5/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko license: Dual BSD/GPL description: IP-over-InfiniBand net driver author: Roland Dreier srcversion: 90FDCFB4E51EBC051E6E941 depends: ib_core,ib_cm,ipv6,ib_sa,ipoib_helper vermagic: 2.6.18-348.12.1.el5 SMP mod_unload gcc-4.1 parm: max_nonsrq_conn_qp:Max number of connected-mode QPs per interface (applied only if shared receive queue is not available) (int) parm: mcast_debug_level:Enable multicast debug tracing if > 0 (int) parm: send_queue_size:Number of descriptors in send queue (int) parm: recv_queue_size:Number of descriptors in receive queue (int) parm: lro:Enable LRO (Large Receive Offload) (bool) parm: lro_max_aggr:LRO: Max packets to be aggregated (default = 64) (int) parm: debug_level:Enable debug tracing if > 0 (int) parm: mc_sendonly_timeout:Enable debug tracing if > 0 (int) module_sig: 883f35051dd3192eba51423e0a33a5c112503d09f595253e79c274925fe2a6f6b9861cc341485a80a0949fd96694e7ccdab4ece8e66b63a9657772699e
こんな感じ
こちらはEtherモードで使うモジュール
# modinfo mlx4_en filename: /lib/modules/2.6.18-348.12.1.el5/kernel/drivers/net/mlx4/mlx4_en.ko version: 1.5.3 (Jan 2011) license: Dual BSD/GPL description: Mellanox ConnectX HCA Ethernet driver author: Liran Liss, Yevgeny Petrilin srcversion: AEA22A080ED4009E1B5E763 alias: pci:v000015B3d0000100Fsv*sd*bc*sc*i* alias: pci:v000015B3d0000100Esv*sd*bc*sc*i* alias: pci:v000015B3d0000100Dsv*sd*bc*sc*i* alias: pci:v000015B3d0000100Csv*sd*bc*sc*i* alias: pci:v000015B3d0000100Bsv*sd*bc*sc*i* alias: pci:v000015B3d0000100Asv*sd*bc*sc*i* alias: pci:v000015B3d00001009sv*sd*bc*sc*i* alias: pci:v000015B3d00001008sv*sd*bc*sc*i* alias: pci:v000015B3d00001007sv*sd*bc*sc*i* alias: pci:v000015B3d00001006sv*sd*bc*sc*i* alias: pci:v000015B3d00001005sv*sd*bc*sc*i* alias: pci:v000015B3d00001004sv*sd*bc*sc*i* alias: pci:v000015B3d00001003sv*sd*bc*sc*i* alias: pci:v000015B3d00001002sv*sd*bc*sc*i* alias: pci:v000015B3d00001001sv*sd*bc*sc*i* alias: pci:v000015B3d00001000sv*sd*bc*sc*i* alias: pci:v000015B3d00006778sv*sd*bc*sc*i* alias: pci:v000015B3d0000676Esv*sd*bc*sc*i* alias: pci:v000015B3d00006746sv*sd*bc*sc*i* alias: pci:v000015B3d00006764sv*sd*bc*sc*i* alias: pci:v000015B3d0000675Asv*sd*bc*sc*i* alias: pci:v000015B3d00006372sv*sd*bc*sc*i* alias: pci:v000015B3d00006750sv*sd*bc*sc*i* alias: pci:v000015B3d00006368sv*sd*bc*sc*i* alias: pci:v000015B3d0000673Csv*sd*bc*sc*i* alias: pci:v000015B3d00006732sv*sd*bc*sc*i* alias: pci:v000015B3d00006354sv*sd*bc*sc*i* alias: pci:v000015B3d0000634Asv*sd*bc*sc*i* alias: pci:v000015B3d00006340sv*sd*bc*sc*i* depends: mlx4_core vermagic: 2.6.18-348.12.1.el5 SMP mod_unload gcc-4.1 parm: inline_thold:treshold for using inline data (int) parm: tcp_rss:Enable RSS for incomming TCP traffic or disabled (0) (uint) parm: udp_rss:Enable RSS for incomming UDP traffic or disabled (0) (uint) parm: num_lro:Number of LRO sessions per ring or disabled (0) (uint) parm: ip_reasm:Allow reassembly of fragmented IP packets (!0) (uint) parm: pfctx:Priority based Flow Control policy on TX[7:0]. Per priority bit mask (uint) parm: pfcrx:Priority based Flow Control policy on RX[7:0]. Per priority bit mask (uint) module_sig: 883f35051dd31a2eba51423e0a33a5c11274c909f735c81822b17e255c262ea74ed96500e4857d09f55d1cd72ead3f9da9dccab9594cbcd664352d8
まぁ、10GのEtherモードを使うことはない。。。かなｗ
さてIPoverIBなのでIPなどを設定しましょ
# vi /etc/sysconfig/network-scripts/ifcfg-ib0 DEVICE=ib0 TYPE=infiniband BOOTPROTO=static BROADCAST=192.168.0.255 IPADDR=192.168.0.100 NETMASK=255.255.255.0 NETWORK=192.168.0.0 ONBOOT=yes #
こんな感じ
さーモジュールを読んでみようー
# modprobe ib_ipoib # lsmod |grep ib ib_ipoib 115233 0 ipoib_helper 35537 2 ib_ipoib mlx4_ib 105085 0 mlx4_core 148873 2 mlx4_ib,mlx4_en ib_umad 50409 0 ib_iser 68161 0 rdma_cm 73301 1 ib_iser ib_cm 71273 2 ib_ipoib,rdma_cm ib_sa 76233 3 ib_ipoib,rdma_cm,ib_cm ib_addr 42697 1 rdma_cm ipv6 437985 24 ib_ipoib,rdma_cm,ib_addr,cnic libcxgbi 91597 1 cxgb3i libiscsi_tcp 53573 3 iscsi_tcp,cxgb3i,libcxgbi libiscsi2 77765 7 be2iscsi,ib_iser,iscsi_tcp,bnx2i,cxgb3i,libcxgbi,libiscsi_tcp scsi_transport_iscsi2 73945 8 be2iscsi,ib_iser,iscsi_tcp,bnx2i,libcxgbi,libiscsi2 ib_mad 72037 4 mlx4_ib,ib_umad,ib_cm,ib_sa ib_core 107841 9 ib_ipoib,mlx4_ib,ib_umad,ib_iser,rdma_cm,ib_cm,iw_cm,ib_sa,ib_mad libata 210769 1 ata_piix scsi_mod 199641 11 be2iscsi,ib_iser,iscsi_tcp,bnx2i,libcxgbi,libiscsi2,scsi_transport_iscsi2,scsi_dh,sg,libata,sd_mod #
いろんなモジュールがありますね
さー
Infiniband君は見えるかなーーー
# ethtool -i ib0
driver: ipoib
version:
firmware-version:
bus-info:
#
# ifconfig ib0
ib0 Link encap:InfiniBand HWaddr 80:04:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
inet addr:192.168.0.100 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST MULTICAST MTU:2044 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:256
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
#
きたでーーー
「Link encap:InfiniBand」
きちんと見えています。
IBのサービスを見てみる
# /etc/init.d/openibd status Low level hardware support loaded: mlx4_ib Upper layer protocol modules: ib_iser ib_ipoib User space access modules: ib_umad Connection management modules: rdma_cm ib_cm iw_cm Configured IPoIB interfaces: ib0 Currently active IPoIB interfaces: ib0
ほー
さてこのままでは再起動するたびにモジュールを読まないといけないので、
# vi /etc/modprobe.conf alias ib0 ib_ipoib alias ib0 ib_umad
追加してあげます
多分、EtherモードとIBの切り替えは
/etc/ofed/mlx4.conf
/etc/modprobe.d/libmlx4.conf
この辺り設定すれば出来るんじゃないかなぁ（まるなげｗ）
[tegaki]接続テストはまだ先だなぁ[/tegaki]