muneda's diary(2013-05-30)

2013-05-30 晴れ，暑い [長年日記]

_1 Intel CPUマシンでSR-IOVでNICをKVMゲストに使わせる

SR-IOVの設定をしててトラブったのでメモ．

今回igbを使ったので/etc/modprobe.d/igb.confに

option igb max_vfs=7

と書いたのだが，そうするとネットワークが切れる．起動時のコンソール画面を確認するとネットワークサービスの起動に失敗している．うーむ，なんだろうとしばらく悩んだ末，そもそもBIOSでSR-IOVを有効にしていなかったという凡ミス．

ようやくVFがホスト側で認識できるようになったので，KVMゲストにVFをつけて起動しようとするも

# virsh start GUEST
error: Failed to start domain GUEST
error: Requested operation is not valid: PCI device 0000:06:10.0 is not assignable

となる．dmesgにはそれっぽいエラーログは見当たらないが，libvirtdのログにエラーが出ているのを発見．

# /var/log/libvirt/libvirtd.log
2013-05-30 19:17:40.543+0000: 2727: error : pciDeviceIsAssignable:1828 : internal error Device 0000:06:10.0 is behind a switch lacking ACS and cannot be assigned
2013-05-30 19:17:40.545+0000: 2727: error : virFileReadAll:457 : Failed to open file '/var/run/libvirt/qemu/eth2_vf0': No such file or directory
2013-05-30 19:17:40.545+0000: 2727: error : qemuRemoveCgroup:759 : internal error Unable to find cgroup for GUEST
2013-05-30 19:17:40.545+0000: 2727: warning : qemuProcessStop:4366 : Failed to remove cgroup for GUEST

一行目がそれでデバイス関連のエラー．ググってみるとすぐに同じメッセージが見つかり/etc/libvirt/qemu.confでコメントアウトされていた以下の行を有効に．

relaxed_acs_check = 1

libvirtdを再起動させてゲストを起動させようとするもやはりまだ動かない．

ただ先ほどとはメッセージが変わって

# virsh start GUEST
error: Failed to start domain GUEST
error: Unable to read from monitor: Connection reset by peer

# less /var/log/libvirt/libvirtd.log
2013-05-30 19:25:15.752+0000: 3628: error : qemuMonitorIORead:513 : Unable to read from monitor: Connection reset by peer

と出るように．でも意味わからん．

# less /var/log/messages
May 30 15:22:12 HOST kernel: pci-stub 0000:06:10.0: claimed by stub
May 30 15:22:12 HOST kernel: igb 0000:06:00.0: setting MAC 52:54:00:85:51:4c on VF 0
May 30 15:22:12 HOST kernel: igb 0000:06:00.0: Reload the VF driver to make this change effective.
May 30 15:22:12 HOST kernel: igb 0000:06:00.0: The VF MAC address has been set, but the PF device is not up.
May 30 15:22:12 HOST kernel: igb 0000:06:00.0: Bring the PF device up before attempting to use the VF device.
May 30 15:22:12 HOST kernel: device vnet0 entered promiscuous mode
May 30 15:22:12 HOST kernel: virbr0: topology change detected, propagating
May 30 15:22:12 HOST kernel: virbr0: port 2(vnet0) entering forwarding state
May 30 15:22:12 HOST kernel: virbr0: port 2(vnet0) entering disabled state
May 30 15:22:12 HOST kernel: device vnet0 left promiscuous mode
May 30 15:22:12 HOST kernel: virbr0: port 2(vnet0) entering disabled state
May 30 15:22:13 HOST kernel: igb 0000:06:00.0: setting MAC 16:da:c5:22:24:25 on VF 0
May 30 15:22:13 HOST kernel: igb 0000:06:00.0: Reload the VF driver to make this change effective.
May 30 15:22:13 HOST kernel: igb 0000:06:00.0: The VF MAC address has been set, but the PF device is not up.
May 30 15:22:13 HOST kernel: igb 0000:06:00.0: Bring the PF device up before attempting to use the VF device.
May 30 15:22:13 HOST kernel: igbvf 0000:06:10.0: enabling device (0000 -> 0002)
May 30 15:22:13 HOST kernel: igbvf 0000:06:10.0: PF still in reset state, assigning new address. Is the PF interface up?
May 30 15:22:13 HOST kernel: igbvf 0000:06:10.0: PF still resetting
May 30 15:22:13 HOST kernel: igbvf 0000:06:10.0: Intel(R) 82576 Virtual Function
May 30 15:22:13 HOST kernel: udev: renamed network interface eth7 to eth12
May 30 15:22:13 HOST kernel: igbvf 0000:06:10.0: Address: 8e:b3:74:7a:53:43

あれもしかしてPF(Physical Function?)つまり実際のデバイスがリンクアップしてないと使えないのでは？実はこの環境では06:00.0のNICにはケーブルが刺さっていない．05:00.0には刺さっているので，xmlを書き換えてそちらのVFを使うように．

# virsh start GUEST
error: Failed to start domain GUEST
error: Unable to read from monitor: Connection reset by peer

がやっぱりダメ．

# /var/log/messages
May 30 15:26:29 HOST kernel: pci-stub 0000:05:10.0: claimed by stub
May 30 15:26:29 HOST kernel: igb 0000:05:00.0: setting MAC 52:54:00:dc:9d:1d on VF 0
May 30 15:26:29 HOST kernel: igb 0000:05:00.0: Reload the VF driver to make this change effective.
May 30 15:26:29 HOST kernel: device vnet0 entered promiscuous mode
May 30 15:26:29 HOST kernel: virbr0: topology change detected, propagating
May 30 15:26:29 HOST kernel: virbr0: port 2(vnet0) entering forwarding state
May 30 15:26:30 HOST kernel: virbr0: port 2(vnet0) entering disabled state
May 30 15:26:30 HOST kernel: device vnet0 left promiscuous mode
May 30 15:26:30 HOST kernel: virbr0: port 2(vnet0) entering disabled state
May 30 15:26:30 HOST kernel: igb 0000:05:00.0: setting MAC 8e:3b:50:e2:f4:35 on VF 0
May 30 15:26:30 HOST kernel: igb 0000:05:00.0: Reload the VF driver to make this change effective.
May 30 15:26:30 HOST kernel: igbvf 0000:05:10.0: enabling device (0000 -> 0002)
May 30 15:26:30 HOST kernel: igb 0000:05:00.0: VF 0 attempted to override administratively set MAC address
May 30 15:26:30 HOST kernel: Reload the VF driver to resume operations
May 30 15:26:30 HOST kernel: igbvf 0000:05:10.0: Intel(R) 82576 Virtual Function
May 30 15:26:30 HOST kernel: udev: renamed network interface eth7 to eth49
May 30 15:26:30 HOST kernel: igbvf 0000:05:10.0: Address: 8e:3b:50:e2:f4:35

でもVFを使うのはちゃんと動いている様子．

# less /var/log/libvirt/libvirtd.log
2013-05-30 19:26:30.269+0000: 3971: error : qemuMonitorIORead:513 : Unable to read from monitor: Connection reset by peer

そして相変わらず役に立たないエラーしかでない．今度はゲスト側のログを見てみる．

# less /var/log/libvirt/qemu/GUEST.log
2013-05-30 19:33:05.650+0000: starting up
LC_ALL=C ... <略>
No IOMMU found.  Unable to assign device "hostdev0"
qemu-kvm: -device pci-assign,host=05:10.0,id=hostdev0,configfd=24,bus=pci.0,addr=0x6: Device 'pci-assign' could not be initialized
2013-05-30 19:33:05.968+0000: shutting down

あ，IOMMUががが．またも凡ミス．intel_iommu=onを追加してリブート．

# virsh start GUEST
Domain GUEST started

ようやくゲストの起動にこぎつけた．ゲストシステムにログインし追加されたデバイスを確認．

# ll /sys/class/net/eth1/device
lrwxrwxrwx. 1 root root 0 May 30 15:41 /sys/class/net/eth1/device -> ../../../0000:00:06.0
# ll /sys/class/net/eth0/device
lrwxrwxrwx. 1 root root 0 May 30 15:41 /sys/class/net/eth0/device -> ../../../virtio0
# ll /sys/class/net/eth1/device/driver
lrwxrwxrwx. 1 root root 0 May 30 15:41 /sys/class/net/eth1/device/driver -> ../../../bus/pci/drivers/igbvf

ということでeth1に無事デバイスが見えるようになりました．