我が家では今年頭に、念願だったArm ServerReady（現SystemReady SR）認定サーバを導入した
導入したサーバは GIGABYTE の R281-T91 で、Marvell ThunderX2 CN9975 を2基搭載している
（ARMv8.1 28コア112スレッド、2.0GHz x2）
今回はopenSUSE Leap のAarch64版を導入し、OSから見える情報や仮想化基盤として使用した際の情報をまとめた
ハードウェア情報
OSから見えるハードウェア情報は以下。
- lscpu
halni@SUSE-ThunerX2:~> lscpu
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 56
On-line CPU(s) list: 0-55
Vendor ID: Cavium
Model name: ThunderX2-99xx
Model: 1
Thread(s) per core: 1
Core(s) per socket: 28
Socket(s): 2
Stepping: 0x1
BogoMIPS: 400.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics cpuid asimdrdm
Caches (sum of all):
L1d: 1.8 MiB (56 instances)
L1i: 1.8 MiB (56 instances)
L2: 14 MiB (56 instances)
L3: 64 MiB (2 instances)
NUMA:
NUMA node(s): 2
NUMA node0 CPU(s): 0-27
NUMA node1 CPU(s): 28-55
Vulnerabilities:
Gather data sampling: Not affected
Indirect target selection: Not affected
Itlb multihit: Not affected
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Mmio stale data: Not affected
Reg file data sampling: Not affected
Retbleed: Not affected
Spec rstack overflow: Not affected
Spec store bypass: Vulnerable
Spectre v1: Mitigation; __user pointer sanitization
Spectre v2: Vulnerable
Srbds: Not affected
Tsa: Not affected
Tsx async abort: Not affected
Vmscape: Not affected
- numactl
halni@SUSE-ThunerX2:~> numactl -H available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 node 0 size: 32056 MB node 0 free: 2494 MB node 1 cpus: 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 node 1 size: 32114 MB node 1 free: 17023 MB node distances: node 0 1 0: 10 20 1: 20 10
- dmidecode（抜粋）
halni@SUSE-ThunerX2:~> sudo dmidecode
Handle 0x0001, DMI type 1, 27 bytes
System Information
Manufacturer: GIGABYTE
Product Name: R281-T91-00
Version: 0100
Serial Number: ********
UUID: ********
Wake-up Type: Power Switch
SKU Number: Default string
Family: Server
Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
Manufacturer: GIGABYTE
Product Name: MT91-FS1-00
Version: 12340002
Serial Number: ********
Asset Tag: ********
Features:
Board is a hosting board
Board is replaceable
Location In Chassis: Default string
Chassis Handle: 0x0003
Type: Motherboard
Contained Object Handles: 0
Handle 0x0003, DMI type 3, 22 bytes
Chassis Information
Manufacturer: GIGABYTE
Type: Rack Mount Chassis
Lock: Not Present
Version: 01234567
Serial Number: ********
Asset Tag: ********
Boot-up State: Safe
Power Supply State: Safe
Thermal State: Safe
Security Status: None
OEM Information: 0x00000000
Height: Unspecified
Number Of Power Cords: 1
Contained Elements: 0
SKU Number: Default string
~~~~~~~~~
Handle 0x0026, DMI type 4, 48 bytes
Processor Information
Socket Designation: Socket 0
Type: Central Processor
Family: ARM
Manufacturer: Cavium Inc.
ID: F1 0A 1F 43 00 00 00 00
Signature: Implementor 0x43, Variant 0x1, Architecture 15, Part 0x0af, Revision 1
Version: Cavium ThunderX2(R) CPU CN9975 v2.1 @ 2.0GHz
Voltage: 0.8 V
External Clock: 33 MHz
Max Speed: 2500 MHz
Current Speed: 2000 MHz
Status: Populated, Enabled
Upgrade: Other
L1 Cache Handle: 0x0027
L2 Cache Handle: 0x0029
L3 Cache Handle: 0x002A
Serial Number: ********
Asset Tag: To be filled by O.E.M.
Part Number: CN9975-2000BG4077-Y21-G
Core Count: 28
Core Enabled: 28
Thread Count: 28
Characteristics:
64-bit capable
Multi-Core
Hardware Thread
Execute Protection
Enhanced Virtualization
Power/Performance Control
~~~~~~~~~
Handle 0x0043, DMI type 4, 48 bytes
Processor Information
Socket Designation: Socket 1
Type: Central Processor
Family: ARM
Manufacturer: Cavium Inc.
ID: F1 0A 1F 43 00 00 00 00
Signature: Implementor 0x43, Variant 0x1, Architecture 15, Part 0x0af, Revision 1
Version: Cavium ThunderX2(R) CPU CN9975 v2.1 @ 2.0GHz
Voltage: 0.8 V
External Clock: 33 MHz
Max Speed: 2500 MHz
Current Speed: 2000 MHz
Status: Populated, Enabled
Upgrade: Other
L1 Cache Handle: 0x0044
L2 Cache Handle: 0x0046
L3 Cache Handle: 0x0047
Serial Number: ********
Asset Tag: To be filled by O.E.M.
Part Number: CN9975-2000BG4077-Y21-G
Core Count: 28
Core Enabled: 28
Thread Count: 28
Characteristics:
64-bit capable
Multi-Core
Hardware Thread
Execute Protection
Enhanced Virtualization
Power/Performance Control
~~~~~~~~~
- lspci
halni@SUSE-ThunerX2:~> sudo lspci 00:00.0 Host bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Host bridge 00:01.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 00:02.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 00:03.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 00:04.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 00:05.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 00:06.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 00:07.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 00:08.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 00:09.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 00:0a.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 00:0b.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 00:0c.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 00:0d.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 00:0e.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 00:0f.0 USB controller: Broadcom Inc. and subsidiaries CN99xx [ThunderX2] Integrated USB 3.0 xHCI Host Controller 00:0f.1 USB controller: Broadcom Inc. and subsidiaries CN99xx [ThunderX2] Integrated USB 3.0 xHCI Host Controller 00:10.0 SATA controller: Broadcom Inc. and subsidiaries CN99xx [ThunderX2] Integrated AHCI/SATA 3 Host Controller 00:10.1 SATA controller: Broadcom Inc. and subsidiaries CN99xx [ThunderX2] Integrated AHCI/SATA 3 Host Controller 0b:00.0 Ethernet controller: QLogic Corp. FastLinQ QL41000 Series 10/25/40/50GbE Controller (rev 02) 0b:00.1 Ethernet controller: QLogic Corp. FastLinQ QL41000 Series 10/25/40/50GbE Controller (rev 02) 0c:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 04) 0d:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41) 0e:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02) 80:00.0 Host bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Host bridge 80:01.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 80:02.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 80:03.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 80:04.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 80:05.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 80:06.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 80:07.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 80:08.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 80:09.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 80:0a.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 80:0b.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 80:0c.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 80:0d.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 80:0e.0 PCI bridge: Cavium, Inc. CN99xx [ThunderX2] Integrated PCI Express RP Bridge 80:0f.0 USB controller: Broadcom Inc. and subsidiaries CN99xx [ThunderX2] Integrated USB 3.0 xHCI Host Controller 80:0f.1 USB controller: Broadcom Inc. and subsidiaries CN99xx [ThunderX2] Integrated USB 3.0 xHCI Host Controller 80:10.0 SATA controller: Broadcom Inc. and subsidiaries CN99xx [ThunderX2] Integrated AHCI/SATA 3 Host Controller 80:10.1 SATA controller: Broadcom Inc. and subsidiaries CN99xx [ThunderX2] Integrated AHCI/SATA 3 Host Controller
- lsmod
halni@SUSE-ThunerX2:~> lsmod Module Size Used by tun 61440 4 xt_CHECKSUM 12288 1 xt_MASQUERADE 16384 3 xt_conntrack 12288 1 ipt_REJECT 12288 2 xt_tcpudp 16384 9 nf_nat_tftp 12288 0 nf_conntrack_tftp 20480 3 nf_nat_tftp bridge 352256 0 stp 12288 1 bridge llc 16384 2 bridge,stp af_packet 61440 2 nft_fib_inet 12288 1 nft_fib_ipv4 12288 1 nft_fib_inet nft_fib_ipv6 12288 1 nft_fib_inet nft_fib 12288 3 nft_fib_ipv6,nft_fib_ipv4,nft_fib_inet nft_reject_inet 12288 15 nf_reject_ipv4 16384 2 nft_reject_inet,ipt_REJECT nf_reject_ipv6 24576 1 nft_reject_inet nft_reject 12288 1 nft_reject_inet nft_ct 24576 8 nft_chain_nat 12288 3 nf_tables 278528 644 nft_ct,nft_reject_inet,nft_fib_ipv6,nft_fib_ipv4,nft_chain_nat,nft_reject,nft_fib,nft_fib_inet ebtable_nat 12288 0 ebtable_broute 12288 0 ip6table_nat 12288 1 ip6table_mangle 12288 1 ip6table_raw 12288 0 ip6table_security 12288 0 iptable_nat 12288 1 nf_nat 57344 5 ip6table_nat,nf_nat_tftp,nft_chain_nat,iptable_nat,xt_MASQUERADE nf_conntrack 184320 6 xt_conntrack,nf_nat,nf_conntrack_tftp,nft_ct,nf_nat_tftp,xt_MASQUERADE nf_defrag_ipv6 24576 1 nf_conntrack nf_defrag_ipv4 12288 1 nf_conntrack iptable_mangle 12288 1 iptable_raw 12288 0 iptable_security 12288 0 rfkill 32768 3 ip_set 61440 0 nfnetlink 20480 3 nf_tables,ip_set ebtable_filter 12288 0 ebtables 40960 3 ebtable_nat,ebtable_filter,ebtable_broute ip6table_filter 12288 1 ip6_tables 32768 5 ip6table_filter,ip6table_raw,ip6table_nat,ip6table_mangle,ip6table_security iptable_filter 12288 1 qrtr 45056 4 rpcrdma 311296 0 sunrpc 688128 1 rpcrdma rdma_ucm 32768 0 ib_iser 57344 0 libiscsi 77824 1 ib_iser scsi_transport_iscsi 155648 2 ib_iser,libiscsi rdma_cm 126976 3 rpcrdma,ib_iser,rdma_ucm iw_cm 53248 1 rdma_cm ib_cm 131072 1 rdma_cm qedr 131072 0 ib_uverbs 172032 2 rdma_ucm,qedr ib_core 450560 8 rdma_cm,rpcrdma,iw_cm,ib_iser,rdma_ucm,ib_uverbs,qedr,ib_cm ax88796b 12288 1 binfmt_misc 24576 1 nls_iso8859_1 12288 1 nls_cp437 16384 1 vfat 20480 1 fat 90112 1 vfat qede 204800 1 qedr asix 57344 0 usbnet 57344 1 asix phylink 69632 1 asix ses 16384 0 qed 864256 2 qede,qedr mii 20480 2 usbnet,asix i2c_smbus 20480 0 enclosure 20480 1 ses ipmi_ssif 45056 0 tiny_power_button 12288 0 button 28672 0 ipmi_devintf 16384 0 ipmi_msghandler 81920 2 ipmi_devintf,ipmi_ssif joydev 28672 0 thunderx2_pmu 24576 0 thermal 24576 0 loop 32768 0 fuse 172032 5 efi_pstore 12288 0 dm_mod 196608 0 dmi_sysfs 24576 0 ip_tables 32768 5 iptable_filter,iptable_security,iptable_raw,iptable_nat,iptable_mangle x_tables 57344 18 ebtables,ip6table_filter,xt_conntrack,ip6table_raw,iptable_filter,iptable_security,ip6table_nat,xt_tcpudp,xt_CHECKSUM,ip6_tables,ipt_REJECT,iptable_raw,ip_tables,iptable_nat,ip6table_mangle,ip6table_security,xt_MASQUERADE,iptable_mangle xfs 1757184 1 libcrc32c 12288 4 nf_conntrack,nf_nat,nf_tables,xfs hid_generic 12288 0 usbhid 73728 0 onboard_usb_hub 20480 0 ahci 57344 4 libahci 57344 1 ahci mpt3sas 454656 0 libata 417792 2 libahci,ahci raid_class 12288 1 mpt3sas scsi_transport_sas 49152 2 ses,mpt3sas sd_mod 69632 4 scsi_dh_emc 12288 0 scsi_dh_rdac 16384 0 scsi_dh_alua 24576 0 t10_pi 16384 1 sd_mod xhci_pci 28672 0 crct10dif_ce 16384 1 xhci_pci_renesas 20480 1 xhci_pci polyval_ce 12288 0 crc64_rocksoft_generic 12288 1 polyval_generic 12288 1 polyval_ce xhci_hcd 331776 1 xhci_pci crc64_rocksoft 16384 1 t10_pi ghash_ce 24576 0 gf128mul 16384 2 polyval_generic,ghash_ce sg 45056 0 crc64 16384 2 crc64_rocksoft,crc64_rocksoft_generic sm4 12288 0 usbcore 385024 6 xhci_hcd,usbnet,usbhid,xhci_pci,onboard_usb_hub,asix sha2_ce 20480 0 scsi_mod 303104 13 ses,scsi_dh_emc,scsi_transport_sas,sd_mod,raid_class,scsi_transport_iscsi,scsi_dh_alua,ib_iser,libiscsi,libata,sg,scsi_dh_rdac,mpt3sas sha256_arm64 24576 1 sha2_ce sha1_ce 12288 0 ast 86016 1 i2c_algo_bit 16384 1 ast usb_common 24576 2 xhci_hcd,usbcore gpio_xlp 12288 0 i2c_xlp9xx 20480 0 efivarfs 24576 1 aes_ce_blk 32768 0 aes_ce_cipher 12288 1 aes_ce_blk
- ipmitool
halni@SUSE-ThunerX2:~> sudo ipmitool chassis status System Power : on Power Overload : false Power Interlock : inactive Main Power Fault : false Power Control Fault : false Power Restore Policy : previous Last Power Event : Chassis Intrusion : inactive Front-Panel Lockout : inactive Drive Fault : false Cooling/Fan Fault : false Sleep Button Disable : not allowed Diag Button Disable : allowed Reset Button Disable : allowed Power Button Disable : allowed Sleep Button Disabled: false Diag Button Disabled : false Reset Button Disabled: false Power Button Disabled: false halni@SUSE-ThunerX2:~>
参考までに消費電力はこんな感じ。
KVMホストとして利用する際の情報
qemu-kvmを使用することで、（当然のことながら）KVMホストとして動作可能である
halni@SUSE-ThunerX2:~> rpm -qf `which qemu-system-aarch64` qemu-arm-8.2.10-150600.3.40.1.aarch64 halni@SUSE-ThunerX2:~> zypper info qemu-arm Information for package qemu-arm: --------------------------------- Repository : Update repository with updates from SUSE Linux Enterprise 15 Name : qemu-arm Version : 8.2.10-150600.3.40.1 Arch : aarch64 Vendor : SUSE LLC <https://www.suse.com/> Installed Size : 56.5 MiB Installed : Yes Status : up-to-date Source package : qemu-8.2.10-150600.3.40.1.src Upstream URL : https://www.qemu.org/ Summary : Machine emulator and virtualizer for ARM architectures
選択可能なCPUモデルは以下の通り
halni@SUSE-ThunerX2:~> sudo /usr/bin/qemu-kvm --cpu ? Available CPUs: a64fx arm1026 arm1136 arm1136-r2 arm1176 arm11mpcore arm926 arm946 cortex-a15 cortex-a35 cortex-a53 cortex-a55 cortex-a57 cortex-a7 cortex-a710 cortex-a72 cortex-a76 cortex-a8 cortex-a9 cortex-m0 cortex-m3 cortex-m33 cortex-m4 cortex-m55 cortex-m7 cortex-r5 cortex-r52 cortex-r5f host max neoverse-n1 neoverse-n2 neoverse-v1 pxa250 pxa255 pxa260 pxa261 pxa262 pxa270-a0 pxa270-a1 pxa270 pxa270-b0 pxa270-b1 pxa270-c0 pxa270-c5 sa1100 sa1110 ti925t
virt-host-validateから、本システムはKVM仮想化支援機構が動作していることが分かる
- virt-host-validate
halni@SUSE-ThunerX2:~> sudo virt-host-validate QEMU: Checking if device /dev/kvm exists : PASS QEMU: Checking if device /dev/kvm is accessible : PASS ★ QEMU: Checking if device /dev/vhost-net exists : PASS QEMU: Checking if device /dev/net/tun exists : PASS QEMU: Checking for cgroup 'cpu' controller support : PASS QEMU: Checking for cgroup 'cpuacct' controller support : PASS QEMU: Checking for cgroup 'cpuset' controller support : PASS QEMU: Checking for cgroup 'memory' controller support : PASS QEMU: Checking for cgroup 'devices' controller support : PASS QEMU: Checking for cgroup 'blkio' controller support : PASS QEMU: Checking for device assignment IOMMU support : PASS QEMU: Checking if IOMMU is enabled by kernel : PASS ★ QEMU: Checking for secure guest support : WARN (Unknown if this platform has Secure Guest support) LXC: Checking for Linux >= 2.6.26 : PASS LXC: Checking for namespace ipc : PASS LXC: Checking for namespace mnt : PASS LXC: Checking for namespace pid : PASS LXC: Checking for namespace uts : PASS LXC: Checking for namespace net : PASS LXC: Checking for namespace user : PASS LXC: Checking for cgroup 'cpu' controller support : PASS LXC: Checking for cgroup 'cpuacct' controller support : PASS LXC: Checking for cgroup 'cpuset' controller support : PASS LXC: Checking for cgroup 'memory' controller support : PASS LXC: Checking for cgroup 'devices' controller support : PASS LXC: Checking for cgroup 'freezer' controller support : FAIL (Enable 'freezer' in kernel Kconfig file or mount/enable cgroup controller in your system) LXC: Checking for cgroup 'blkio' controller support : PASS LXC: Checking if device /sys/fs/fuse/connections exists : PASS halni@SUSE-ThunerX2:~>
なおARMv8.2からL2ゲスト（Nested KVM）が動くが、ThunderX2はARMv8.1のため利用できない。残念…
virtマシンタイプ・sbsaマシンそれぞれのKVMゲストが動作していることが分かる
halni@SUSE-ThunerX2:~> sudo virsh list --all
Id Name State
-------------------------------
18 sbsa-arm-suse running
19 virt-arm-suse running
halni@SUSE-ThunerX2:~> sudo virsh dumpxml virt-arm-suse | grep "type arch"
<type arch='aarch64' machine='virt-8.2'>hvm</type> ★ virt-8.2
halni@SUSE-ThunerX2:~> sudo virsh dumpxml sbsa-arm-suse | grep "type arch"
<type arch='aarch64' machine='sbsa-ref'>hvm</type> ★ sbsa-ref
halni@SUSE-ThunerX2:~>
