ESXi Host

esxcli vsan cluster

# esxcli vsan cluster
Available Commands:
  get                   Get information about the vSAN cluster that this host is joined to.
  join                  Join the host to a vSAN cluster.
  leave                 Leave the vSAN cluster the host is currently joined to.
  list                  Get information about the vSAN clusters that this host is joined to.
  new                   Create a vSAN cluster with current host joined. A random sub-cluster UUID will be generated.
  restore               Restore the persisted vSAN cluster configuration.

我們可以看到，除了可以獲取當前vSAN cluster的詳細訊息之外，也可以將Host加入或移除vSAN Cluster之外

# esxcli vsan cluster get
Cluster Information
   Enabled: true
   Current Local Time: 2023-08-20T13:38:32Z
   Local Node UUID: 63003af1-640c-87f6-85d3-02007444d88c
   Local Node Type: NORMAL
   Local Node State: AGENT
   Local Node Health State: HEALTHY
   Sub-Cluster Master UUID: 63003ae6-92df-1255-9453-020074c6ad8b
   Sub-Cluster Backup UUID: 63003b06-7d5e-4c7c-4b6f-020074c6bbaa
   Sub-Cluster UUID: 5252eeac-5b6e-92cd-f76b-a890203522c7
   Sub-Cluster Membership Entry Revision: 3
   Sub-Cluster Member Count: 4
   Sub-Cluster Member UUIDs: 63003ae6-92df-1255-9453-020074c6ad8b, 63003b06-7d5e-4c7c-4b6f-020074c6bbaa, 63003aea-e18d-be96-3912-020074ed789f, 63003af1-640c-87f6-85d3-02007444d88c
   Sub-Cluster Member HostNames: sc1-10-78-93-157.eng.vmware.com, sc1-10-78-86-136.eng.vmware.com, sc1-10-78-95-9.eng.vmware.com, sc1-10-78-85-223.eng.vmware.com
   Sub-Cluster Membership UUID: a9440063-9232-c388-fa87-020074c6ad8b
   Unicast Mode Enabled: true
   Maintenance Mode State: OFF
   Config Generation: 59c0eefe-98d4-4167-9638-52b24a862fed 3 2023-08-20T04:55:23.533
   Mode: REGULAR

我們可以看到目前vSAN主機擔任的是AGENT的角色，AGENT角色的作用，可以參考CMMDS那個章節介紹。也能看到主機的健康狀態、Master Node等等訊息

esxcli vsan debug

# esxcli vsan debug
Usage: esxcli vsan debug {cmd} [cmd options]

Available Namespaces:
  disk                  Debug commands for vSAN physical disks
  object                Debug commands for vSAN objects
  resync                Debug commands for vSAN resyncing objects
  advcfg                Debug commands for vSAN advanced configuration options.
  controller            Debug commands for vSAN disk controllers
  evacuation            Debug commands for simulating host, disk or disk group evacuation in various modes and their impact on objects in vSAN cluster
  limit                 Debug commands for vSAN limits
  memory                Debug commands for vSAN memory consumption.
  mob                   Debug commands for vSAN Managed Object Browser Service.
  vmdk                  Debug commands for vSAN VMDKs

我們可以透過debug指令查詢此Host上更細節的資訊，例如Disk的容量、Object、memory、vmdk等等的，例如我們可以透過limit，查看當前Host使用的Capacity、Components

# esxcli vsan debug limit get
   Component Limit Health: green
   Max Components: 750
   Free Components: 742
   Disk Free Space Health: green
   Lowest Free Disk Space: 96 %
   Used Disk Space: 2820082236 bytes
   Used Disk Space (GB): 2.63 GB
   Total Disk Space: 96619986944 bytes
   Total Disk Space (GB): 89.98 GB

也可以觀察物理硬碟的健康狀態

# esxcli vsan debug disk list
UUID: 5229d71d-422e-a985-dc72-33b326a3dcf4
   Name: mpx.vmhba0:C0:T3:L0
   Owner: sc1-10-78-85-223.eng.vmware.com
   Version: 17
   Disk Group: 5229d71d-422e-a985-dc72-33b326a3dcf4
   Disk Tier: Cache
   SSD: true
   In Cmmds: true
   In Vsi: true
   Fault Domain: N/A
   Model: Virtual disk
   Encryption: false
   Compression: false
   Deduplication: false
   Dedup Ratio: N/A
   Overall Health: green
   Metadata Health: green
   Operational Health: green
   Congestion Health:
         State: green
         Congestion Value: 0
         Congestion Area: none
         All Congestion Fields:
   Space Health:

esxcli vsan health cluster

我們能透過此命令，在vSAN Host上面對Cluster進行基本的健康檢查

# esxcli vsan health cluster list
Health Test Name                                    Status
--------------------------------------------------  ------
Overall health findings                             green (OK)
Cluster                                             green
  Advanced vSAN configuration in sync               green
  vSAN daemon liveness                              green
  vSAN Disk Balance                                 green
  Resync operations throttling                      green
  Software version compatibility                    green
  Disk format version                               green
Network                                             green
  Hosts with connectivity issues                    green
  vSAN cluster partition                            green
...

我們也可以指定一個特定的健康檢查，來查看細節，例如我們想查看Disk balance的情況

# esxcli vsan health cluster get -t diskbalance
vSAN Disk Balance        green

Checks the vSAN disk balance status on all hosts.
Ask VMware: http://www.vmware.com/esx/support/askvmware/index.php?eventtype=com.vmware.vsan.health.test.diskbalance

Overview
Metric                    Value
------------------------------------
Average Disk Usage        3 %
Maximum Disk Usage        4 %
Maximum Load Variance     2 %
Average Load Variance     1 %

Disk Balance
Host     Device     Rebalance State     Data To Move (GB)
--------------------------------------------------------------

esxcli vsan iscsi

iSCSI也是vSAN一個很好用的功能，所以有必要說說

# esxcli vsan iscsi
Usage: esxcli vsan iscsi {cmd} [cmd options]

Available Namespaces:
  initiatorgroup        Commands to manipulate vSAN iSCSI target initiator group
  target                Commands for vSAN iSCSI target configuration
  defaultconfig         Operation for default configuration for vSAN iSCSI Target
  homeobject            Commands for the vSAN iSCSI target home object
  status                Enable or disable iSCSI target support, query status.

透過status，我們可以查看iscsi服務已經enable了沒

# esxcli vsan iscsi status get
   Enabled: true

若enable了，可以看看我們現有target的詳細訊息

# esxcli vsan iscsi target list
Alias    iSCSI Qualified Name (IQN)                                Interface  Port  Authentication type  LUNs  Is Compliant  UUID                                  I/O Owner UUID
-------  --------------------------------------------------------  ---------  ----  -------------------  ----  ------------  ------------------------------------  --------------
target1  iqn.1998-01.com.vmware:52d9508e346523dc-fcfb76c951cd3aec  vmk0       3260  No-Authentication       1          true  49ec0063-d44c-d840-dc9f-020074c6bbaa  63003b06-7d5e-4c7c-4b6f-020074c6bbaa
target2  iqn.1998-01.com.vmware:520b1fcc2b101a18-6aaa64b27b3abc64  vmk0       3260  No-Authentication       0          true  25ed0063-d098-6f0b-6b90-020074c6bbaa  63003b06-7d5e-4c7c-4b6f-020074c6bbaa

並觀察Lun的使用狀況

esxcli vsan iscsi target lun list -t target1
ID  Alias        Size  UUID                                  Is Compliant  Status
--  -----  ----------  ------------------------------------  ------------  ------
 0  lun0   102400 MiB  d0ec0063-bc6d-8ff2-84c6-020074c6bbaa          true  online

esxcli vsan maintenancemode

然而，該command沒辦法讓host進入或離開Maintenance mode，若要進入或離開，需透過其他指令

Enter Maintenance mode

esxcli system maintenanceMode set --enable true -m noAction

Exit Maintenance mode

esxcli system maintenanceMode set --enable false

esxcli vsan network

# esxcli vsan network list
Interface
   VmkNic Name: vmk0
   IP Protocol: IP
   Interface UUID: 52219973-1c85-f21e-8990-2871036add42
   Agent Group Multicast Address: 224.2.3.4
   Agent Group IPv6 Multicast Address: ff19::2:3:4
   Agent Group Multicast Port: 23451
   Master Group Multicast Address: 224.1.2.3
   Master Group IPv6 Multicast Address: ff19::1:2:3
   Master Group Multicast Port: 12345
   Host Unicast Channel Bound Port: 12321
   Data-in-Transit Encryption Key Exchange Port: 0
   Multicast TTL: 5
   Traffic Type: vsan

當遇到網路分區錯誤時，可以透過上述指令進行故障排除

vCenter

RVC

rvc管理工具內建於vCenter，當管理人員ssh登入進去vCenter的console後，可以執行

# rvc <user>@<vc-ip>
Welcome to RVC. Try the 'help' command.
0 /
1 localhost/

health summary

#> vsan.health.health_summary /localhost/DataCenter/computers/Vsan1Cluster
Overall health findings: yellow (Cluster health issue)
+------------------------------------------------------+---------+
| Health check                                         | Result  |
+------------------------------------------------------+---------+
| Cluster                                              | Warning |
|   Time is synchronized across hosts and VC           | Passed  |
|   Advanced vSAN configuration in sync                | Passed  |
|   vSAN daemon liveness                               | Passed  |
|   vSAN Disk Balance                                  | Passed  |
|   Resync operations throttling                       | Passed  |
|   vSAN Direct homogeneous disk claiming              | Passed  |
|   vCenter state is authoritative                     | Passed  |
|   vSAN cluster configuration consistency             | Passed  |
|   vSphere cluster members match vSAN cluster members | Passed  |
|   Host Maintenance Mode                              | Warning |
|   Software version compatibility                     | Passed  |
|   Disk format version                                | Passed  |
|   vSAN extended configuration in sync                | Passed  |
+------------------------------------------------------+---------+
...

RVC的規則是指令 - Target，例如我們使用的指令是vsan.health.health_summary，target是cluster，所以返回了整個Cluster的健康狀態

RVC有非常多的指令集，有興趣的讀者可以查閱

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/vsan/vmware-ruby-vsphere-console-command-reference-for-virtual-san.pdf

更多vSAN相關的文章，歡迎參閱我的部落格: https://kaichiachen.github.io/2023/12/03/vsan/esxcli_list/