iT邦幫忙

2026 iThome 鐵人賽

DAY 5
0

Lab

本實驗會整合前面所使用的協定,但會聚焦 EVPN、VRR 和 Active-Active 模式等。

EVPN Layer 2 實作基本設定指令可以用這以下:

  • 配置 VXLAN: 包含 VLAN 到 VNI 的映射,以及 VXLAN 本地隧道 IP 位址(VTEP IP)。
  • 配置 BGP: 設定基礎的 BGP 路由協定。
  • 啟動 EVPN 位址族 (Address Family): 在 BGP 鄰居之間啟用 EVPN 資訊交換(L2VPN EVPN)。讓 BGP 能夠攜帶 MAC 位址和 ARP 資訊。

本實驗的配置可導引至

VRR 配置:

  • 連接至每台伺服器: 一個 Bond 介面或交換器連接埠(Switch port)介面。
  • 連接至每台對等交換器(Peer Switch): 一個或多個介面。為了容納交換器之間更高的頻寬並提供鏈路冗餘,通常使用 Bond 介面。VLAN 介面必須為實體和虛擬介面配置唯一的 IP 地址;交換器在發起 ARP 請求時會使用該唯一地址。

本實驗的配置可導引至

配置 VXLAN Active-Active 模式:

  • Anycast IP
    • 核心是讓兩台實體交換器共用一個 Anycast IP(設在 Loopback 上)
  • clagd
    • 負責動態管理 Anycast IP
  • PROTO_DOWN 狀態
    • 為了防止網路迴圈或流量黑洞。如果兩台交換器設定不一致,介面會卡在 PROTO_DOWN。
  • 對稱模式下的 MAC 地址
    • 網路環境有 L3 路由(EVPN Symmetric),除了 IP 要一樣,Anycast MAC 也要設定

要整合 VXLAN Active-Active 模式,需要設定:MLAGVXLAN 介面路由協定(例如 OSPF 或 BGP),以及 EVPN 或靜態 VXLAN 隧道(static VXLAN tunnels)

本實驗的配置可導引至

EVPN 與 VXLAN 的 Active-active 模式:

  • clagd-vxlan-anycast-ipvxlan-local-tunnelip 參數皆位於兩台對等交換器的 Loopback 區塊(Stanza)下。
  • 兩台交換器都將 Anycast 地址宣告到路由架構(Routed fabric)中。
  • 兩台交換器上的 VNI 配置必須完全相同。
  • Peerlink 必須屬於橋接器(Bridge)的一部分。

針對 EVPN 對稱式(Symmetric) 配置且具備 VXLAN Active-active 模式下的 Type-5 路由,Cumulus Linux 會使用主要 IP 地址宣告(Primary IP Address Advertisement)

實驗架構圖如下

LAB 架構

針對上述詳細配置內容可跳至leaf01 配置

參考資源如下:

Border

Border01

- set:
    system:
      hostname: border01
      global:
        anycast-mac: 44:38:39:FF:00:FF
    bridge:
      domain:
        br_default:
          vlan:
            101,102: {}
    mlag:
      backup:
        10.10.10.64: {}
      peer-ip: linklocal
      priority: 1000
      init-delay: 10
    interface:
      lo:
        ip:
          address:
            10.10.10.63/32: {}
      swp1:
        type: swp
      swp2:
        type: swp
      swp5:
        type: swp
      swp6:
        type: swp
      peerlink:
        description: MLAG Peer-link with border02
        bond:
          mode: lacp
          lacp-rate: fast
          member:
            swp1-2: {}
        type: peerlink
      vlan101:
        type: svi
        vlan: 101
        ip:
          vrf: RED
          address:
            10.1.101.64/24: {}
          vrr:
            state:
              up: {}
            address:
              10.1.101.1/24: {}
      vlan102:
        type: svi
        vlan: 102
        ip:
          vrf: BLUE
          address:
            10.1.102.64/24: {}
          vrr:
            state:
              up: {}
            address:
              10.1.102.1/24: {}
    nve:
      vxlan:
        enable: on
        mlag:
          shared-address: 10.0.1.255
        source:
          address: 10.10.10.63
        arp-nd-suppress: on
    evpn:
      enable: on
    router:
      bgp:
        autonomous-system: 65253
        router-id: 10.10.10.63
    vrf:
      default:
        router:
          bgp:
            address-family:
              ipv4-unicast:
                redistribute:
                  connected:
                    enable: on
            peer-group:
              underlay:
                remote-as: external
                address-family:
                  l2vpn-evpn:
                    enable: on
            neighbor:
              swp5:
                peer-group: underlay
              swp6:
                peer-group: underlay
              peerlink.4094:
                peer-group: underlay
      RED:
        evpn:
          vni:
            '4001': {}
        router:
          bgp:
            autonomous-system: 65253
            router-id: 10.10.10.63
            address-family:
              ipv4-unicast:
                redistribute:
                  static: {}
                route-export:
                  to-evpn: {}
          static:
            10.1.30.0/24:
              via:
                10.1.101.4: {}
      BLUE:
        evpn:
          vni:
            '4002': {}
        router:
          bgp:
            autonomous-system: 65253
            router-id: 10.10.10.63
            address-family:
              ipv4-unicast:
                redistribute:
                  static: {}
                route-export:
                  to-evpn: {}
          static:
            10.1.10.0/24:
              via:
                10.1.102.4: {}
            10.1.20.0/24:
              via:
                10.1.102.4: {}

Border02

- set:
    system:
      hostname: border02
      global:
        anycast-mac: 44:38:39:FF:00:FF
    bridge:
      domain:
        br_default:
          vlan:
            101,102: {}
    mlag:
      backup:
        10.10.10.63: {}
      peer-ip: linklocal
      priority: 2000
      init-delay: 10
    interface:
      lo:
        ip:
          address:
            10.10.10.64/32: {}
      swp1:
        type: swp
      swp2:
        type: swp
      swp5:
        type: swp
      swp6:
        type: swp
      peerlink:
        description: MLAG Peer-link with border01
        bond:
          mode: lacp
          lacp-rate: fast
          member:
            swp1-2: {}
        type: peerlink
      vlan101:
        type: svi
        vlan: 101
        ip:
          vrf: RED
          address:
            10.1.101.65/24: {}
          vrr:
            state:
              up: {}
            address:
              10.1.101.1/24: {}
      vlan102:
        type: svi
        vlan: 102
        ip:
          vrf: BLUE
          address:
            10.1.102.65/24: {}
          vrr:
            state:
              up: {}
            address:
              10.1.102.1/24: {}
    nve:
      vxlan:
        enable: on
        mlag:
          shared-address: 10.0.1.255
        source:
          address: 10.10.10.64
        arp-nd-suppress: on
    evpn:
      enable: on
    router:
      bgp:
        autonomous-system: 65254
        router-id: 10.10.10.64
    vrf:
      default:
        router:
          bgp:
            address-family:
              ipv4-unicast:
                redistribute:
                  connected:
                    enable: on
            peer-group:
              underlay:
                remote-as: external
                address-family:
                  l2vpn-evpn:
                    enable: on
            neighbor:
              swp5:
                peer-group: underlay
              swp6:
                peer-group: underlay
              peerlink.4094:
                peer-group: underlay
      RED:
        evpn:
          vni:
            '4001': {}
        router:
          bgp:
            autonomous-system: 65254
            router-id: 10.10.10.64
            address-family:
              ipv4-unicast:
                redistribute:
                  static: {}
                route-export:
                  to-evpn: {}
          static:
            10.1.30.0/24:
              via:
                10.1.101.4: {}
      BLUE:
        evpn:
          vni:
            '4002': {}
        router:
          bgp:
            autonomous-system: 65254
            router-id: 10.10.10.64
            address-family:
              ipv4-unicast:
                redistribute:
                  static: {}
                route-export:
                  to-evpn: {}
          static:
            10.1.10.0/24:
              via:
                10.1.102.4: {}
            10.1.20.0/24:
              via:
                10.1.102.4: {}

Spine

spine01

- set:
    system:
      hostname: spine01
    interface:
      lo:
        ip:
          address:
            10.10.10.101/32: {}
      swp1:
        type: swp
      swp2:
        type: swp
      swp3:
        type: swp
      swp4:
        type: swp
      swp5:
        type: swp
      swp6:
        type: swp
    router:
      bgp:
        autonomous-system: 65199
        router-id: 10.10.10.101
    vrf:
      default:
        router:
          bgp:
            address-family:
              ipv4-unicast:
                redistribute:
                  connected:
                    enable: on
            peer-group:
              underlay:
                remote-as: external
                address-family:
                  l2vpn-evpn:
                    enable: on
            path-selection:
              multipath:
                aspath-ignore: on
            neighbor:
              swp1:
                peer-group: underlay
              swp2:
                peer-group: underlay
              swp3:
                peer-group: underlay
              swp4:
                peer-group: underlay
              swp5:
                peer-group: underlay
              swp6:
                peer-group: underlay

spine02

- set:
    system:
      hostname: spine02
    interface:
      lo:
        ip:
          address:
            10.10.10.102/32: {}
      swp1:
        type: swp
      swp2:
        type: swp
      swp3:
        type: swp
      swp4:
        type: swp
      swp5:
        type: swp
      swp6:
        type: swp
    router:
      bgp:
        autonomous-system: 65199
        router-id: 10.10.10.102
    vrf:
      default:
        router:
          bgp:
            address-family:
              ipv4-unicast:
                redistribute:
                  connected:
                    enable: on
            peer-group:
              underlay:
                remote-as: external
                address-family:
                  l2vpn-evpn:
                    enable: on
            path-selection:
              multipath:
                aspath-ignore: on
            neighbor:
              swp1:
                peer-group: underlay
              swp2:
                peer-group: underlay
              swp3:
                peer-group: underlay
              swp4:
                peer-group: underlay
              swp5:
                peer-group: underlay
              swp6:
                peer-group: underlay

leaf

leaf01

- set:
    system:
      hostname: leaf01
      global:
        anycast-mac: 44:38:39:FF:00:AA
    bridge:
      domain:
        br_default:
          vlan:
            '10':
              vni:
                '10': {}
            '20':
              vni:
                '20': {}
            '30':
              vni:
                '30': {}
    mlag:
      backup:
        10.10.10.2: {}
      peer-ip: linklocal
      priority: 1000
      init-delay: 10
    interface:
      lo:
        ip:
          address:
            10.10.10.1/32: {}
      swp1:
        type: swp
      swp2:
        type: swp
      swp5:
        type: swp
      swp6:
        type: swp
      swp7:
        type: swp
      swp10:
        type: swp
      swp11:
        type: swp
      bond1:
        bond:
          member:
            swp5: {}
          mlag:
            id: 1
          lacp-bypass: on
        link:
          mtu: 9000
        bridge:
          domain:
            br_default:
              access: 10
      bond2:
        bond:
          member:
            swp6: {}
          mlag:
            id: 2
          lacp-bypass: on
        link:
          mtu: 9000
        bridge:
          domain:
            br_default:
              access: 20
      bond3:
        bond:
          member:
            swp7: {}
          mlag:
            id: 3
          lacp-bypass: on
        link:
          mtu: 9000
        bridge:
          domain:
            br_default:
              access: 30
      peerlink:
        bond:
          member:
            swp10-11: {}
      vlan10:
        type: svi
        vlan: 10
        ip:
          vrf: RED
          address:
            10.1.10.2/24: {}
          vrr:
            state:
              up: {}
            address:
              10.1.10.1/24: {}
      vlan20:
        type: svi
        vlan: 20
        ip:
          vrf: RED
          address:
            10.1.20.2/24: {}
          vrr:
            state:
              up: {}
            address:
              10.1.20.1/24: {}
      vlan30:
        type: svi
        vlan: 30
        ip:
          vrf: BLUE
          address:
            10.1.30.2/24: {}
          vrr:
            state:
              up: {}
            address:
              10.1.30.1/24: {}
    nve:
      vxlan:
        enable: on
        mlag:
          shared-address: 10.0.1.12
        source:
          address: 10.10.10.1
        arp-nd-suppress: on
    evpn:
      enable: on
    router:
      bgp:
        autonomous-system: 65101
        router-id: 10.10.10.1
    vrf:
      default:
        router:
          bgp:
            peer-group:
              underlay:
                remote-as: external
                address-family:
                  l2vpn-evpn:
                    enable: on
            neighbor:
              swp1:
                peer-group: underlay
              swp2:
                peer-group: underlay
              peerlink.4094:
                peer-group: underlay
            address-family:
              ipv4-unicast:
                redistribute:
                  connected:
                    enable: on
      RED:
        evpn:
          vni:
            '4001': {}
        router:
          bgp:
            autonomous-system: 65101
            router-id: 10.10.10.1
            address-family:
              ipv4-unicast:
                redistribute:
                  connected:
                    enable: on
                route-export:
                  to-evpn: {}
      BLUE:
        evpn:
          vni:
            '4002': {}
        router:
          bgp:
            autonomous-system: 65101
            router-id: 10.10.10.1
            address-family:
              ipv4-unicast:
                redistribute:
                  connected:
                    enable: on
                route-export:
                  to-evpn: {}
BGP-EVPN-based layer 2
  1. 配置 VXLAN (資料層面)

    建立隧道封裝資料

    • VLAN 到 VNI 的映射: 將本地 VLAN ID(如 VLAN 10)與全域 VXLAN 識別碼(如 VNI 10)綁定。

      bridge:
        domain:
          br_default:
            vlan:
              '10':
                vni:
                  '10': {}
              '20':
                vni:
                  '20': {}
              '30':
                vni:
                  '30': {}    
      
    • 設定 VTEP IP: 指定 nve (Network Virtual Edge) 的來源位址,通常是交換器的 Loopback IP。

      nve:
        vxlan:
          enable: on
          mlag:
            shared-address: 10.0.1.12
          source:
            address: 10.10.10.1 # VTEP IP
          arp-nd-suppress: on
      
    • NVUE 系統會自動建立一個 vxlan48 裝置並將其加入預設橋接域 (br_default)。

      $ nv show interface vxlan48
                                operational                   applied
      -------------------------  ----------------------------  -------
      type                       vxlan
      vlan                       0
      bridge
        [domain]                 br_default
      ptp
        enable                   off
      parent                     br_default
      ip
        [address]                fe80::98ca:33ff:feb0:2f76/64
      link
        mac-address              9a:ca:33:b0:2f:76
        mtu                      9216
        stats
          in-bytes               26.92 KB
          in-pkts                501
          in-drops               0
          in-errors              0
          out-bytes              158.20 KB
          out-pkts               884
          out-drops              102
          out-errors             0
          carrier-transitions    2
          carrier-up-count       1
          carrier-down-count     1
        protodown                disabled
        oper-status              up
        admin-status             up
        oper-status-last-change  2026/03/15 06:24:27.407
      ifindex                    18
      
  2. 配置 BGP (Underlay 基礎路由)

    EVPN 依賴 BGP 來傳遞路由,這步驟是為了讓交換器之間能互相溝通。

    • 設定 ASN (如 65101) 與 Router ID。

        router:
          bgp:
            autonomous-system: 65101
            router-id: 10.10.10.1
      
    • 建立鄰居,在介面上啟用 BGP 並指定對等體類型。

      vrf:
        default:
          router:
            bgp:
              peer-group:
                underlay:
                  remote-as: external # 使用 eBGP
                  address-family:
                    l2vpn-evpn:
                      enable: on
              neighbor: # 鄰居資訊
                swp1: # 其接到 spine 的介面也會宣告
                  peer-group: underlay
                swp2:
                  peer-group: underlay
                peerlink.4094:
                  peer-group: underlay
                  ...
      
    • 通告路由,將自己的 VTEP IP (10.10.10.1/32) 加入 BGP 網路中。

      vrf:
        default:
          router:
            bgp:
              peer-group:
            ...
              address-family:
                ipv4-unicast: # 將這交換器上直接連接的網段(例如 Loopback 0 的 IP,通常作為 VTEP IP)通告到 BGP 中
                  redistribute:
                    connected:
                      enable: on
      
  3. 啟動 EVPN 位址族 (控制層面)

    這是讓 BGP 開始交換 Layer 2 資訊(MAC/IP)的關鍵。

      evpn: # 全域啟用
        enable: on
    
    ...
      vrf:
        default:
          router:
            bgp:
              peer-group:
                underlay:
                  remote-as: external
                  address-family:
                    l2vpn-evpn:
                      enable: on # BGP 啟用 l2vpn-evpn 位址族  
              neighbor:
                swp1:
                  # 鄰居下介面參考 peer-group,確保與 Spine 或其他 Leaf 的鄰居關係中,允許交換 l2vpn-evpn 類型的路由訊息。
                  peer-group: underlay
                swp2:
                  peer-group: underlay
                peerlink.4094:
                  peer-group: underlay
    
VRR 交換器配置

本實驗會是包含 VRR 的 MLAG 配置。

我們實驗類似於此圖配置,可參考此架構。

MLAG From Nvidia

在實驗中,兩台 Leaf 交換器 (leaf01, leaf02) 透過 peer link 互連。一台伺服器 (server01) 透過 bond (鏈路聚合) 同時連接到 leaf01 和 leaf02。這兩台 Leaf 交換器在邏輯上被伺服器視為單一交換器。

  1. 定義 MLAG 與 Bonding。將所有從雙連線設備(Dual-connected device)連接至 MLAG 配對(MLAG pair)的介面都納入 Bond 中,即使該 Bond 在單台實體交換器上僅包含一條鏈路。

          bond1:
            bond:
              member: # 將介面納入 Bond
                swp5: {}
              mlag:
                id: 1
              lacp-bypass: on
            link:
              mtu: 9000
            bridge:
              domain:
                br_default:
                  access: 10
    
  2. 每個 MLAG 定義唯一的 MLAG ID。

          bond1:
            bond:
              member:
                swp5: {}
              mlag:
                id: 1 # MLAG ID
              lacp-bypass: on
            link:
              mtu: 9000
            bridge:
              domain:
                br_default:
                  access: 10
    
  3. 將建立的 Bond 加入橋接器 (Bridge)

    必須將 MLAG Bond 上配置的所有 VLAN 都加入到橋接器中。這樣一來,當 MLAG Bond 失效時,連接到下方設備的流量才能透過對等鏈路(Peerlink)進行重新定向。

          bond1:
            bond:
              member:
                swp5: {}
              mlag:
                id: 1
              lacp-bypass: on
            link:
              mtu: 9000
            bridge: # 將實驗中定義的 VLAN 10 加至橋接器
              domain:
                br_default:
                  access: 10
    
  4. 建立 inter-chassis bond 與對等鏈路 VLAN

    建立 inter-chassis bond 以及對等鏈路 VLAN(作為 VLAN 子介面),並提供對等鏈路 IP 地址、MLAG Bond 介面、MLAG 系統 MAC 地址以及備援介面(Backup interface)。

    ...
          peerlink: # inter-chassis bond,預設是 peerlink,將對等連結 VLAN 設定為名為 peerlink.4094。使用 peerlink.4094 可確保 VLAN 獨立於橋接器和生成樹轉送決策。
            bond:
              member:
                swp10-11: {}
      ...
        mlag:
          backup: # 定義備援位置
            10.10.10.2: {}
          peer-ip: linklocal
          priority: 1000
          init-delay: 10
    
    
  5. 定義 VLAN 10 的介面與 IP 與定義 VRR

    ...
          vlan10:
            type: svi
            vlan: 10
            ip:
              vrf: RED
              address:
                10.1.10.2/24: {}
              vrr:
                state:
                  up: {}
                address:
                  10.1.10.1/24: {}
    
  6. VRR MAC Address(本實驗沒有特別配置)

    Cumulus Linux 設定了一個全網架構範圍(Fabric-wide)的 MAC 地址,以確保各個 VRR 交換器之間的一致性,這在 EVPN 多網架構環境中特別有用。實驗如下配置來全域更改 VRR MAC 地址,也可以針對特定的 VLAN 覆蓋(Override)全域設定。

    設定範圍是 00:00:5E:00:01:0000:00:5E:00:01:FF 之間的保留範圍內,避免衝突。指令: nv set system global fabric-mac 00:00:5E:00:01:FF。預設為 00:00:5e:00:01:01

  7. Server 端配置 LACP

    每台伺服器必須具備兩個網路介面。交換器端將這些介面配置為執行 LACP 的 Bond(鏈路聚合),伺服器端也必須同步將這兩個介面配置為 Teaming、連接埠聚合 (Port Aggregation)、連接埠群組 (Port Group) 或執行 LACP 的 EtherChannel。

    伺服器可以透過靜態 (Static) 或 DHCP 方式進行配置,其預設閘道 (Gateway) 必須設定為虛擬路由器 (Virtual Router) 的 IP 地址。此預設閘道地址固定不變。此外,伺服器與交換器之間的鏈路應配置為 Active-Active(主動-主動) 模式,以支援 FHRP(第一跳備援協定)。

    為配置內容。

VXLAN Active-Active

下圖是本範例實作相似架構

  1. anycast IP 配置

    
        nve:
          vxlan:
            enable: on
            mlag:
              shared-address: 10.0.1.12 # anycast IP
            source:
              address: 10.10.10.1
            arp-nd-suppress: on
    
    

    原則上要點是 anycast IP。

leaf02

- set:
    system:
      hostname: leaf02
      global:
        anycast-mac: 44:38:39:FF:00:AA
    bridge:
      domain:
        br_default:
          vlan:
            '10':
              vni:
                '10': {}
            '20':
              vni:
                '20': {}
            '30':
              vni:
                '30': {}
    mlag:
      backup:
        10.10.10.1: {}
      peer-ip: linklocal
      priority: 1000
      init-delay: 10
    interface:
      lo:
        ip:
          address:
            10.10.10.2/32: {}
      swp1:
        type: swp
      swp2:
        type: swp
      swp5:
        type: swp
      swp6:
        type: swp
      swp7:
        type: swp
      swp10:
        type: swp
      swp11:
        type: swp
      bond1:
        bond:
          member:
            swp5: {}
          mlag:
            id: 1
          lacp-bypass: on
        link:
          mtu: 9000
        bridge:
          domain:
            br_default:
              access: 10
      bond2:
        bond:
          member:
            swp6: {}
          mlag:
            id: 2
          lacp-bypass: on
        link:
          mtu: 9000
        bridge:
          domain:
            br_default:
              access: 20
      bond3:
        bond:
          member:
            swp7: {}
          mlag:
            id: 3
          lacp-bypass: on
        link:
          mtu: 9000
        bridge:
          domain:
            br_default:
              access: 30
      peerlink:
        bond:
          member:
            swp10-11: {}
      vlan10:
        type: svi
        vlan: 10
        ip:
          vrf: RED
          address:
            10.1.10.3/24: {}
          vrr:
            state:
              up: {}
            address:
              10.1.10.1/24: {}
      vlan20:
        type: svi
        vlan: 20
        ip:
          vrf: RED
          address:
            10.1.20.3/24: {}
          vrr:
            state:
              up: {}
            address:
              10.1.20.1/24: {}
      vlan30:
        type: svi
        vlan: 30
        ip:
          vrf: BLUE
          address:
            10.1.30.3/24: {}
          vrr:
            state:
              up: {}
            address:
              10.1.30.1/24: {}
    nve:
      vxlan:
        enable: on
        mlag:
          shared-address: 10.0.1.12
        source:
          address: 10.10.10.2
        arp-nd-suppress: on
    evpn:
      enable: on
    router:
      bgp:
        autonomous-system: 65102
        router-id: 10.10.10.2
    vrf:
      default:
        router:
          bgp:
            peer-group:
              underlay:
                remote-as: external
                address-family:
                  l2vpn-evpn:
                    enable: on
            neighbor:
              swp1:
                peer-group: underlay
              swp2:
                peer-group: underlay
              peerlink.4094:
                peer-group: underlay
            address-family:
              ipv4-unicast:
                redistribute:
                  connected:
                    enable: on
      RED:
        evpn:
          vni:
            '4001': {}
        router:
          bgp:
            autonomous-system: 65102
            router-id: 10.10.10.2
            address-family:
              ipv4-unicast:
                redistribute:
                  connected:
                    enable: on
                route-export:
                  to-evpn: {}
      BLUE:
        evpn:
          vni:
            '4002': {}
        router:
          bgp:
            autonomous-system: 65102
            router-id: 10.10.10.2
            address-family:
              ipv4-unicast:
                redistribute:
                  connected:
                    enable: on
                route-export:
                  to-evpn: {}

leaf03

- set:
    system:
      hostname: leaf03
      global:
        anycast-mac: 44:38:39:FF:00:BB
    bridge:
      domain:
        br_default:
          vlan:
            '10':
              vni:
                '10': {}
            '20':
              vni:
                '20': {}
            '30':
              vni:
                '30': {}
    mlag:
      backup:
        10.10.10.4: {}
      peer-ip: linklocal
      priority: 1000
      init-delay: 10
    interface:
      lo:
        ip:
          address:
            10.10.10.3/32: {}
      swp1:
        type: swp
      swp2:
        type: swp
      swp5:
        type: swp
      swp6:
        type: swp
      swp7:
        type: swp
      swp10:
        type: swp
      swp11:
        type: swp
      bond1:
        bond:
          member:
            swp5: {}
          mlag:
            id: 1
          lacp-bypass: on
        link:
          mtu: 9000
        bridge:
          domain:
            br_default:
              access: 10
      bond2:
        bond:
          member:
            swp6: {}
          mlag:
            id: 2
          lacp-bypass: on
        link:
          mtu: 9000
        bridge:
          domain:
            br_default:
              access: 20
      bond3:
        bond:
          member:
            swp7: {}
          mlag:
            id: 3
          lacp-bypass: on
        link:
          mtu: 9000
        bridge:
          domain:
            br_default:
              access: 30
      peerlink:
        bond:
          member:
            swp10-11: {}
      vlan10:
        type: svi
        vlan: 10
        ip:
          vrf: RED
          address:
            10.1.10.4/24: {}
          vrr:
            state:
              up: {}
            address:
              10.1.10.1/24: {}
      vlan20:
        type: svi
        vlan: 20
        ip:
          vrf: RED
          address:
            10.1.20.4/24: {}
          vrr:
            state:
              up: {}
            address:
              10.1.20.1/24: {}
      vlan30:
        type: svi
        vlan: 30
        ip:
          vrf: BLUE
          address:
            10.1.30.4/24: {}
          vrr:
            state:
              up: {}
            address:
              10.1.30.1/24: {}
    nve:
      vxlan:
        enable: on
        mlag:
          shared-address: 10.0.1.34
        source:
          address: 10.10.10.3
        arp-nd-suppress: on
    evpn:
      enable: on
    router:
      bgp:
        autonomous-system: 65103
        router-id: 10.10.10.3
    vrf:
      default:
        router:
          bgp:
            peer-group:
              underlay:
                remote-as: external
                address-family:
                  l2vpn-evpn:
                    enable: on
            neighbor:
              swp1:
                peer-group: underlay
              swp2:
                peer-group: underlay
              peerlink.4094:
                peer-group: underlay
            address-family:
              ipv4-unicast:
                redistribute:
                  connected:
                    enable: on
      RED:
        evpn:
          vni:
            '4001': {}
        router:
          bgp:
            autonomous-system: 65103
            router-id: 10.10.10.3
            address-family:
              ipv4-unicast:
                redistribute:
                  connected:
                    enable: on
                route-export:
                  to-evpn: {}
      BLUE:
        evpn:
          vni:
            '4002': {}
        router:
          bgp:
            autonomous-system: 65103
            router-id: 10.10.10.3
            address-family:
              ipv4-unicast:
                redistribute:
                  connected:
                    enable: on
                route-export:
                  to-evpn: {}

leaf04

- set:
    system:
      hostname: leaf04
      global:
        anycast-mac: 44:38:39:FF:00:BB
    bridge:
      domain:
        br_default:
          vlan:
            '10':
              vni:
                '10': {}
            '20':
              vni:
                '20': {}
            '30':
              vni:
                '30': {}
    mlag:
      backup:
        10.10.10.3: {}
      peer-ip: linklocal
      priority: 1000
      init-delay: 10
    interface:
      lo:
        ip:
          address:
            10.10.10.4/32: {}
      swp1:
        type: swp
      swp2:
        type: swp
      swp5:
        type: swp
      swp6:
        type: swp
      swp7:
        type: swp
      swp10:
        type: swp
      swp11:
        type: swp
      bond1:
        bond:
          member:
            swp5: {}
          mlag:
            id: 1
          lacp-bypass: on
        link:
          mtu: 9000
        bridge:
          domain:
            br_default:
              access: 10
      bond2:
        bond:
          member:
            swp6: {}
          mlag:
            id: 2
          lacp-bypass: on
        link:
          mtu: 9000
        bridge:
          domain:
            br_default:
              access: 20
      bond3:
        bond:
          member:
            swp7: {}
          mlag:
            id: 3
          lacp-bypass: on
        link:
          mtu: 9000
        bridge:
          domain:
            br_default:
              access: 30
      peerlink:
        bond:
          member:
            swp10-11: {}
      vlan10:
        type: svi
        vlan: 10
        ip:
          vrf: RED
          address:
            10.1.10.5/24: {}
          vrr:
            state:
              up: {}
            address:
              10.1.10.1/24: {}
      vlan20:
        type: svi
        vlan: 20
        ip:
          vrf: RED
          address:
            10.1.20.5/24: {}
          vrr:
            state:
              up: {}
            address:
              10.1.20.1/24: {}
      vlan30:
        type: svi
        vlan: 30
        ip:
          vrf: BLUE
          address:
            10.1.30.5/24: {}
          vrr:
            state:
              up: {}
            address:
              10.1.30.1/24: {}
    nve:
      vxlan:
        enable: on
        mlag:
          shared-address: 10.0.1.34
        source:
          address: 10.10.10.4
        arp-nd-suppress: on
    evpn:
      enable: on
    router:
      bgp:
        autonomous-system: 65104
        router-id: 10.10.10.4
    vrf:
      default:
        router:
          bgp:
            peer-group:
              underlay:
                remote-as: external
                address-family:
                  l2vpn-evpn:
                    enable: on
            neighbor:
              swp1:
                peer-group: underlay
              swp2:
                peer-group: underlay
              peerlink.4094:
                peer-group: underlay
            address-family:
              ipv4-unicast:
                redistribute:
                  connected:
                    enable: on
      RED:
        evpn:
          vni:
            '4001': {}
        router:
          bgp:
            autonomous-system: 65104
            router-id: 10.10.10.4
            address-family:
              ipv4-unicast:
                redistribute:
                  connected:
                    enable: on
                route-export:
                  to-evpn: {}
      BLUE:
        evpn:
          vni:
            '4002': {}
        router:
          bgp:
            autonomous-system: 65104
            router-id: 10.10.10.4
            address-family:
              ipv4-unicast:
                redistribute:
                  connected:
                    enable: on
                route-export:
                  to-evpn: {}

server

server01

network:
  version: 2
  ethernets:
    enp2s0: {}
    enp2s1: {}
  bonds:
    bond0:
      interfaces: [enp2s0, enp2s1]
      addresses: [10.1.10.101/24]
      parameters:
        mode: 802.3ad
        mii-monitor-interval: 100
        lacp-rate: fast
        transmit-hash-policy: layer3+4
      routes:
        - to: default
          via: 10.1.10.1 # vrr

server02

network:
  version: 2
  ethernets:
    enp2s0: {}
    enp2s1: {}
  bonds:
    bond0:
      interfaces: [enp2s0, enp2s1]
      addresses: [10.1.20.102/24]
      parameters:
        mode: 802.3ad
        mii-monitor-interval: 100
        lacp-rate: fast
        transmit-hash-policy: layer3+4
      routes:
        - to: default
          via: 10.1.20.1

server03

network:
  version: 2
  ethernets:
    enp2s0: {}
    enp2s1: {}
  bonds:
    bond0:
      interfaces: [enp2s0, enp2s1]
      addresses: [10.1.30.103/24]
      parameters:
        mode: 802.3ad
        mii-monitor-interval: 100
        lacp-rate: fast
        transmit-hash-policy: layer3+4
      routes:
        - to: default
          via: 10.1.30.1

server04

network:
  version: 2
  ethernets:
    enp2s0: {}
    enp2s1: {}
  bonds:
    bond0:
      interfaces: [enp2s0, enp2s1]
      addresses: [10.1.10.104/24]
      parameters:
        mode: 802.3ad
        mii-monitor-interval: 100
        lacp-rate: fast
        transmit-hash-policy: layer3+4
      routes:
        - to: default
          via: 10.1.10.1

server05

network:
  version: 2
  ethernets:
    enp2s0: {}
    enp2s1: {}
  bonds:
    bond0:
      interfaces: [enp2s0, enp2s1]
      addresses: [10.1.20.105/24]
      parameters:
        mode: 802.3ad
        mii-monitor-interval: 100
        lacp-rate: fast
        transmit-hash-policy: layer3+4
      routes:
        - to: default
          via: 10.1.20.1

server06

network:
  version: 2
  ethernets:
    enp2s0: {}
    enp2s1: {}
  bonds:
    bond0:
      interfaces: [enp2s0, enp2s1]
      addresses: [10.1.30.106/24]
      parameters:
        mode: 802.3ad
        mii-monitor-interval: 100
        lacp-rate: fast
        transmit-hash-policy: layer3+4
      routes:
        - to: default
          via: 10.1.30.1

驗證

MLAG 與 VXLAN Active-Active 狀態驗證

  1. MLAG 配置資訊

    確認 local-role(為 primary 或 secondary)以及 peer-alive 是否為 True。同時確認 anycast-ip 是否正確顯示為設定的共享 IP。兩台對等 Leaf 間必須顯示完全相同的 Anycast IP。如果這裡不一致,遠端 VTEP 將無法把這兩台交換器視為同一個邏輯節點。

    交換器是否共用同一個虛擬 MAC,這是 Active-Active 環境中避免 MAC 震盪(Flapping)的關鍵。

    ~$ nv show mlag
                    operational            applied
    --------------  ---------------------  ----------
    enable          on                     on
    mac-address     44:38:39:ff:00:aa      auto
    peer-ip         fe80::e86:d2ff:fe61:a  linklocal # Peer Ip Address
    priority        1000                   1000 # Mlag Priority
    init-delay      10                     10
    debug           off                    off # Enable MLAG debugging ? (是否啟用 debugging)
    [backup]        10.10.10.2             10.10.10.2 # Set of MLAG backups (設定備援)
    peer-priority   1000
    backup-active   True
    local-id        0c:11:bb:5e:00:0a
    peer-id         0c:86:d2:61:00:0a
    local-role      primary
    peer-role       secondary
    peer-interface  peerlink.4094
    peer-alive      True
    backup-reason
    anycast-ip      10.0.1.12  # Vxlan Anycast Ip address
    
  2. 顯示交換器上的 MLAG 鄰居訊息:

    確認交換器是否正確學到對等交換器的 MAC 與 IP 資訊。

    $ nv show mlag neighbor
    dynamic
    ==========
          interface  ip-address                 MAC address        vlan-id
        -  ---------  -------------------------  -----------------  -------
        1  vlan30     10.1.30.1                  00:00:5e:00:01:01  30
        2  vlan20     fe80::e032:a2ff:fe76:2b81  e2:32:a2:76:2b:81  20
        3  vlan30     fe80::70c9:fdff:fe06:8397  72:c9:fd:06:83:97  30
        4  vlan10     fe80::84d8:28ff:fefa:90a2  86:d8:28:fa:90:a2  10
    
    
    
    permanent
    ============
            address-family  interface    ip-address                MAC address        vlan-id
        --  --------------  -----------  ------------------------  -----------------  -------
        1   2               vlan30       10.1.30.2                 0c:11:c2:bb:5e:0c  30
        2   2               vlan10       10.1.10.2                 0c:11:c2:bb:5e:0c  10
        3   2               vlan20       10.1.20.2                 0c:11:c2:bb:5e:0c  20
        4   10              vlan30       fe80::e11:c2ff:febb:5e0c  0c:11:c2:bb:5e:0c  30
        5   10              vlan4006_l3  fe80::e11:c2ff:febb:5e0c  0c:11:c2:bb:5e:0c  4006
        6   10              vlan10       fe80::e11:c2ff:febb:5e0c  0c:11:c2:bb:5e:0c  10
        7   10              vlan20       fe80::e11:c2ff:febb:5e0c  0c:11:c2:bb:5e:0c  20
        8   10              vlan4063_l3  fe80::e11:c2ff:febb:5e0c  0c:11:c2:bb:5e:0c  4063
        9   2               vlan30       10.1.30.3                 0c:c2:86:c3:92:6d  30
        10  2               vlan10       10.1.10.3                 0c:c2:86:c3:92:6d  10
        11  2               vlan20       10.1.20.3                 0c:c2:86:c3:92:6d  20
        12  10              vlan30       fe80::ec2:86ff:fec3:926d  0c:c2:86:c3:92:6d  30
        13  10              vlan4006_l3  fe80::ec2:86ff:fec3:926d  0c:c2:86:c3:92:6d  4006
        14  10              vlan10       fe80::ec2:86ff:fec3:926d  0c:c2:86:c3:92:6d  10
        15  10              vlan20       fe80::ec2:86ff:fec3:926d  0c:c2:86:c3:92:6d  20
        16  10              vlan4063_l3  fe80::ec2:86ff:fec3:926d  0c:c2:86:c3:92:6d  4063
    
  3. 顯示 MLAG 行為以及 MLAG 對之間的有無任何不一致之處:

    檢查 Conflicts 欄位是否為 -。如果出現 Proto-Down,通常代表兩台交換器的配置(如 VLAN 或 VNI)不一致。在 VXLAN Active-Active 模式下,兩台 Leaf 交換器必須共用一個 Anycast IP,且 clagd 服務必須正常管理此 IP。

    ~$ clagctl
    The peer is alive
        Our Priority, ID, and Role: 1000 0c:11:bb:5e:00:0a primary
        Peer Priority, ID, and Role: 1000 0c:86:d2:61:00:0a secondary
              Peer Interface and IP: peerlink.4094 fe80::e86:d2ff:fe61:a (linklocal)
                  VxLAN Anycast IP: 10.0.1.12 # 必須顯示設定的共享 IP
                          Backup IP: 10.10.10.2 (active) # 需顯示 active 表示有運作
                        System MAC: 44:38:39:ff:00:aa # MAC 共用
    
    CLAG Interfaces
    Our Interface      Peer Interface     CLAG Id   Conflicts              Proto-Down Reason
    ----------------   ----------------   -------   --------------------   -----------------
              bond1   bond1              1         -                      -
              bond2   bond2              2         -                      -
              bond3   bond3              3         -                      -
            vxlan48   vxlan48            -         -                      -
    
  4. EVPN 路由通告

    在 Active-Active 模式下,兩台 MLAG 交換器都會向 Spine 通告相同的 VNI 資訊,但會使用 Anycast IP 作為下一跳(Next-hop)。

    觀察遠端交換器收到的路由。如果 Active-Active 運作正常,遠端交換器看到的 Type-2(MAC/IP)或 Type-3 路由,其 Next-hop 應該是那組 Anycast IP(10.0.1.12),而不是單台交換器的實體 IP(10.10.10.1)。

    簡單說明帶有 MAC 地址的是 Type-2(主機路由),帶有網段資訊(如 10.1.30.0/24)的是 Type-5(網路預置路由)。

    ~$ sudo vtysh -c "show bgp l2vpn evpn route" | grep "10.0.1.12" -B 5 -A 5
                        10.10.10.1 (leaf01)
                                                0         32768 ?
                        ET:8 RT:65101:4001 Rmac:0c:11:c2:bb:5e:0c
    Route Distinguisher: 10.10.10.1:4
    *> [2]:[0]:[48]:[0c:c2:86:c3:92:6d] RD 10.10.10.1:4
                        10.0.1.12 (leaf01)
                                                          32768 i
                        ET:8 RT:65101:20
    *> [2]:[0]:[48]:[e2:32:a2:76:2b:81] RD 10.10.10.1:4
                        10.0.1.12 (leaf01)
                                                          32768 i
                        ET:8 RT:65101:20
    *> [2]:[0]:[48]:[e2:32:a2:76:2b:81]:[128]:[fe80::e032:a2ff:fe76:2b81] RD 10.10.10.1:4
                        10.0.1.12 (leaf01)
                                                          32768 i
                        ET:8 RT:65101:20
    *> [3]:[0]:[32]:[10.0.1.12] RD 10.10.10.1:4
                        10.0.1.12 (leaf01)
                                                          32768 i
                        ET:8 RT:65101:20
    Route Distinguisher: 10.10.10.1:5
    *> [2]:[0]:[48]:[0c:c2:86:c3:92:6d] RD 10.10.10.1:5
                        10.0.1.12 (leaf01)
                                                          32768 i
                        ET:8 RT:65101:30
    *> [2]:[0]:[48]:[72:c9:fd:06:83:97] RD 10.10.10.1:5
                        10.0.1.12 (leaf01)
                                                          32768 i
                        ET:8 RT:65101:30
    *> [2]:[0]:[48]:[72:c9:fd:06:83:97]:[128]:[fe80::70c9:fdff:fe06:8397] RD 10.10.10.1:5
                        10.0.1.12 (leaf01)
                                                          32768 i
                        ET:8 RT:65101:30
    *> [3]:[0]:[32]:[10.0.1.12] RD 10.10.10.1:5
                        10.0.1.12 (leaf01)
                                                          32768 i
                        ET:8 RT:65101:30
    Route Distinguisher: 10.10.10.1:6
    *> [2]:[0]:[48]:[0c:c2:86:c3:92:6d] RD 10.10.10.1:6
                        10.0.1.12 (leaf01)
                                                          32768 i
                        ET:8 RT:65101:10
    *> [2]:[0]:[48]:[86:d8:28:fa:90:a2] RD 10.10.10.1:6
                        10.0.1.12 (leaf01)
                                                          32768 i
                        ET:8 RT:65101:10
    *> [2]:[0]:[48]:[86:d8:28:fa:90:a2]:[128]:[fe80::84d8:28ff:fefa:90a2] RD 10.10.10.1:6
                        10.0.1.12 (leaf01)
                                                          32768 i
                        ET:8 RT:65101:10
    *> [3]:[0]:[32]:[10.0.1.12] RD 10.10.10.1:6
                        10.0.1.12 (leaf01)
                                                          32768 i
                        ET:8 RT:65101:10
    Route Distinguisher: 10.10.10.2:2
    *> [5]:[0]:[24]:[10.1.30.0] RD 10.10.10.2:2
                        10.10.10.2 (leaf02)
    

BGP 與 EVPN 控制層面驗證

EVPN 依賴 BGP 傳遞 MAC 與 IP 路由,必須確認鄰居關係與路由交換是否正常。

  1. VNI 資訊

    leaf01# show evpn vni
    VNI        Type VxLAN IF              # MACs   # ARPs   # Remote VTEPs  Tenant VRF      VLAN       BRIDGE
    10         L2   vxlan48               5        2        1               RED             10         br_default
    30         L2   vxlan48               5        2        1               BLUE            30         br_default
    20         L2   vxlan48               5        2        1               RED             20         br_default
    4001       L3   vxlan48               5        5        n/a             RED
    4002       L3   vxlan48               5        5        n/a             BLUE
    
  2. EVPN 資訊

    $ nv show evpn
                          operational   applied
    ---------------------  ------------  -------------
    enable                               on
    route-advertise
      nexthop-setting                    system-ip-mac
      svi-ip               off           off
      default-gateway      off           off
    dad
      enable               on            on
      mac-move-threshold   5             5
      move-window          180           180
      duplicate-action     warning-only  warning-only
    [vni]
    multihoming
      enable                             off
      mac-holdtime         1080
      neighbor-holdtime    1080
      startup-delay        180
      startup-delay-timer  --:--:--
      uplink-count         0
      uplink-active        0
    l2vni-count            3
    l3vni-count            2
    
  3. 檢查 BGP 鄰居摘要 (IPv4 Underlay)

    確認通往 Spine 或 Peerlink 的 BGP 狀態為 Established。

    leaf01# show bgp summary
    
    IPv4 Unicast Summary (VRF default):
    BGP router identifier 10.10.10.1, local AS number 65101 vrf-id 0
    BGP table version 16
    RIB entries 21, using 4704 bytes of memory
    Peers 3, using 60 KiB of memory
    Peer groups 1, using 64 bytes of memory
    
    Neighbor              V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
    leaf02(peerlink.4094) 4      65102       569       568        0    0    0 00:24:39           25       11 N/A
    spine01(swp1)         4      65199       579       567        0    0    0 00:24:42           23       11 N/A
    spine02(swp2)         4      65199       577       567        0    0    0 00:24:39           23       11 N/A
    
    Total number of neighbors 3
    
    L2VPN EVPN Summary (VRF default):
    BGP router identifier 10.10.10.1, local AS number 65101 vrf-id 0
    BGP table version 0
    RIB entries 47, using 10528 bytes of memory
    Peers 3, using 60 KiB of memory
    Peer groups 1, using 64 bytes of memory
    
    Neighbor              V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
    leaf02(peerlink.4094) 4      65102       569       568        0    0    0 00:24:39           39       54 N/A
    spine01(swp1)         4      65199       579       567        0    0    0 00:24:42           39       54 N/A
    spine02(swp2)         4      65199       577       567        0    0    0 00:24:39           39       54 N/A
    
    Total number of neighbors 3
    
  4. 檢查 EVPN 位址族 (L2VPN EVPN) 狀態

    確認 BGP 鄰居是否已成功啟動 l2vpn-evpn 能力,並有收發路由紀錄。

    leaf01# show bgp l2vpn evpn summary
    BGP router identifier 10.10.10.1, local AS number 65101 vrf-id 0
    BGP table version 0
    RIB entries 47, using 10528 bytes of memory
    Peers 3, using 60 KiB of memory
    Peer groups 1, using 64 bytes of memory
    
    Neighbor              V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
    leaf02(peerlink.4094) 4      65102       683       682        0    0    0 00:30:23           39       54 N/A
    spine01(swp1)         4      65199       694       682        0    0    0 00:30:26           39       54 N/A
    spine02(swp2)         4      65199       691       682        0    0    0 00:30:23           39       54 N/A
    
    Total number of neighbors 3
    
  5. 查看特定的 EVPN 路由 (Type-2/Type-5)
    檢查是否有通告本地的 MAC/IP (Type-2) 或網路段資訊 (Type-5)

    # show bgp l2vpn evpn route
    BGP table version is 3, local router ID is 10.10.10.1
    Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
    Origin codes: i - IGP, e - EGP, ? - incomplete
    EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id]
    EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
    EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
    EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
    EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]
    
      Network          Next Hop            Metric LocPrf Weight Path
                        Extended Community
    Route Distinguisher: 10.10.10.1:2
    *> [5]:[0]:[24]:[10.1.30.0] RD 10.10.10.1:2
                        10.10.10.1 (leaf01)
                                                0         32768 ?
                        ET:8 RT:65101:4002 Rmac:0c:11:c2:bb:5e:0c
    Route Distinguisher: 10.10.10.1:3
    *> [5]:[0]:[24]:[10.1.10.0] RD 10.10.10.1:3
                        10.10.10.1 (leaf01)
                                                0         32768 ?
                        ET:8 RT:65101:4001 Rmac:0c:11:c2:bb:5e:0c
    ...
    *> [2]:[0]:[48]:[0c:c2:b6:52:c2:bc] RD 10.10.10.3:4
                        10.0.1.34 (spine01)
                                                              0 65199 65103 i
                        RT:65103:20 ET:8 MM:0, sticky MAC
    *  [2]:[0]:[48]:[0c:c2:b6:52:c2:bc] RD 10.10.10.3:4
                        10.0.1.34 (spine02)
                                                              0 65199 65103 i
                        RT:65103:20 ET:8 MM:0, sticky MAC
    *  [2]:[0]:[48]:[0c:c2:b6:52:c2:bc] RD 10.10.10.3:4
                        10.0.1.34 (leaf02)
                                                              0 65102 65199 65103 i
                        RT:65103:20 ET:8 MM:0, sticky MAC
    ...
    *> [3]:[0]:[32]:[10.0.1.34] RD 10.10.10.3:4
                        10.0.1.34 (spine01)
                                                              0 65199 65103 i
                        RT:65103:20 ET:8
    *  [3]:[0]:[32]:[10.0.1.34] RD 10.10.10.3:4
                        10.0.1.34 (spine02)
                                                              0 65199 65103 i
                        RT:65103:20 ET:8
    *  [3]:[0]:[32]:[10.0.1.34] RD 10.10.10.3:4
                        10.0.1.34 (leaf02)
                                                              0 65102 65199 65103 i
                        RT:65103:20 ET:8
    Route Distinguisher: 10.10.10.3:5
    *> [2]:[0]:[48]:[0c:c2:b6:52:c2:bc] RD 10.10.10.3:5
                        10.0.1.34 (spine01)
                                                              0 65199 65103 i
                        RT:65103:30 ET:8 MM:0, sticky MAC
    ...
    *  [5]:[0]:[24]:[10.1.30.0] RD 10.10.10.64:3
                        10.10.10.64 (leaf02)
                                                              0 65102 65199 65254 ?
                        RT:65254:4001 ET:8 Rmac:0c:06:c2:84:c2:b7
    
    Displayed 54 prefixes (132 paths)
    

VXLAN 與 Bridge 資料層面驗證

確認封裝介面是否啟動,以及 MAC 位址是否透過隧道正確學習。

  1. 檢查 NVE (Network Virtual Edge) 狀態
    source address 應為本地 Loopback IP,而 mlag shared-address 應為 Anycast IP。

    $ nv show nve vxlan
                              operational  applied
    ------------------------  -----------  ----------
    enable                    on           on
    mac-learning              off          off
    port                      4789         4789
    arp-nd-suppress           on           on
    mtu                       9216         9216
    flooding
      enable                  on           on
      [head-end-replication]  evpn         evpn
    source
      address                 10.10.10.1   10.10.10.1
    mlag
      shared-address          10.0.1.12    10.0.1.12
    encapsulation
      dscp
        action                derive       derive
    decapsulation
      dscp
        action                derive       derive
    
  2. 查看 Bridge 內的 MAC 位址表
    來自遠端 VTEP 的主機 MAC 應顯示在 vxlan48 (或對應的隧道介面) 後方。

    Interface 欄位下值是 vxlan48,表示出現在 vxlan48 後方的 MAC,代表這些主機位於遠端機櫃,是透過 VXLAN 隧道學過來的。如果是 peerlink,通常代表該流量是為了繞過失效鏈路而走交換器間的通道。顯示在 bond1bond2 等的 MAC,代表是本地直接連線的伺服器。

    $ nv show bridge domain br_default mac-table
    entry-id  MAC address        vlan  interface   remote-dst   src-vni  entry-type    last-update  age
    --------  -----------------  ----  ----------  -----------  -------  ------------  -----------  -------
    1         72:c9:fd:06:83:97  30    bond3                                           0:01:34      0:44:40
    2         0c:11:bb:5e:00:07        bond3                             permanent     0:45:11      0:45:11
    3         86:d8:28:fa:90:a2  10    bond1                                           0:00:17      0:42:54
    4         0c:11:bb:5e:00:05        bond1                             permanent     0:45:11      0:45:11
    5         e2:32:a2:76:2b:81  20    bond2                                           0:00:56      0:44:32
    6         0c:11:bb:5e:00:06        bond2                             permanent     0:45:11      0:45:11
    7         0c:c2:86:c3:92:6d  30    peerlink                          static        0:44:48      0:01:34
    8         0c:c2:86:c3:92:6d  20    peerlink                          static        0:44:48      0:44:48
    9         0c:c2:86:c3:92:6d  10    peerlink                          static        0:44:48      0:44:48
    10        0c:11:bb:5e:00:0a        peerlink                          permanent     0:45:11      0:45:11
    11        02:83:b3:ec:79:d9  10    vxlan48                           extern_learn  0:42:51      0:42:51
    12        7a:7a:fa:91:e7:df  30    vxlan48                           extern_learn  0:42:55      0:42:55
    13        ce:c1:ef:d4:cd:3e  20    vxlan48                           extern_learn  0:43:17      0:43:17
    14        0c:06:c2:84:c2:b7  4063  vxlan48                           extern_learn  0:44:49      0:44:49
    15        0c:c2:b4:c3:bd:83  4063  vxlan48                           extern_learn  0:44:49      0:44:49
    16        0c:c2:b6:52:c2:bc  4063  vxlan48                           extern_learn  0:44:49      0:44:49
    17        0c:20:c3:ab:c3:c6  4063  vxlan48                           extern_learn  0:44:49      0:44:49
    18        0c:c2:86:c3:92:6d  4063  vxlan48                           extern_learn  0:44:49      0:44:49
    19        0c:06:c2:84:c2:b7  4006  vxlan48                           extern_learn  0:44:49      0:44:49
    20        0c:c2:b4:c3:bd:83  4006  vxlan48                           extern_learn  0:44:49      0:44:49
    21        0c:c2:b6:52:c2:bc  4006  vxlan48                           extern_learn  0:44:49      0:44:49
    22        0c:20:c3:ab:c3:c6  4006  vxlan48                           extern_learn  0:44:49      0:44:49
    23        0c:c2:86:c3:92:6d  4006  vxlan48                           extern_learn  0:44:49      0:44:49
    24        0c:c2:b6:52:c2:bc  10    vxlan48                           static        0:44:48      0:44:48
    25        0c:c2:b6:52:c2:bc  30    vxlan48                           static        0:44:48      0:44:48
    26        0c:c2:b6:52:c2:bc  20    vxlan48                           static        0:44:48      0:44:48
    27        0c:20:c3:ab:c3:c6  10    vxlan48                           extern_learn  0:44:48      0:44:48
    28        0c:20:c3:ab:c3:c6  30    vxlan48                           extern_learn  0:44:48      0:44:48
    29        0c:20:c3:ab:c3:c6  20    vxlan48                           extern_learn  0:44:48      0:44:48
    30        0e:84:8e:b2:fe:ff        vxlan48                           permanent     0:45:11      0:45:11
    31        0c:20:c3:ab:c3:c6        vxlan48     10.0.1.34    20       static        0:44:48      0:44:50
    32        0c:c2:b4:c3:bd:83        vxlan48     10.10.10.63  4002     extern_learn  0:44:49      0:44:49
    33        7a:7a:fa:91:e7:df        vxlan48     10.0.1.34    30       extern_learn  0:42:55      0:42:55
    34        0c:c2:b4:c3:bd:83        vxlan48     10.10.10.63  4001     extern_learn  0:44:49      0:44:49
    35        0c:c2:b6:52:c2:bc        vxlan48     10.10.10.4   4001     extern_learn  0:44:49      0:44:49
    36        00:00:00:00:00:00        vxlan48     10.0.1.34    20       permanent     0:44:50      0:07:57
    37        0c:c2:b6:52:c2:bc        vxlan48     10.10.10.4   4002     extern_learn  0:44:49      0:44:49
    38        0c:20:c3:ab:c3:c6        vxlan48     10.10.10.3   4001     extern_learn  0:44:49      0:44:49
    39        02:83:b3:ec:79:d9        vxlan48     10.0.1.34    10       extern_learn  0:42:51      0:42:51
    40        0c:06:c2:84:c2:b7        vxlan48     10.10.10.64  4002     extern_learn  0:44:49      0:44:49
    41        0c:20:c3:ab:c3:c6        vxlan48     10.0.1.34    10       static        0:44:48      0:44:50
    42        0c:c2:86:c3:92:6d        vxlan48     10.10.10.2   4002     extern_learn  0:44:49      0:44:49
    43        0c:c2:b6:52:c2:bc        vxlan48     10.0.1.34    30       static        0:44:48      0:44:50
    44        0c:20:c3:ab:c3:c6        vxlan48     10.0.1.34    30       static        0:44:48      0:44:50
    45        0c:c2:86:c3:92:6d        vxlan48     10.10.10.2   4001     extern_learn  0:44:49      0:44:49
    46        0c:c2:b6:52:c2:bc        vxlan48     10.0.1.34    10       static        0:44:48      0:44:50
    47        0c:20:c3:ab:c3:c6        vxlan48     10.10.10.3   4002     extern_learn  0:44:49      0:44:49
    48        00:00:00:00:00:00        vxlan48     10.0.1.34    10       permanent     0:44:50      0:13:47
    49        ce:c1:ef:d4:cd:3e        vxlan48     10.0.1.34    20       extern_learn  0:43:17      0:43:11
    50        0c:06:c2:84:c2:b7        vxlan48     10.10.10.64  4001     extern_learn  0:44:49      0:44:49
    51        0c:c2:b6:52:c2:bc        vxlan48     10.0.1.34    20       static        0:44:48      0:44:50
    52        00:00:00:00:00:00        vxlan48     10.0.1.34    30       permanent     0:44:50      0:44:50
    53        00:00:5e:00:01:01        br_default                        permanent
    54        44:38:39:ff:00:aa        br_default                        permanent
    55        44:38:39:ff:00:aa  4063  br_default                        permanent     0:45:11      0:45:11
    56        0c:11:c2:bb:5e:0c  4063  br_default                        permanent     0:45:11      0:45:11
    57        00:00:5e:00:01:01  20    br_default                        permanent     0:45:11      0:45:11
    58        0c:11:c2:bb:5e:0c  20    br_default                        permanent     0:45:11      0:45:11
    59        00:00:5e:00:01:01  10    br_default                        permanent     0:45:11      0:45:11
    60        0c:11:c2:bb:5e:0c  10    br_default                        permanent     0:45:11      0:45:11
    61        44:38:39:ff:00:aa  4006  br_default                        permanent     0:45:11      0:45:11
    62        0c:11:c2:bb:5e:0c  4006  br_default                        permanent     0:45:11      0:45:11
    63        00:00:5e:00:01:01  30    
    

上一篇
NVUE - EVPN & VRR
系列文
ToR 的復仇-頂架也能跑馬拉松16
圖片
  熱門推薦
圖片
{{ item.channelVendor }} | {{ item.webinarstarted }} |
{{ formatDate(item.duration) }}
直播中

尚未有邦友留言

立即登入留言