AMD Instinct MI250X

From revRE Wiki

General Availability

Temporary page for trying to bring up the MI250X on general systems. The MI200 is the category for the MI250 OAM and the MI250X OAM, which is the higher performance variant of the MI250.

The MI250X (HPE part # P41858-001) was originally targeted for the Cray EX235a. For PCIe to OAM connection there is currently only one available the nVidia A100 OAM carrier ( https://world.taobao.com/lang/zh-tw/goods/10666585.htm ) and general part 102-D65201-0B making up the Frontier server ( https://www.servethehome.com/behold-the-amd-instinct-mi250x-oam-at-sc21/ )

Vvios; https://support.hpe.com/connect/s/softwaredetails?language=ja&softwareId=MTX_3d12b8be3c1844998007a58aa1&tab=releaseNotes

113-D65201-046-609321 (HFP internal /Semantic version 2.0.0) // AMD_MI200-2.0.0-20220810115835_94954b0.x86_64.rpm

See also MI200 Tuning Guide ; https://rocm.docs.amd.com/en/docs-6.0.0/how-to/tuning-guides/mi200.html

The second block has a matrix named “Hops between two GPUs”, where 1 means the two GPUs are directly connected with XGMI, 2 means both GPUs are linked to the same CPU socket and GPU communications will go through the CPU, and 3 means both GPUs are linked to different CPU sockets so communications will go through both CPU sockets. This number is one for all GPUs in this case since they are all connected to each other through the Infinity Fabric links.

The third block outputs the link types between the GPUs. This can either be “XGMI” for AMD Infinity Fabric links or “PCIE” for PCIe Gen4 links.

And; https://instinct.docs.amd.com/projects/system-acceptance/en/latest/gpus/mi250.html

The MI250 is built on the CDNA 2 architecture (gfx90a) in an OCP Accelerator Module (OAM) form factor. Each MI250 OAM hosts two Graphics Compute Dies (GCDs), each enumerated as an independent GPU by ROCm tools, with 128 GB of HBM2e memory per OAM (64 GB per GCD). GPUs are connected to each other and to the host CPUs through AMD Infinity Fabric™ (xGMI).

see: https://x-dev.pages.jsc.fz-juelich.de/2022/08/01/mi250-first-performances.html where they use the Gigabyte G262-ZO0 server again with EPYC CPU.

AMD MI250 FRONT: https://x.com/wkmyrhang/status/1537917867030478849

AMD MI250 BACK: https://www.reddit.com/media?url=https%3A%2F%2Fpreview.redd.it%2Fis-a-1400-amd-mi250-128gb-hbm2e-still-worth-buying-for-llms-v0-39rabu9gzi1h1.jpg%3Fwidth%3D1080%26crop%3Dsmart%26auto%3Dwebp%26s%3D5030d4b807286b6beebb8b3933a0c5fb2aa7ddbc ( https://www.reddit.com/r/homelab/comments/1tey8ae/is_a_1400_amd_mi250_128gb_hbm2e_still_worth/ )

Probable Card Layout

Referring to https://vengineer.hatenablog.com/entry/2022/05/07/090000 and https://vengineer.hatenablog.com/entry/2022/01/27/070000 you can see that the MI250X has a single link PCIe x16 which is confirmed available on the OAM v2 socket PCIE_TX/RX socket pairs ( https://www.opencompute.org/documents/oai-oam-pinlist-pinmap-r2-0-v1-0-xlsx ) which seems like whats happening here at the lspci for the card https://github.com/amd/amd-lab-notes/discussions/10#discussioncomment-17346151

Which closely matches the press release details for the board

https://www.tweaktown.com/news/88075/amd-unveils-mi250x-mcm-gpu-diagram-58-billion-transistors-6nm-tsmc/index.html

OAM Socket doesnt have additional PCIe lanes, single 16x set is documented. Whilst it does offer 7 other 16 lane SERDES. S1..S7_TX/RX.


https://instinct.docs.amd.com/latest/gpu-arch/mi250.html contradicts this and does state theres dual PCIe lanes for each module.

This page also suggests the PCIe might be running in x8 for each GCD0/1 https://github.com/ROCm/amdgpu/issues/115

[   10.150167] pci 0000:4e:00.0: [1002:740c] type 00 class 0x038000
[   10.150207] pci 0000:4e:00.0: reg 0x10: [mem 0x3a020000000-0x3a02fffffff 64bit pref]
[   10.150219] pci 0000:4e:00.0: reg 0x18: [mem 0x3a030000000-0x3a0301fffff 64bit pref]
[   10.150233] pci 0000:4e:00.0: reg 0x24: [mem 0xf1700000-0xf177ffff]
[   10.150240] pci 0000:4e:00.0: reg 0x30: [mem 0xf1780000-0xf179ffff pref]
[   10.150356] pci 0000:4e:00.0: PME# supported from D1 D2 D3hot D3cold
[   10.150469] pci 0000:4e:00.0: 126.024 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x8 link at 0000:4a:10.0 (capable of 504.112 Gb/s with 32.0 GT/s PCIe x16 link)

Confirmation from the Supermicro for Mi250

https://learn-more.supermicro.com/hubfs/PDFs/Universal.GPU.Template.V5.pdf


Driver Review

d

Aldebaran (MI250/MI250X) Dual-GCD Architecture in the AMDGPU Driver

TL;DR

  • No PCIe bifurcation. No PCI-to-xGMI bridge chip. Both GCDs share one PCI function.
  • The intra-card link between the two GCDs is called WAFL (on-package Infinity Fabric, PCS_GOPX1 registers), which is distinct from external xGMI3x16 links used between separate cards.
  • Single-GCD (MI210) vs dual-GCD (MI250/250X) is discovered at runtime by reading an MMIO register — not from the PCI device ID.

1. PCIe Topology — One Function, Two Dies

amd/amdgpu/amdgpu_drv.c:2177-2181: 

{0x1002, 0x7408, ...  CHIP_ALDEBARAN},  // MI250X
{0x1002, 0x740C, ...  CHIP_ALDEBARAN},  // MI250
{0x1002, 0x740F, ...  CHIP_ALDEBARAN},  // OAM variant
{0x1002, 0x7410, ...  CHIP_ALDEBARAN},  // MI210 (single GCD)

All four device IDs map to a single CHIP_ALDEBARAN enum. The host sees one PCI function. The word "bifurcation" does not appear anywhere in the driver. There is no SR-IOV per GCD, no separate virtual function per die, and no bridge chip between PCIe and the xGMI fabric.

2. Runtime Dual-GCD Detection

amd/amdgpu/gfxhub_v1_2.c:593-631 — During GMC (Graphics Memory Controller) init, the driver reads two hardware registers: 

MC_VM_XGMI_LFB_CNTL:
  PF_MAX_REGION  →  num_physical_nodes = max_region + 1
  PF_LFB_REGION  →  this GCD's physical_node_id (0 or 1)

MC_VM_XGMI_LFB_SIZE:
  PF_LFB_SIZE    →  node_segment_size (per-GCD VRAM partition)

Results stored in adev->gmc.xgmi:

Field MI210 (single-GCD) MI250 / MI250X (dual-GCD)
num_physical_nodes 1 2
physical_node_id 0 0 (GCD 0) or 1 (GCD 1)
node_segment_size Full VRAM Half VRAM each

This gate (num_physical_nodes > 1) drives nearly every dual-GCD code path in the driver.

3. Addressing Both GCDs Through One BAR

Since both GCDs share one PCI function and one MMIO aperture, the driver uses address-space partitioning rather than separate BARs.

VRAM partitioning (amd/amdgpu/gmc_v9_0.c:1660): 

base += adev->gmc.xgmi.physical_node_id * adev->gmc.xgmi.node_segment_size;

GCD 0 owns the lower half of VRAM; GCD 1 owns the upper half — both within the same BAR. No second aperture is required.

Register access uses instance-indexed macros — GET_INST(GC, i) and for_each_inst(i, xcc_mask) — to address each GCD's IP blocks. The IP base-address table (amd/amdgpu/aldebaran_reg_base_init.c:29-54) is indexed [HWIP][instance] and is populated from the hardware IP discovery ROM during boot.

4. Per-Die Asymmetry: TCP Channel Steering

amd/amdgpu/gfx_v9_4_2.c:63-86, selection logic at lines 724-749.

The two GCDs sit in different physical positions on the interposer, so their memory channel routing must be tuned differently. The driver bakes in two sets of compile-time golden register constants: 

static const struct soc15_reg_golden golden_settings_gc_9_4_2_alde_die_0[] = {
    SOC15_REG_GOLDEN_VALUE(GC, 0, regTCP_CHAN_STEER_0, 0x3fffffff, 0x141dc920),
    SOC15_REG_GOLDEN_VALUE(GC, 0, regTCP_CHAN_STEER_1, 0x3fffffff, 0x3b458b93),
    // ... STEER_2 through STEER_5
};

static const struct soc15_reg_golden golden_settings_gc_9_4_2_alde_die_1[] = {
    SOC15_REG_GOLDEN_VALUE(GC, 0, regTCP_CHAN_STEER_0, 0x3fffffff, 0x2591aa38),
    SOC15_REG_GOLDEN_VALUE(GC, 0, regTCP_CHAN_STEER_1, 0x3fffffff, 0x0ac9e88b),
    // ... STEER_2 through STEER_5 (all different values)
};

Selection is a switch (die_id) on physical_node_id read from hardware. This is the clearest driver-level acknowledgment that the two dies are physically asymmetric.

5. xGMI Hive — Grouping Both GCDs

amd/amdgpu/amdgpu_xgmi.c:978-1126

On init of each GCD, the driver calls into the on-die PSP (Platform Security Processor) firmware (amd/amdgpu/ta_xgmi_if.h:32-47):

PSP Command Code Result
TA_COMMAND_XGMI__GET_HIVE_ID 0x02 Both GCDs on same MI250 return identical 64-bit hive_id
TA_COMMAND_XGMI__GET_NODE_ID 0x01 Each GCD returns a unique 64-bit node_id
TA_COMMAND_XGMI__GET_TOPOLOGY_INFO 0x03 Full peer graph for topology programming
TA_COMMAND_XGMI__GET_PEER_LINKS 0x0B Per-peer link count and speed

The driver then calls amdgpu_get_xgmi_hive(), which searches a global xgmi_hive_list by hive_id. The first GCD creates the hive; the second GCD finds it and joins. Both end up on the same amdgpu_hive_info.device_list.

This design is uniform: the xGMI hive abstraction handles both two dies on one card and eight cards in a rack via the same code path.

Fallback when no PSP is available: hive_id = 16, node_id = physical_node_id + 16.

6. WAFL vs. xGMI3x16 — Two Distinct Link Types

amd/amdgpu/amdgpu_xgmi.c:77-107 (register arrays), error dispatch at lines 1437-1455.

WAFL — Intra-Package (between the two GCDs on one MI250)

static const int walf_pcs_err_status_reg_aldebaran[] = {
    smnPCS_GOPX1_PCS_ERROR_STATUS,              // GCD 0 endpoint
    smnPCS_GOPX1_PCS_ERROR_STATUS + 0x100000,   // GCD 1 endpoint
};
  • 2 register entries — one per GCD side of the intra-package link
  • Register family: PCS_GOPX1 ("GOP × 1 lane") — AMD's on-package Infinity Fabric variant
  • Header definitions: amd/include/asic_reg/wafl/wafl2_4_0_0_smn.h and wafl2_4_0_0_sh_mask.h
  • 15 monitored error types: DataLoss, Training, CRC, BERExceeded, TxMetaData, ReplayBuf Parity, DataParity, ReplayFifo Overflow/Underflow, ElasticFifo Overflow, DeskewErr, FCInitTimeout, RecoveryTimeout, ReadySerialTimeout

xGMI3x16 — Inter-Card (between separate MI250 cards in a hive)

static const int xgmi3x16_pcs_err_status_reg_aldebaran[] = {
    smnPCS_XGMI3X16_PCS_ERROR_STATUS,           // Link 0
    smnPCS_XGMI3X16_PCS_ERROR_STATUS + 0x100000, // Link 1
    // ... + 0x200000 through + 0x700000          // Links 2-7
};
  • 8 register entries — up to 8 outbound inter-card links per GCD
  • Register family: PCS_XGMI3X16
  • 26 monitored error types including TxReplayTimeout, RxReplayTimeout, MarginCRCErr
  • Called with is_xgmi_pcs = true vs WAFL's false in amdgpu_xgmi_query_pcs_error_status()

Link Parameters (Aldebaran, IP_VERSION 9.4.2)

Parameter Value
Speed 32 GT/s
Width 16 lanes
Per-link bandwidth ~64 GB/s (32 × 16 / 8)

7. Reset Coordination

amd/amdgpu/aldebaran.c:175-177 (source comment):

"Mode2 reset doesn't need any sync between nodes in XGMI hive, instead launch them together so that they can be completed asynchronously on multiple nodes"


Both GCDs issue their resets concurrently without barriers. The amdgpu_hive_info.task_barrier and reset_domain provide opt-in synchronization for cases that require it.

When aid_mask is non-zero (dual-GCD configuration), the IH (Interrupt Handler) IP block is included in the reset sequence (aldebaran.c:88-93): 

static inline uint32_t aldebaran_get_ip_block_mask(struct amdgpu_device *adev)
{
    uint32_t ip_block_mask = BIT(AMD_IP_BLOCK_TYPE_GFX) |
                             BIT(AMD_IP_BLOCK_TYPE_SDMA);

    if (adev->aid_mask)  /* multi-GCD path */
        ip_block_mask |= BIT(AMD_IP_BLOCK_TYPE_IH);

    return ip_block_mask;
}

8. NBIO — No Bridge, Just Alternate Register Names

amd/amdgpu/nbio_v7_4.c — Aldebaran's NBIO differences are silicon-level register name variants, all accessed via SMN (System Management Network). None implement a bridge:

Register Purpose
mmRCC_DEV0_EPF0_STRAP0_ALDE Revision ID (replaces standard STRAP0)
mmBIF_DOORBELL_INT_CNTL_ALDE RAS interrupt control
mmBIF_INTR_CNTL_ALDE BIF interrupt control
mmBIF_MMSCH1_DOORBELL_RANGE_ALDE VCN instance 1 doorbell range
smnRAS_GLOBAL_STATUS_LO_ALDE RAS global status
smnPARITY_ERROR_STATUS_UNCORR_GRP2_ALDE Uncorrectable parity errors

SDMA instance 4 doorbell offset also differs: 0x3be4 on Aldebaran vs 0x3be0 on Arcturus.

9. KFD / ROCm Topology

amd/amdkfd/kfd_topology.c:861-871

Both GCDs propagate the same hive_id into KFD. Nodes in the same hive with xGMI sharing enabled skip explicit P2P IO-link creation — they are treated as one locality. The location_id field encodes node_id to distinguish the two GCDs to ROCm user-space tools.

Fallback when hive_id = 0 and num_nodes > 1: use pci_dev_id() as the hive ID (same physical card, no PSP).

Architecture Diagram

Host CPU
    |
    |  PCIe x16  (single PCI function — 0x7408 or 0x740C)
    |
  NBIO v7.4  (SMN-addressed, Aldebaran register variants)
    |
    +------- GCD 0  (physical_node_id = 0)
    |          |   XCC, SDMA 0-3, MMHUB inst 0, GFXHub inst 0
    |          |   VRAM: base + 0 * node_segment_size
    |          |
    |        WAFL  (PCS_GOPX1, 32 GT/s x 16 lanes ~ 64 GB/s)
    |          |
    +------- GCD 1  (physical_node_id = 1)
               |   XCC, SDMA 4-7, MMHUB inst 1, GFXHub inst 1
               |   VRAM: base + 1 * node_segment_size
               |
            xGMI3x16 ---> (up to 8 links outward to peer MI250 cards)

Summary Table

Question Answer
PCIe bifurcation? No — one PCI function for both GCDs
PCI-to-xGMI bridge chip? No — direct PCIe to NBIO
Intra-card link name WAFL (PCS_GOPX1 registers)
Inter-card link name xGMI3x16 (PCS_XGMI3X16 registers)
How MI250 vs MI210 detected Runtime: MC_VM_XGMI_LFB_CNTL.PF_MAX_REGION
How GCDs are grouped PSP-supplied hive_id (identical for both GCDs on same card)
Per-die difference in driver TCP channel steering golden registers (regTCP_CHAN_STEER_*)
VRAM addressing Single BAR, offset by physical_node_id × node_segment_size
Reset scope Per-GCD registers, fired concurrently without barriers

Key Source Files

File Topic
amd/amdgpu/amdgpu_xgmi.c Hive assembly, PCS error registers, bandwidth/hops
amd/amdgpu/gfxhub_v1_1.c / gfxhub_v1_2.c LFB/node detection, physical_node_id read
amd/amdgpu/gfx_v9_4_2.c Per-die TCP channel steering golden settings
amd/amdgpu/aldebaran.c IP block masks, reset coordination, aid_mask usage
amd/amdgpu/nbio_v7_4.c NBIO/PCIe register variants for Aldebaran
amd/amdgpu/gmc_v9_0.c VRAM base offset calculation, MMHUB client IDs
amd/amdkfd/kfd_topology.c ROCm/KFD hive_id propagation and topology nodes
amd/amdgpu/ta_xgmi_if.h PSP firmware xGMI command interface
amd/include/asic_reg/wafl/ WAFL register definitions and bit masks

References

  1. https://forum.level1techs.com/t/linux-driver-for-amd-instinct-mi210-mi250x/204395/3
  2. https://www.ebay.com/itm/177571378959
  3. https://www.reddit.com/r/NVIDIA_SXM2PCIE/comments/1b4mjhx/tesla_sxm2345oam_to_pcie/
  4. https://c-payne.com/collections/pcie-gen5-retimer-cards
  5. x
Retrieved from "https://wiki.digitalartisans.org/index.php?title=AMD_Instinct_MI250X&oldid=144"