GPU adapter information
Instances For
Equations
Equations
- One or more equations did not get rendered due to their size.
Instances For
Get the number of available GPU adapters
Get information about a specific GPU adapter by index
Get the default GPU device. This is a simplified wrapper that gets the first available adapter and creates a device from it.
Get a GPU device with advanced features enabled (Subgroups, Float16). This is required for high-performance compute tasks like subgroup matrix multiplication.
Get a GPU device from a specific adapter index. Use this to select which GPU to use in multi-GPU systems. Example:
let device ← getDeviceByIndex inst 0 -- Use first GPU
let device ← getDeviceByIndex inst 1 -- Use second GPU
Check if the device was created with subgroup support.
Returns true if subgroupAdd and related operations are available.
metal_replacer STEP 1 PoC: the name/props of the MTLDevice behind this WGPUDevice (via Dawn's GetMTLDevice). Proves the Metal interop is live — the foundation for swapping in llama.cpp's Metal kernels. See METAL_REPLACER_INTEGRATION.md.
DG_GPUBUSY: read+reset the compute-pass count + host (record/finish) vs GPU (submit+wait) time split accumulated since the last read. Quantifies encoder-per-dispatch host overhead.
DEVPLAN M1: the most recent Tint-MSL dump captured in-process by the HESPER_DUMP_MSL logging
callback (empty if nothing dumped yet / env not set). Feed to mslOccupancyProbe. Set
HESPER_DUMP_MSL_QUIET=1 to capture without the stderr spew (sweep mode).
DEVPLAN M1: compile a Tint-dumped MSL source with the real Metal compiler and report the pipeline's resource stats: "maxThreads=N execWidth=M tgMem=K". maxThreads is the register-pressure inverse signal (drops on spill); this is the MANDATORY resource column of the autotune sweep. macOS only (IO error elsewhere).
Native serialized GPU timing of a dumped Tint-MSL kernel: nDispatches back-to-back in
one SERIAL Metal encoder, GPU ms from GPUStart/EndTime. Buffers in BINDING ORDER.
The honest per-kernel bench (Dawn adds ~35 µs/dispatch that drowns small matvecs).
DEVPLAN §13 cold-stream bench: like mslBenchSerial, but the buffer at rotSlot
rotates across copies per dispatch, so successive dispatches read DIFFERENT
memory — cold-stream BW instead of the warm-SLC number (an isolated bench
re-reading one weight 300× flattered the matvecs to 90–97 %; a real token streams
every weight once). Returns GPU ms for the whole run as a string.
M3 de-risk probe: run nDispatches of a dumped Tint-MSL kernel back-to-back in ONE
native Metal encoder with Serial or Concurrent dispatch type (no barriers — timing
only, racing writes tolerated). Returns GPU wall ms as a string.
DG_GPUBUSY: read+reset the MSL-path split (count / WaitForCommandsToBeScheduled / encode+commit / kernel GPU time). Complements gpuBusyRead (which covers the Dawn WGSL path only).
Tagged GPU-time isolation (metal backend): set the tag that subsequent command buffers' GPU busy time is attributed to. Flush at tag switches so a CB never straddles tags. Completion-handler based — adds NO waits, so it does not inflate the measured range.
Read a tag's accumulated GPU nanoseconds (cumulative since process start).
Vendored ggml (llama.cpp, MIT) MoE id-mapping: ids [nTok,nUsed] u32 → tpe [nExpert] counts + hids [nExpert,nTok] id lists. Metal backend only; commits immediately to the hm queue (flushBatch producers first).
Vendored ggml indirect matmul (kernel_mul_mm_id): kind 0=Q4_K 1=Q8_0 2=Q5_0; act bcast [nTok,K] (actPerSlot=0) or per-slot [nTok,nUsed,K] (=1); dst [nTok,nUsed,outRows] f32 scattered by the kernel.
Exp 2 Phase A (native replay): clear the recorded dispatch sequence.
Exp 2 Phase A: push a barrier marker (layer boundary) into the replay sequence.
Only honored by replayRun mode=2 (concurrent + barriers).
Exp 2 Phase A: record one dispatch for native replay. bufs must be in MSL
[[buffer(i)]] order (parse the Tint-CLI entry signature); entry = the MSL kernel
function name; tgBytes = threadgroup memory upper bound (0 if none; set via
setThreadgroupMemoryLength at index 0); writeMask bit i = bufs[i] is written
(read_write binding) — drives mode-3 hazard barriers. PSO compiled once per MSL.
Exp 2 Phase A: replay the recorded token in ONE native command buffer. mode: 0 = Serial (sanity vs Dawn GPU time), 1 = Concurrent no-barrier (upper bound), 2 = Concurrent + barriers at recorded markers (realistic). Returns "count=<ops> min=<ms> avg=<ms>". TIMING ONLY — buffer contents end up garbage. Device comes from the record-time stash, so this is callable from backend-generic code.
Exp 2 Phase B: execute the recorded token ONCE as the REAL computation. Commits to DAWN'S OWN MTLCommandQueue when reachable (single-queue FIFO ordering with Dawn's staging writes and readbacks), waits for completion. mode as in replayRun; 3 = concurrent + automatic hazard barriers. Returns "ms=<gpu ms> barriers=<n> queue=<dawn|own>".
DEVPLAN §12 per-op-class profiling: replay ONLY the given dispatch indices
(into the recorded sequence, barriers excluded), serially, iters times.
Returns "count=<n> min=<ms> avg=<ms>".
metal_replacer STEP 4: Apple's tuned MPS f16 matmul (C=A·Bᵀ, our reg-matmul shape) as the CEILING — returns "ms/call | GFLOPS | %peak". Diff vs the WGSL reg (harness) at the same shape to quantify the WGSL→Tint→Metal gap. macOS DEBUG/REFERENCE only. See METAL_REPLACER_INTEGRATION.md.
metal_replacer MSL PoC: bench the hand-written native-Metal port of
q4kMatmulGroupedRegIndexedKernel on the given Dawn buffers (same algorithm as the WGSL kernel).
Returns ms/iter from MTLCommandBuffer GPU timestamps. Caller syncs input writes first and
writes c once (Dawn lazy-clear). macOS DEBUG/REFERENCE only.
HOT-PATH MSL q4k gate/up dispatch (DG_MSL, ~1.61× vs WGSL/Tint). PSO cached after the first call; encode+commit with NO CPU wait. ORDERING CONTRACT: caller must flushBatch (commit the Dawn producer encoder) immediately BEFORE; Dawn buffers are hazard-tracked so Metal orders the command buffers by commit order on the shared buffers. macOS only.
HOT-PATH MSL Q8_0 MoE-down dispatch (DG_MSLDOWN): the native port of q8MatmulGroupedRegIndexedScatterKernel (A read direct from the grouped geglu output, ragged sub-tile skip, C scatter-on-store through pos/slot into dst[slot,NTOK,N]). Same ordering contract as mslQ4kDispatch (flushBatch before; hazard-tracked commit order). macOS only.
HOT-PATH MSL Q5_0 MoE-down dispatch: the Q5_0 (22B/block) analogue of mslQ8DownDispatch — covers the 16/30 layers whose down_exps are Q5_0 (previously the WGSL warp fallback + staged scatter). Same ordering contract. macOS only.
SINGLE-STREAM (DG_MSLONESTREAM): encode the MSL gate/up (q4k) AND the FUSED MSL down (q8/q5, reads sGatheredGU + inline geglu) into ONE MTLCommandBuffer with two compute encoders + ONE commit, so the gate/up→down MSL chain runs back-to-back with no inter-cb handoff bubble. gate/up writes guC=sGatheredGU which the down reads as its A; down reuses guIdx as pos. isQ5 selects the down kernel. Implies the fused down. macOS only.
Check if the device was created with the Chromium experimental
subgroup matrix feature. subgroup_matrix_left/right/result types
and subgroupMatrixLoad/Store/MultiplyAccumulate are available iff
this returns true.
Check if the device was created with ShaderF16 support. f16 values
and related arithmetic are available iff this returns true.
Tick the device (process callbacks and events). Should be called regularly when doing async operations.
Wait for GPU work to complete (takes Future from dispatchCompute)
List all available GPU adapters with their information
Equations
- One or more equations did not get rendered due to their size.