GGUF Tensor Loader #

Loads tensor data from parsed GGUF files and uploads to GPU buffers.

Architecture #

GGUF File (on disk)
  │
  ├─► Parser → GGUFFile structure (in memory)
  │            ├─ Header
  │            ├─ Metadata
  │            ├─ Tensor Info (names, shapes, offsets)
  │            └─ Data Blob (raw bytes)
  │
  ├─► Loader → Extract specific tensors by name
  │            ├─ Lookup tensor info
  │            ├─ Extract ByteArray slice
  │            └─ Return (data, shape, quantization type)
  │
  └─► Upload → GPU Buffer
               ├─ If quantized (TQ2_0): Upload packed + scales
               └─ If Float32: Upload directly

Tensor Naming Conventions #

GGUF files use specific naming patterns (from llama.cpp):

Embedding #

token_embd.weight → [vocab_size, dim]

Transformer Blocks (per layer N) #

blk.N.attn_norm.weight → [dim]           (RMSNorm scales)
blk.N.attn_q.weight → [dim, dim]         (Q projection)
blk.N.attn_k.weight → [dim, kv_dim]      (K projection)
blk.N.attn_v.weight → [dim, kv_dim]      (V projection)
blk.N.attn_output.weight → [dim, dim]    (Output projection)

blk.N.ffn_norm.weight → [dim]            (RMSNorm scales)
blk.N.ffn_gate.weight → [dim, ffn_dim]   (Gate projection)
blk.N.ffn_up.weight → [dim, ffn_dim]     (Up projection)
blk.N.ffn_down.weight → [ffn_dim, dim]   (Down projection)

Output #

output_norm.weight → [dim]               (Final RMSNorm)
output.weight → [dim, vocab_size]        (LM head)

Quantization Handling #

Different tensors use different quantization:

TQ2_0 (ternary): Most weight matrices (BitLinear)
Float32: RMSNorm scales, some small tensors
FP16: Sometimes used for scales

We handle this by:

Check tensor's ggml_type field
Extract appropriate data format
Upload to GPU with correct interpretation

Performance #

Zero-copy extraction: Tensor data is a ByteArray.extract slice - no memory copying.

References #

llama.cpp: convert_hf_to_gguf.py (tensor naming)
llama.cpp: ggml-quants.h (quantization types)
GGUF spec: https://github.com/ggerganov/ggml/blob/master/docs/gguf.md

Tensor Data Types #

source

inductive Hesper.GGUF.Loader.GGMLType :

Type

GGML quantization type IDs (from ggml-quants.h)

F32 : GGMLType
F16 : GGMLType
BF16 : GGMLType
Q4_0 : GGMLType
Q4_1 : GGMLType
Q5_0 : GGMLType
Q5_1 : GGMLType
Q8_0 : GGMLType
Q8_1 : GGMLType
Q2_K : GGMLType
Q3_K : GGMLType
Q4_K : GGMLType
Q5_K : GGMLType
Q6_K : GGMLType
Q8_K : GGMLType
IQ2_XXS : GGMLType
TQ2_0 : GGMLType
IQ4_NL_4_4 : GGMLType
IQ4_NL_4_8 : GGMLType
IQ4_NL_8_8 : GGMLType
MXFP4 : GGMLType
Unknown (id : Nat) : GGMLType

Instances For

source

instance Hesper.GGUF.Loader.instReprGGMLType :

Repr GGMLType

Equations

Hesper.GGUF.Loader.instReprGGMLType = { reprPrec := Hesper.GGUF.Loader.instReprGGMLType.repr }

source

def Hesper.GGUF.Loader.instReprGGMLType.repr :

GGMLType → Nat → Std.Format

Equations

One or more equations did not get rendered due to their size.

Instances For

source

def Hesper.GGUF.Loader.GGMLType.fromNat (n : Nat) :

GGMLType

Equations

Instances For

source

def Hesper.GGUF.Loader.GGMLType.bytesPerElement (t : GGMLType) :

Nat

Equations

Instances For

source

instance Hesper.GGUF.Loader.instToStringGGMLType :

ToString GGMLType

Equations

One or more equations did not get rendered due to their size.

Tensor Info #

source

structure Hesper.GGUF.Loader.TensorInfo :

Type

Tensor metadata extracted from GGUF

name : String
shape : Array Nat
ggmlType : GGMLType
offset : Nat
size : Nat

Instances For

source

def Hesper.GGUF.Loader.instReprTensorInfo.repr :

TensorInfo → Nat → Std.Format

Equations

One or more equations did not get rendered due to their size.

Instances For

source

instance Hesper.GGUF.Loader.instReprTensorInfo :

Repr TensorInfo

Equations

Hesper.GGUF.Loader.instReprTensorInfo = { reprPrec := Hesper.GGUF.Loader.instReprTensorInfo.repr }

Tensor Extraction #

source

def Hesper.GGUF.Loader.convertGGMLType (t : GGUF.GGMLType) :

GGMLType

Find tensor by name in GGUF file

@param gguf Parsed GGUF file @param name Tensor name (e.g., "blk.0.attn_q.weight") @return Tensor info if found

Equations

Instances For

source

def Hesper.GGUF.Loader.findTensor (gguf : GGUFFile) (name : String) :

Except String TensorInfo

Equations

One or more equations did not get rendered due to their size.

Instances For

source

def Hesper.GGUF.Loader.getTensorData (gguf : GGUFFile) (name : String) :

Except String (TensorInfo × ByteArray)

Extract tensor data by name

Returns raw bytes for the tensor. Caller must interpret based on ggmlType.

@param gguf Parsed GGUF file @param name Tensor name @return (TensorInfo, ByteArray) - metadata and data

Equations

One or more equations did not get rendered due to their size.

Instances For

source

def Hesper.GGUF.Loader.getTensorDataM (gguf : GGUFFile) (name : String) :

IO (TensorInfo × ByteArray)

mmap-aware tensor data getter. When the file was loaded via loadGGUFMmap and dataBlob is empty (zero-copy mode), copies the bytes from the mmap region instead. Otherwise falls back to the pure-byteArray path.

Equations

One or more equations did not get rendered due to their size.

Instances For

source

def Hesper.GGUF.Loader.mkEmptyByteArray :

ByteArray

Extract Float32 tensor data

Unpacks F32 data into Float array.

@param gguf Parsed GGUF file @param name Tensor name @return Array of Float32 values

Equations

Hesper.GGUF.Loader.mkEmptyByteArray = ByteArray.empty

Instances For

source

def Hesper.GGUF.Loader.getFloat32Tensor (gguf : GGUFFile) (name : String) :

Except String (Array Float)

Equations

One or more equations did not get rendered due to their size.

Instances For

source

def Hesper.GGUF.Loader.getF16Tensor (gguf : GGUFFile) (name : String) :

Except String ByteArray

Extract F16 (Float16) tensor data as ByteArray

Returns raw F16 data as ByteArray (ready for GPU upload).

@param gguf Parsed GGUF file @param name Tensor name @return ByteArray of F16 data

Equations

One or more equations did not get rendered due to their size.

Instances For

source

def Hesper.GGUF.Loader.getI2_S_Tensor (gguf : GGUFFile) (name : String) :

Except String (ByteArray × Float × Nat)

Extract I2_S (BitNet ternary) tensor data

Returns packed 2-bit ternary data and scale factor. I2_S format: 2 bits per weight {-1, 0, +1} Encoding: 00 → -1, 01 → 0, 10 → +1

@param gguf Parsed GGUF file @param name Tensor name @return (packed_data: ByteArray, scale: Float, num_elements: Nat)

Equations

One or more equations did not get rendered due to their size.

Instances For

source

def Hesper.GGUF.Loader.getTQ2_0Tensor (gguf : GGUFFile) (name : String) :

Except String (ByteArray × ByteArray × Nat)

Extract TQ2_0 quantized tensor data

Returns packed ternary data and FP16 scales for TQ2_0 tensors.

@param gguf Parsed GGUF file @param name Tensor name @return (packed_data, scales_data, num_blocks)

Equations

One or more equations did not get rendered due to their size.

Instances For

GPU Upload #

source

def Hesper.GGUF.Loader.extractFloat32Tensor (gguf : GGUFFile) (name : String) :

IO ByteArray

Extract Float32 tensor as ByteArray

@param gguf Parsed GGUF file @param name Tensor name @return ByteArray of Float32 data (ready for GPU upload)

Equations

One or more equations did not get rendered due to their size.

Instances For

source

def Hesper.GGUF.Loader.extractF16Tensor (gguf : GGUFFile) (name : String) :

IO ByteArray

Extract F16 tensor as ByteArray

@param gguf Parsed GGUF file @param name Tensor name @return ByteArray of F16 data (ready for GPU upload)

Equations

One or more equations did not get rendered due to their size.

Instances For

source

def Hesper.GGUF.Loader.extractI2_S_Tensor (gguf : GGUFFile) (name : String) :

IO ByteArray

Extract I2_S tensor and dequantize to Float32

Dequantizes BitNet ternary weights to F32 for GPU computation.

@param gguf Parsed GGUF file @param name Tensor name @return ByteArray of F32 dequantized data

Equations

One or more equations did not get rendered due to their size.

Instances For

source

def Hesper.GGUF.Loader.extractTQ2_0Tensor (gguf : GGUFFile) (name : String) :

IO (ByteArray × ByteArray)

Extract TQ2_0 quantized tensor as ByteArrays

Returns two ByteArrays: packed data and scales.

@param gguf Parsed GGUF file @param name Tensor name @return (packed_data, scales_data)

Equations

One or more equations did not get rendered due to their size.

Instances For

Utilities #

source

def Hesper.GGUF.Loader.listTensors (gguf : GGUFFile) :

IO Unit

List all tensor names in GGUF file

Useful for debugging and validation.

@param gguf Parsed GGUF file

Equations

One or more equations did not get rendered due to their size.

Instances For

source

def Hesper.GGUF.Loader.validateTensor (gguf : GGUFFile) (name : String) (expectedShape : Array Nat) :

Except String Unit

Validate tensor exists and has expected shape

@param gguf Parsed GGUF file @param name Tensor name @param expectedShape Expected dimensions @return true if tensor exists with correct shape

Equations

One or more equations did not get rendered due to their size.

Instances For

source

def Hesper.GGUF.Loader.getQ4KTensor (gguf : GGUFFile) (name : String) :

Except String (ByteArray × Nat)

Extract Q4_K tensor data as raw ByteArray

Returns raw Q4_K block data ready for GPU upload. Q4_K blocks: 144 bytes per 256 elements.

@param gguf Parsed GGUF file @param name Tensor name @return (raw_data: ByteArray, num_elements: Nat)

Equations

One or more equations did not get rendered due to their size.

Instances For

source

def Hesper.GGUF.Loader.extractQ4KTensor (gguf : GGUFFile) (name : String) :

IO (ByteArray × Nat)

Extract Q4_K tensor and upload raw block data to GPU

@param gguf Parsed GGUF file @param name Tensor name @return (ByteArray of raw Q4_K block data, num_elements)

Equations

One or more equations did not get rendered due to their size.

Instances For