GGUF Quantization and Dequantization

Implements unpacking of quantized tensor formats, with focus on ternary quantization for BitNet.

Ternary Quantization Formats

TQ2_0 (2-bit packing)

• Block size: 256 elements
• Packing: 4 ternary values per byte (2 bits each)
• Scale: 1 FP16 scale per block
• Encoding: {-1, 0, 1} → {0b11, 0b00, 0b01}
  • 0b00 → 0
  • 0b01 → +1
  • 0b11 → -1 (two's complement)
• Layout: [packed_bytes (64)] [scale (FP16=2 bytes)]

TQ1_0 (base-3 encoding)

• Block size: 256 elements
• Packing: 5 ternary values per byte (3^5 = 243 < 256)
• More compact but more complex to unpack
• Layout: [packed_bytes (51)] [extra (4)] [scale (FP16=2 bytes)]

References

• llama.cpp/ggml/src/ggml-quants.c (lines 2103-2271) - TQ2_0 implementation
• llama.cpp/convert_hf_to_gguf.py (lines 3152-3190) - Quantization logic

Ternary Value Helpers

def Hesper.GGUF.Quantization.unpackTernary2bit (packed : UInt8) (idx : Nat) : Int8

Unpack a single 2-bit ternary value from packed byte @param packed The packed byte containing 4 values @param idx The index (0-3) of the value to extract @return Ternary value: -1, 0, or 1

Equations

• One or more equations did not get rendered due to their size.

def Hesper.GGUF.Quantization.packTernary2bit (v0 v1 v2 v3 : Int8) : UInt8

Pack 4 ternary values into a single byte (inverse operation) Used for testing and verification

Equations

• One or more equations did not get rendered due to their size.

TQ2_0 Block Structure

structure Hesper.GGUF.Quantization.TQ2_0_Block : Type

TQ2_0 block: 256 elements packed into 64 bytes + 1 FP16 scale Total: 66 bytes per block

• qs : ByteArray
• d : UInt16

def Hesper.GGUF.Quantization.TQ2_0_Block.blockSize : Nat

Block size in elements

Equations

• Hesper.GGUF.Quantization.TQ2_0_Block.blockSize = 256

def Hesper.GGUF.Quantization.TQ2_0_Block.blockSizeBytes : Nat

Block size in bytes (64 packed + 2 scale)

Equations

• Hesper.GGUF.Quantization.TQ2_0_Block.blockSizeBytes = 66

def Hesper.GGUF.Quantization.TQ2_0_Block.fp16ToFloat32Simple (bits : UInt16) : Float

Simplified FP16 to Float32 conversion Handles normalized values only (sufficient for most scales)

Equations

• One or more equations did not get rendered due to their size.

def Hesper.GGUF.Quantization.TQ2_0_Block.parse (data : ByteArray) (offset : Nat) : Except String TQ2_0_Block

Parse TQ2_0 block from ByteArray

Equations

• One or more equations did not get rendered due to their size.

def Hesper.GGUF.Quantization.TQ2_0_Block.unpack (block : TQ2_0_Block) : Array Float

Unpack TQ2_0 block to Float32 array

Equations

• One or more equations did not get rendered due to their size.

TQ1_0 Block Structure (TODO: Lower priority)

structure Hesper.GGUF.Quantization.TQ1_0_Block : Type

TQ1_0 block: 256 elements packed via base-3 encoding More compact but more complex unpacking

• qs : ByteArray
• qh : ByteArray
• d : UInt16

High-Level Dequantization API

def Hesper.GGUF.Quantization.dequantizeTQ2_0 (data : ByteArray) (numElements : Nat) : Except String (Array Float)

Dequantize TQ2_0 tensor to Float32 array

Equations

• One or more equations did not get rendered due to their size.

def Hesper.GGUF.Quantization.dequantizeTQ2_0.processBlocks (data : ByteArray) (numElements numBlocks fuel blockIdx : Nat) (acc : Array Float) : Except String (Array Float)

Equations

• One or more equations did not get rendered due to their size.
• Hesper.GGUF.Quantization.dequantizeTQ2_0.processBlocks data numElements numBlocks 0 blockIdx acc = Except.error "dequantizeTQ2_0: fuel exhausted"

def Hesper.GGUF.Quantization.dequantize (data : ByteArray) (ggmlType : GGMLType) (numElements : Nat) : Except String (Array Float)

Dequantize tensor based on GGMLType

Equations

• One or more equations did not get rendered due to their size.
• Hesper.GGUF.Quantization.dequantize data Hesper.GGUF.GGMLType.TQ2_0 numElements = Hesper.GGUF.Quantization.dequantizeTQ2_0 data numElements
• Hesper.GGUF.Quantization.dequantize data Hesper.GGUF.GGMLType.TQ1_0 numElements = Except.error "TQ1_0 dequantization not yet implemented"
• Hesper.GGUF.Quantization.dequantize data Hesper.GGUF.GGMLType.F16 numElements = Except.error "F16 dequantization not yet implemented"
• Hesper.GGUF.Quantization.dequantize data ggmlType numElements = Except.error (toString "Dequantization not implemented for " ++ toString ggmlType)

Quantization (for testing)

def Hesper.GGUF.Quantization.float32ToFP16Simple (f : Float) : UInt16

Helper: Simplified Float32 to FP16 conversion

Equations

• One or more equations did not get rendered due to their size.

def Hesper.GGUF.Quantization.quantizeTQ2_0 (values : Array Float) : ByteArray

Quantize Float32 array to TQ2_0 format Used for testing and validation (simplified version)

Equations

• One or more equations did not get rendered due to their size.

def Hesper.GGUF.Quantization.quantizeTQ2_0.processBlock (values : Array Float) (numBlocks fuel blockIdx : Nat) (acc : ByteArray) : ByteArray

Equations

• One or more equations did not get rendered due to their size.
• Hesper.GGUF.Quantization.quantizeTQ2_0.processBlock values numBlocks 0 blockIdx acc = acc