LoRA Weight Initialization

Creates and initializes LoRA adapter weights for BitNet finetuning.

Initialization Strategy

• A matrix: Kaiming uniform initialization (preserves signal magnitude)
• B matrix: Zero initialization (LoRA output starts at zero, preserving base model behavior)

This ensures that at the start of training, the LoRA-augmented model produces exactly the same output as the base model.

structure Hesper.LoRA.RNG : Type

Simple pseudo-random number generator (xoshiro128+) for weight initialization. Deterministic given a seed, which is important for reproducibility.

• s0 : UInt64
• s1 : UInt64

def Hesper.LoRA.RNG.create (seed : UInt64) : RNG

Equations

• One or more equations did not get rendered due to their size.

def Hesper.LoRA.RNG.next (rng : RNG) : UInt64 × RNG

Generate next random UInt64 and advance state

Equations

• One or more equations did not get rendered due to their size.

def Hesper.LoRA.RNG.nextFloat (rng : RNG) : Float × RNG

Generate a Float in [0, 1)

Equations

• rng.nextFloat = match rng.next with | (bits, rng') => have f := (bits >>> 11).toFloat / (1 <<< 53).toFloat; (f, rng')

def Hesper.LoRA.RNG.nextUniform (rng : RNG) (bound : Float) : Float × RNG

Generate a Float in [-bound, bound) using uniform distribution. Used for Kaiming uniform initialization.

Equations

• rng.nextUniform bound = match rng.nextFloat with | (f, rng') => (f * 2.0 * bound - bound, rng')

def Hesper.LoRA.kaimingUniformBound (fanIn : Nat) : Float

Generate Kaiming uniform initialization values. bound = sqrt(3 / fanIn) where fanIn = inDim for the A matrix. This preserves the variance of activations through the network.

Equations

• Hesper.LoRA.kaimingUniformBound fanIn = (3.0 / fanIn.toFloat).sqrt

def Hesper.LoRA.generateKaimingWeights (numElements fanIn : Nat) (seed : UInt64) : ByteArray

Create a ByteArray of FP32 values with Kaiming uniform initialization

Equations

• One or more equations did not get rendered due to their size.

def Hesper.LoRA.generateZeroWeights (numElements : Nat) : ByteArray

Create a ByteArray of zeros (numElements FP32 values)

Equations

• Hesper.LoRA.generateZeroWeights numElements = { data := Array.replicate (numElements * 4) 0 }

def Hesper.LoRA.createWeight (device : WebGPU.Device) (inDim outDim rank : Nat) (seed : UInt64) : IO Weight

Create a single LoRA weight pair for one projection. A is Kaiming initialized, B is zero initialized.

Equations

• One or more equations did not get rendered due to their size.

def Hesper.LoRA.createWeightGrad (device : WebGPU.Device) (weight : Weight) : IO WeightGrad

Create gradient buffers for a single LoRA weight pair (initialized to zero)

Equations

• One or more equations did not get rendered due to their size.

def Hesper.LoRA.createAdamState (device : WebGPU.Device) (weight : Weight) : IO AdamState

Create Adam optimizer state for a single LoRA weight pair (initialized to zero)

Equations

• One or more equations did not get rendered due to their size.

def Hesper.LoRA.createAdapter (device : WebGPU.Device) (config : Config) (numLayers dim kvDim : Nat) (seed : UInt64 := 42) : IO Adapter

Create a full LoRA adapter for a BitNet model. Applies LoRA to Q and V attention projections in all transformer layers.

@param device GPU device @param config LoRA configuration @param numLayers Number of transformer layers (e.g., 30 for BitNet-2B) @param dim Model hidden dimension (e.g., 2560 for BitNet-2B) @param kvDim KV dimension for V projection (e.g., 640 for BitNet-2B with GQA 4:1) @param seed Random seed for weight initialization

Equations

• One or more equations did not get rendered due to their size.

def Hesper.LoRA.createAdapterGrad (device : WebGPU.Device) (adapter : Adapter) : IO AdapterGrad

Create gradient buffers for the full adapter

Equations

• One or more equations did not get rendered due to their size.

def Hesper.LoRA.createAdapterAdamState (device : WebGPU.Device) (adapter : Adapter) : IO AdapterAdamState

Create Adam optimizer state for the full adapter

Equations

• One or more equations did not get rendered due to their size.

def Hesper.LoRA.createSavedActivations (device : WebGPU.Device) (adapter : Adapter) (dim kvDim : Nat) : IO SavedActivations

Create saved activation buffers for backward pass

Equations

• One or more equations did not get rendered due to their size.