Neural Network Activation Functions

GPU kernels for common activation functions using the ShaderM monad. All operations are element-wise on tensors.

structure Hesper.NN.Activation.ActivationConfig : Type

Configuration for activation function kernels

• size : Nat

  Number of elements to process

• workgroupSize : Nat

  Workgroup size (threads per workgroup)

def Hesper.NN.Activation.instInhabitedActivationConfig.default : ActivationConfig

Equations

• Hesper.NN.Activation.instInhabitedActivationConfig.default = { size := default, workgroupSize := default }

instance Hesper.NN.Activation.instInhabitedActivationConfig : Inhabited ActivationConfig

Equations

• Hesper.NN.Activation.instInhabitedActivationConfig = { default := Hesper.NN.Activation.instInhabitedActivationConfig.default }

instance Hesper.NN.Activation.instReprActivationConfig : Repr ActivationConfig

Equations

• Hesper.NN.Activation.instReprActivationConfig = { reprPrec := Hesper.NN.Activation.instReprActivationConfig.repr }

def Hesper.NN.Activation.instReprActivationConfig.repr : ActivationConfig → Nat → Std.Format

Equations

• One or more equations did not get rendered due to their size.

def Hesper.NN.Activation.ActivationConfig.numWorkgroups (config : ActivationConfig) : Nat

Number of workgroups needed

Equations

• config.numWorkgroups = (config.size + config.workgroupSize - 1) / config.workgroupSize

def Hesper.NN.Activation.reluKernel (config : ActivationConfig) : WGSL.Monad.ShaderM Unit

ReLU activation kernel: f(x) = max(0, x)

Equations

• One or more equations did not get rendered due to their size.

def Hesper.NN.Activation.leakyReluKernel (config : ActivationConfig) (alpha : Float := 1e-2) : WGSL.Monad.ShaderM Unit

Leaky ReLU kernel: f(x) = max(alpha * x, x)

Equations

• One or more equations did not get rendered due to their size.

def Hesper.NN.Activation.geluKernel (config : ActivationConfig) : WGSL.Monad.ShaderM Unit

GELU activation kernel (approximate)

Equations

• One or more equations did not get rendered due to their size.

def Hesper.NN.Activation.sigmoidKernel (config : ActivationConfig) : WGSL.Monad.ShaderM Unit

Sigmoid activation kernel: f(x) = 1 / (1 + exp(-x))

Equations

• One or more equations did not get rendered due to their size.

def Hesper.NN.Activation.tanhKernel (config : ActivationConfig) : WGSL.Monad.ShaderM Unit

Tanh activation kernel: f(x) = tanh(x)

Equations

• One or more equations did not get rendered due to their size.

def Hesper.NN.Activation.swishKernel (config : ActivationConfig) : WGSL.Monad.ShaderM Unit

Swish/SiLU activation kernel: f(x) = x * sigmoid(x)

Equations

• One or more equations did not get rendered due to their size.

def Hesper.NN.Activation.eluKernel (config : ActivationConfig) (alpha : Float := 1.0) : WGSL.Monad.ShaderM Unit

ELU activation kernel: f(x) = x if x > 0 else alpha * (exp(x) - 1)

Equations

• One or more equations did not get rendered due to their size.

def Hesper.NN.Activation.softplusKernel (config : ActivationConfig) : WGSL.Monad.ShaderM Unit

Softplus activation kernel: f(x) = log(1 + exp(x))

Equations

• One or more equations did not get rendered due to their size.

def Hesper.NN.Activation.mishKernel (config : ActivationConfig) : WGSL.Monad.ShaderM Unit

Mish activation kernel: f(x) = x * tanh(softplus(x))

Equations

• One or more equations did not get rendered due to their size.

def Hesper.NN.Activation.softmaxKernel (config : ActivationConfig) : WGSL.Monad.ShaderM Unit

Softmax activation kernel (simplified version for demo) Note: This is a simplified implementation. Production softmax requires multi-pass reduction for finding max and sum across the entire array.

Equations

• One or more equations did not get rendered due to their size.

def Hesper.NN.Activation.generateReLUShader (config : ActivationConfig) : String

Generate WGSL shader for ReLU

Equations

• Hesper.NN.Activation.generateReLUShader config = Hesper.WGSL.CodeGen.generateWGSL "main" { x := config.workgroupSize } [] [] (Hesper.NN.Activation.reluKernel config)

def Hesper.NN.Activation.generateLeakyReLUShader (config : ActivationConfig) (alpha : Float := 1e-2) : String

Generate WGSL shader for Leaky ReLU

Equations

• Hesper.NN.Activation.generateLeakyReLUShader config alpha = Hesper.WGSL.CodeGen.generateWGSL "main" { x := config.workgroupSize } [] [] (Hesper.NN.Activation.leakyReluKernel config alpha)

def Hesper.NN.Activation.generateGELUShader (config : ActivationConfig) : String

Generate WGSL shader for GELU

Equations

• Hesper.NN.Activation.generateGELUShader config = Hesper.WGSL.CodeGen.generateWGSL "main" { x := config.workgroupSize } [] [] (Hesper.NN.Activation.geluKernel config)

def Hesper.NN.Activation.generateSigmoidShader (config : ActivationConfig) : String

Generate WGSL shader for Sigmoid

Equations

• Hesper.NN.Activation.generateSigmoidShader config = Hesper.WGSL.CodeGen.generateWGSL "main" { x := config.workgroupSize } [] [] (Hesper.NN.Activation.sigmoidKernel config)

def Hesper.NN.Activation.generateTanhShader (config : ActivationConfig) : String

Generate WGSL shader for Tanh

Equations

• Hesper.NN.Activation.generateTanhShader config = Hesper.WGSL.CodeGen.generateWGSL "main" { x := config.workgroupSize } [] [] (Hesper.NN.Activation.tanhKernel config)

def Hesper.NN.Activation.generateSwishShader (config : ActivationConfig) : String

Generate WGSL shader for Swish/SiLU

Equations

• Hesper.NN.Activation.generateSwishShader config = Hesper.WGSL.CodeGen.generateWGSL "main" { x := config.workgroupSize } [] [] (Hesper.NN.Activation.swishKernel config)

def Hesper.NN.Activation.generateELUShader (config : ActivationConfig) (alpha : Float := 1.0) : String

Generate WGSL shader for ELU

Equations

• Hesper.NN.Activation.generateELUShader config alpha = Hesper.WGSL.CodeGen.generateWGSL "main" { x := config.workgroupSize } [] [] (Hesper.NN.Activation.eluKernel config alpha)

def Hesper.NN.Activation.generateSoftplusShader (config : ActivationConfig) : String

Generate WGSL shader for Softplus

Equations

• Hesper.NN.Activation.generateSoftplusShader config = Hesper.WGSL.CodeGen.generateWGSL "main" { x := config.workgroupSize } [] [] (Hesper.NN.Activation.softplusKernel config)

def Hesper.NN.Activation.generateMishShader (config : ActivationConfig) : String

Generate WGSL shader for Mish

Equations

• Hesper.NN.Activation.generateMishShader config = Hesper.WGSL.CodeGen.generateWGSL "main" { x := config.workgroupSize } [] [] (Hesper.NN.Activation.mishKernel config)

def Hesper.NN.Activation.generateSoftmaxShader (config : ActivationConfig) : String

Generate WGSL shader for Softmax (simplified version)

Equations

• Hesper.NN.Activation.generateSoftmaxShader config = Hesper.WGSL.CodeGen.generateWGSL "main" { x := config.workgroupSize } [] [] (Hesper.NN.Activation.softmaxKernel config)