Alpaca Dataset Loader

Parses Stanford Alpaca-format JSON datasets for instruction finetuning.

Format

[
  {
    "instruction": "Give three tips for staying healthy.",
    "input": "",
    "output": "1. Eat a balanced diet..."
  },
  ...
]

Prompt Template

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Input:
{input}

### Response:
{output}

The model is trained with teacher forcing: the loss is computed only on the output tokens (after "### Response:\n").

structure Hesper.Training.AlpacaDataset.Example : Type

A single Alpaca training example

• instruction : String
• input : String
• output : String

def Hesper.Training.AlpacaDataset.instReprExample.repr : Example → Nat → Std.Format

Equations

• One or more equations did not get rendered due to their size.

instance Hesper.Training.AlpacaDataset.instReprExample : Repr Example

Equations

• Hesper.Training.AlpacaDataset.instReprExample = { reprPrec := Hesper.Training.AlpacaDataset.instReprExample.repr }

structure Hesper.Training.AlpacaDataset.TokenizedExample : Type

A tokenized training example ready for the model

• tokens : Array Nat

  Full token sequence (prompt + output + EOS)

• promptLen : Nat

  Index where the output starts (loss computed from here)

• seqLen : Nat

  Total sequence length

instance Hesper.Training.AlpacaDataset.instReprTokenizedExample : Repr TokenizedExample

Equations

• Hesper.Training.AlpacaDataset.instReprTokenizedExample = { reprPrec := Hesper.Training.AlpacaDataset.instReprTokenizedExample.repr }

def Hesper.Training.AlpacaDataset.instReprTokenizedExample.repr : TokenizedExample → Nat → Std.Format

Equations

• One or more equations did not get rendered due to their size.

def Hesper.Training.AlpacaDataset.formatPrompt (ex : Example) : String

Format an Alpaca example into the standard prompt template

Equations

• One or more equations did not get rendered due to their size.

def Hesper.Training.AlpacaDataset.formatFullSequence (ex : Example) : String

Format the full sequence (prompt + output) for training

Equations

• Hesper.Training.AlpacaDataset.formatFullSequence ex = Hesper.Training.AlpacaDataset.formatPrompt ex ++ ex.output

def Hesper.Training.AlpacaDataset.parseExample (json : Lean.Json) : Except String Example

Parse a single JSON object into an Alpaca Example

Equations

• One or more equations did not get rendered due to their size.

def Hesper.Training.AlpacaDataset.loadDataset (path : String) : IO (Array Example)

Load an Alpaca dataset from a JSON file. The file should contain a JSON array of objects.

Equations

• One or more equations did not get rendered due to their size.

def Hesper.Training.AlpacaDataset.tokenizeExample (encode : String → Array Nat) (ex : Example) (eosToken : Nat) (maxSeqLen : Nat := 512) : TokenizedExample

Tokenize an Alpaca example using the provided encode function.

@param encode Tokenizer encode function (String → Array Nat) @param example The Alpaca example @param eosToken End-of-sequence token ID @param maxSeqLen Maximum sequence length (truncate if longer) @return TokenizedExample with prompt boundary marked

Equations

• One or more equations did not get rendered due to their size.

def Hesper.Training.AlpacaDataset.tokenizeDataset (encode : String → Array Nat) (examples : Array Example) (eosToken : Nat) (maxSeqLen : Nat := 512) : Array TokenizedExample

Tokenize an entire dataset

Equations

• One or more equations did not get rendered due to their size.

def Hesper.Training.AlpacaDataset.printStats (examples : Array TokenizedExample) : IO Unit

Print dataset statistics

Equations

• One or more equations did not get rendered due to their size.