KiwiLM

KiwiLM was a small PyTorch research project for developing and comparing causal language model architectures trained on the TinyStories dataset.

The experiments compared (mostly) parameter-matched gated convolutional, hybrid and Transformer models, analyzing how the different placement of gated convolution and causal attention layers affected language quality, training & generation throughput.

KiwiLM serves as an experimental ground for more efficient hybrid architectures that can preserve useful context without relying exclusively on quadratic self-attention.