Findings
-
[1] 2026-09-22 Hardware-Agnostic Models in vLLM
TL;DR To achieve state-of-the-art performance at the frontier, vLLM is changing its internal implementation in ways that make it incompatible with fullgraph torch.compile. This may have consequences for users who care about out-of-tree accelerators, older GPUs, or more exotic models.… Figure 1: The current state of model definitions in vLLM. All three flavours resolve to a single implementation of each common layer, shown here for RowParallelLinear. SpyreRowParallelLinear is an out-of-tree plugin overriding that layer on one accelerator. While the modeling… Figure 2: Hardware-agnostic Layers in vLLM. As the legacy model definitions are gradually removed, models will either be re-implemented in the flat way (e.g., a different implementation for NVIDIA, AMD, XPU etc), or they will fallback to the transformers backend.…
-
[2] 2026-09-22 How Shopify built a continual learning loop with PyTorch and vLLM
TL;DR: This case study explores how Shopify compresses production failures into model weights every day, beats frontier-model quality, and cuts serving costs 96% by building a continual learning loop with PyTorch and vLLM. Frontier models are often the fastest way… So we treat it as an autoresearch problem, in the spirit of Karpathy’s recent project: an agent proposes a change to a prompt, a tool definition, or the harness; evaluates it against the judge; keeps the change if the score… Beyond the harness: continual learning that compounds Frontier models help you launch, and the first improvements live in the discrete artifacts around them: prompts, context, tool definitions, and control flow. Those changes strengthen the harness but leave the model unchanged.… Agrawal et al. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. arXiv:2507.19457 Zhang et al. Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models. arXiv:2510.04618 Hsieh et al. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller…
Where Kimbodo Comes In
Kimbodo builds and operates this in production for businesses — see our Posit & Shiny Development practice, or Estimate My Shiny Project.
