Can the experiment run itself?
Making inference faster means trying changes that might not work. A kernel can be quick and still return the wrong answer. A faster decode step can also hide a slower first token.
neoKernel puts that work inside a research loop. A model proposes a change to a Qwen3 inference engine. The harness checks the source, runs the candidate and compares it with a reference. It then accepts the change or restores the previous engine.
The useful output is a record of what happened. That includes broken proposals, failed checks and experiments that looked promising until they were measured elsewhere.
Version 1 covers the research loop and selected local measurements from direct engine development and automated harness proposals.
One idea at a time.
The proposer sees the current source, a profile and recent experiment history. It returns a bounded change. The harness owns the checks and the decision.
For Version 2, I plan to run multiple agent swarms in parallel and let them share findings. Check the Future work section for more details.
Give the model a small job.
The model receives the current engine, profile, recent log and allowed playbook. It returns complete source files with a hypothesis and expected effect.
proposal item hypothesis expected_effect files risk reasoning
From proposals to working code
One of the first proposals failed before it could even run. The model had returned a malformed diff, so there was no way to test the change it intended. I switched to asking for complete files and left Git to generate the diff for the record. Proposals could still only touch the engine entry point and its Python kernels.
The early loop gave each proposal one attempt. Later, I added a chance to learn from a failed test: after the source check, the candidate runs unit tests on an L4, and the model gets the error output with up to two chances to fix its code. A candidate then has to pass public correctness checks before it reaches the H100 benchmark.
Picking up after a failed run
A failed experiment should leave something useful behind. Each trial has its own branch, saved copies of the source and a journal of its progress, so a rejected candidate can be inspected even after the previous engine has been restored. Accepted changes are committed and merged locally, with pushing left outside the loop. The harness also reserves the estimated budget before starting remote work.
A record of each attempt
Each trial saves its proposal, measurements and diagnostic output. Interrupted runs also leave a record, so an earlier result cannot be mistaken for the latest attempt.
Implementation loop.py, transaction.py and storage.py
Anti Reward Hacking
The candidate does not grade itself. Its score comes from checks and measurements owned by the harness.
The parent process supplies prompts and times the arriving tokens. It stops the candidate before an independent native model replays the generated sequence. Every output token is checked against the model's logits on the prefix the candidate actually produced.
A token passes if it is the native argmax or is within 2.0 logits of it. The small margin allows for differences in BF16 arithmetic. One position outside that margin fails the workload.
Every run draws its prompts from one recorded seed, and the candidate and the reference see the same prompts, so a result can be replayed rather than retold. Each replay records the worst margin it saw even when it passed, which makes an engine sitting close to the limit visible before it fails.
The reference is also checked against itself, on a separate budget of 0.75 logits. That is a measurement floor, not a verdict on any candidate: it is calibrated from the reference's own worst self-replay margin of 0.5 across two seeded five-sample runs of all six workloads. When the reference misses it, the run has left the regime the candidate margin was calibrated in, so that workload is reported as a harness error and retried instead of being charged to the candidate.
Move the slider to test the rule. This is a teaching example, not a measured token.
The local runner isolates the candidate process and controls its inputs. It is not a hardened sandbox. Parent and child share an operating system identity, and peak allocator memory is reported by the child.
The results from the runs
These are selected local six-workload measurements. Direct development established the engine foundation, and the automated harness subsequently explored proposals on top of it. The highest locally measured candidate reached 796.0 tok/s, an 8.92% improvement over the engine used as its immediate baseline.
Individual trials were not monotonic.
| Stage | How it was produced | Tokens / s |
|---|---|---|
| Initial baseline | Starting engine | 216.3 |
| CUDA graph milestone | Direct engine development | 323.7 |
| Attention-layout candidate | Automated harness proposal built on the developed engine | 796.0 |
Measure the whole generation.
Throughput includes the prompt and every output token. The harness also records time to first token, time per output token, memory and sample spread. A good average cannot excuse a failed correctness check.
The external evaluator is Dryft. Its official score uses three hidden workloads. The local harness uses three public shapes and three proxy shapes. Those proxies are guesses, not the hidden test set. The official latency limit is 1.10× native, while the local limit is 1.05×.
| Workload | Batch | Prompt | Output |
|---|---|---|---|
| public-0 | 1 | 512 | 32 |
| public-1 | 4 | 2048 | 32 |
| public-2 | 16 | 512 | 128 |
Rules from the repository's engine contract and local judge.
Start with the source checks.
The project uses Python 3.11. Run these from the repository root. They do not start a GPU job.
py -3.11 -m neokernel guard py -3.11 -m neokernel report
The automated loop needs the dependencies in neokernel/requirements.txt, a configured Modal account, the pinned checkpoint in its volume and BASETEN_API_KEY in the environment. It starts from a clean main branch.
Version 1 uses GLM through Baseten for proposals and Modal for GPU jobs. In Version 2, I am working toward an API provider agnostic and GPU provider agnostic harness. Check the Future work section for more details.
py -3.11 -m neokernel auto --steps 1 --workloads all --max-gpu-minutes 60
This command uses paid GPU compute and model inference. The 60 minute value bounds the loop's GPU budget. Persistent reservations can stop a run earlier.
Use --resume after an interrupted loop. Recovery preserves the candidate snapshot before restoring rejected work. The optional sweep command searches numeric settings without a model. Its existence is not evidence of a measured win.
A reusable research system.
I want neoKernel to become a Python package for running inference experiments with a model and compute provider of your choice. Version 2 will start with parallel search and provider independence. The plans below are future work, not features available in Version 1.
Multiple agent swarms
I plan to run several swarms in parallel. Each swarm would explore a different hypothesis, with agents handling proposals, implementation and profiling. They would share measured findings and failed attempts so one group can build on another's work.
Each candidate would have its own workspace and budget. A shared scheduler would reserve GPUs for timing runs so competing jobs do not distort the measurements. The independent judge would still decide which candidates pass. Combining changes would require a fresh evaluation.
API provider independence
The proposal loop should work through a small model adapter rather than depend on one inference API. The adapter would handle credentials, structured responses, retries and usage reporting. The experiment format and evaluation rules would stay the same when the provider changes.
GPU provider independence
The compute layer should support different cloud providers and local GPUs through a common job interface. A job would carry its engine source, environment requirements and workload definition. Each backend would return the same result format.
Changing providers should not mean comparing unlike machines as if they were identical. Results would retain GPU type, runtime versions and the paired native baseline. Supporting another GPU architecture would still require compatible kernels and fresh correctness checks.
A Python package
The longer term goal is an installable package with a Python API and a small command line interface. Users would supply an engine, reference implementation, workloads and a budget. Provider adapters would handle execution. The package would keep the experiment history and export a reproducible report.
I want saved experiments to include source hashes, environment details and replayable checks. That would make it easier to reproduce a result, compare search strategies and build on another person's work.
Research directions
Parallel search and shared memory. KernelAgent explores optimization paths in parallel and carries findings between rounds. I want to study whether swarms find better candidates within a fixed budget than the single proposal loop. KernelAgent.
Compiler autotuning. Ansor combines program search with a learned cost model. A useful direction for neoKernel is to pair structural proposals with numeric tuning, then prioritize candidates using measured hardware feedback. Ansor.
Evaluation beyond one model. KernelBench evaluates generated kernels for correctness and speed. I want a broader set of operators, model shapes and held-out workloads, while retaining whole-generation replay. This would help test whether improvements generalize beyond the cases used during search. KernelBench.
Stronger Anti Reward Hacking. I want tighter process isolation, protected evaluation inputs and checks that a candidate cannot modify. Failed attempts should remain visible. Search quality should be measured by reproducible, valid improvements under a fixed budget.
Code, evidence and research.
Download the run records or explore the source behind the experiments.
Technical references
These sources inform the research directions above. Their results are not neoKernel results.