When I started Impossible Computing, I wanted to explore inference technology from all avenues. Everything from how to improve the agent harnesses, to better model evaluation, to improved serving techniques. That last one proved to be difficult to do quickly. Coming up with a new attention algorithm, for instance, and then testing it out, took quite a bit of time, even with the help of agents.
The question became: can we make it easier to experiment with GPU inference without starting from scratch in CUDA every time? And that led me to creating Jacq, a small DSL language and compiler that turns a text file describing a model or kernel into CUDA C++.
Today I’m sharing Jacq as an early open-source project. I want you to be able to read a small example, generate some code, and try an idea of your own. There’s a lot more to build, and I’d love to learn from people who are interested in the same problems.
What does a .jac file look like?
Here is a complete kernel example from the repository. It describes matrix multiplication: C = A × B.
kernel matmul {
tensors {
A[M, N] : fp16 @global -> stream
B[N, K] : fp16 @global -> stream
C[M, K] : fp32 @registers -> global
}
params {
M = 256
N = 128
K = 64
tile_m = 64
}
phase compute over A.dim(M) by tile_m {
load A_tile = A[tile, *]
C_tile = einsum("mn, nk -> mk", A_tile, B)
}
}The tensors block names the inputs and output, their shapes and data types. The annotations describe where the data is intended to live or move. The params block gives this example its dimensions and tile size. The compute phase describes the operation on each tile; einsum is a compact way to express the matrix multiplication.
That’s the experience I’m aiming for: start with something you can read, generate a CUDA implementation, inspect it, and try a change. Today Jacq recognizes a defined set of kernel patterns and emits the corresponding implementations.
There is a model language too
The .jac format can describe a kernel or a model. A model file includes a config block for dimensions, a weights block identifying the source and dtype, and layer and pipeline sections. Jacq parses these sections and can show a model overview. Here is an excerpt from the TinyLlama example:
model tinyllama {
config {
hidden = 2048
layers = 22
heads = 32
kv_heads = 4
head_dim = 64
// More model settings follow
}
weights {
format = huggingface
source = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
dtype = fp16
}
// The full file also has layer and pipeline sections
}From the repository root, Jacq can validate the complete TinyLlama example and generate a standalone CUDA inference runtime plus a Python weight exporter:
cargo build -p jac-cli
./target/debug/jacq --validate examples/tinyllama.jac
./target/debug/jacq --emit-model examples/tinyllama.jacRunning the generated runtime is a separate step: it needs the CUDA toolkit, a GPU, Python dependencies, and model weights. Note: in the current implementation, model code generation uses fixed runtime templates configured by supported settings. Layer and pipeline descriptions are parsed and can be inspected, but arbitrary edits to those expressions do not yet change the generated computation graph. That’ll come soon!
Why inference and the KV cache?
When a transformer generates text, it keeps a KV cache: information from tokens it has already processed that helps it attend to the history. That cache takes memory, and managing and reading it can become a significant part of inference. I’m interested in how a language and compiler can make it easier to explore different layouts and kernels for that work.
The standalone path is useful for quick, self-contained experiments. I tested generated runtimes for TinyLlama 1.1B, Qwen2.5 0.5B, and Mistral 7B v0.1 against Hugging Face.
I also want to take ideas into a real serving system, so there is an experimental path for emitting callable kernels and integrating a quantized KV cache with vLLM. The current approach patches a pinned vLLM 0.8.5 source checkout.
On my workstation, I ran Qwen2.5-Coder-14B-Instruct-AWQ on an RTX 4090 with a 512-token maximum model length. Stock fp16 and both KIVI cache-writer paths matched on seven completion requests and a chat request. A later check of the Jacq writer matched stock on various completions and a chat response; tracing counted 27,600 Jacq writer calls and 27,600 decode calls. Keep in mind this is all for R&D purposes, and I haven’t tried the generated code in a production environment.
Where Jacq stands
Jacq is early, and I’m releasing it now so others can explore it with me. Alongside the model and serving checks above, the compiler has CPU tests and synthetic kernel cases that pass on RTX 4090s. Next I want to broaden GPU testing, make results easier to reproduce, and keep improving the examples and docs.
If you’re curious about compilers, CUDA, or inference, I’d love for you to try it. Start with a .jac file, generate some code, and let me know what works and what gets in your way. Bug reports, experiments, and ideas for making Jacq easier to use are all welcome.