Memra Guide: A High-Performance LLM Inference Engine in Rust

Aug 7, 2026

Introduction

In the rapidly evolving landscape of Large Language Models (LLMs), performance is paramount. As models grow larger and applications demand lower latency, the underlying inference engine becomes a critical component of the stack. While established frameworks dominate the space, a new project called Memra is making waves by taking a from-scratch approach. Memra is a high-performance LLM inference engine written entirely in Rust and CUDA, specifically engineered to extract maximum performance from modern NVIDIA GPUs by building close to the metal.

What Is Memra?

Memra is an open-source, from-scratch LLM inference engine designed for NVIDIA GPUs, with a primary focus on the Blackwell (sm_120a) and Hopper (sm_90a) architectures. Written in Rust for safety and high-level logic, it pairs with hand-written CUDA kernels for performance-critical operations, completely avoiding dependencies like ggml. The project, maintained by Avi Fenesh and licensed under the MIT license, distinguishes itself with a strict “exactness discipline.” This principle guarantees that optimized techniques like speculative decoding produce results that are token-for-token identical to plain, non-speculative decoding, ensuring that performance gains never compromise model output integrity.

Why Memra Matters

The existence of mature, well-funded inference frameworks like llama.cpp and vLLM begs the question: why build a new one from scratch? Memra’s significance lies in its uncompromising focus on performance through specialization. By targeting specific, modern GPU architectures, it can leverage hardware features that more general-purpose frameworks cannot, or do not, prioritize. This results in significant performance gains, with the project’s own benchmarks showing it running certain models faster than established players on the same hardware.

Furthermore, Memra’s commitment to bit-exact outputs is a crucial differentiator. In many scientific, financial, or mission-critical applications, the stochastic nature of some optimizations is unacceptable. Memra provides a guarantee of deterministic output, which is a powerful feature for developers who require verifiability and reproducibility. This makes Memra a critical project for developers and researchers pushing the boundaries of what’s possible with LLMs on consumer and datacenter NVIDIA hardware.

Key Features

  • From-Scratch Rust + CUDA Engine: Built without reliance on external frameworks like ggml, giving the developers full control over the entire execution stack for maximum optimization.
  • Dual-Architecture GPU Support: Specifically tuned for NVIDIA’s Blackwell (e.g., RTX 50-series) and Hopper (e.g., H100) microarchitectures, with automatic detection at build time to deploy the most efficient kernels.
  • Strict Exactness-Gated Inference: A core design principle ensuring that outputs from speculative decoding and other optimizations are guaranteed to be token-identical to simple greedy decoding. This eliminates any trade-off between speed and correctness.
  • Advanced Speculative Decoding: Implements MTP (Multi-token Prediction) speculative decoding with adaptive draft depth and confidence cuts to accelerate inference without sacrificing accuracy.
  • CUDA Graph Optimization: Leverages CUDA graphs to minimize CPU-to-GPU traffic during the decoding phase, reducing overhead and improving throughput.
  • OpenAI-Compatible Server: Ships with a built-in server that mimics the OpenAI API, allowing for easy integration with existing applications and tools. It supports batched decoding and cross-request prefill batching for efficient concurrent serving.
  • Optimized Memory Management: Includes advanced techniques for Mixture-of-Experts (MoE) models, such as resident-if-fits expert residency, to run large models on consumer hardware with 24GB of VRAM.

How Memra Compares

Memra enters a competitive field dominated by highly respected open-source projects. Its primary differentiators are its specialized hardware focus and its strict exactness guarantees. Here’s how it stacks up against two of the most popular alternatives: llama.cpp and vLLM.

Aspect Memra llama.cpp vLLM
Primary Goal Max performance on specific NVIDIA GPUs with bit-exactness CPU-first inference, broad hardware support (CPU, Metal, CUDA, ROCm) High-throughput serving on NVIDIA & AMD GPUs
Core Technology Rust + custom CUDA kernels C++ (gguf format) Python + PagedAttention
Exactness Guarantee Yes (core design principle) Generally, but can vary with quantization/optimizations Generally, but focus is on throughput
Ease of Use Requires source build, targeted at experts Very high, simple to build and run High, pip installable Python library
Hardware Focus NVIDIA Hopper & Blackwell Broad, CPU-centric with GPU offload NVIDIA & AMD GPUs

Memra vs. Llama.cpp: The most significant difference is the target hardware. Llama.cpp is renowned for its exceptional CPU performance and its portability across a vast range of hardware. Memra is laser-focused on extracting every ounce of performance from high-end NVIDIA GPUs, choosing not to support other hardware. Developers with the latest NVIDIA cards may see higher performance with Memra, while those on CPUs, Macs, or older GPUs will find llama.cpp to be the superior choice.

Memra vs. VLLM: Both Memra and vLLM target high-throughput serving on NVIDIA GPUs. However, vLLM’s key innovation is PagedAttention, which is a memory management algorithm to reduce memory waste. Memra’s approach is different, focusing on hand-tuned kernels and a strict, bit-exact execution that guarantees reproducibility. VLLM is a mature Python-based ecosystem ideal for general high-throughput serving, while Memra is a lower-level, Rust-based system for those who need absolute maximum speed and guaranteed deterministic outputs on specific hardware.

Getting Started: Installation

Memra is designed to be built from source to ensure it is perfectly compiled for your specific GPU architecture. Pre-built binaries for Linux x86_64 are also made available on the project’s GitHub releases page for convenience.

Prerequisites

You must have the Rust toolchain (cargo) and a compatible NVIDIA CUDA toolkit installed on your system. The build script will automatically detect your GPU’s compute capability.

Building from Source

The installation process involves cloning the repository and using Cargo to build the release binary.

# Clone the GitHub repository
git clone https://github.com/avifenesh/memra.git

# Navigate into the project directory
cd memra

# Build the project in release mode
cargo build --release

The build script automatically detects the GPU architecture. You can override this by setting the `MEMRA_CUDA_ARCH` environment variable. After a successful build, the binaries will be located in the `target/release/` directory.

How to Use Memra

Once built, Memra provides a few key executables. The most important first step is to verify that all custom CUDA kernels are working correctly on your hardware.

Kernel Verification

Run the `kernel-check` binary to test every CUDA kernel against its CPU-based reference implementation. This is a critical step to ensure the integrity of the engine on your specific machine.

./target/release/kernel-check

A successful run should end with the message `ALL GREEN`. If it does not, you should not proceed, as it indicates a mismatch or issue with the compiled kernels.

Running the Server

The primary way to interact with Memra is by running its OpenAI-compatible server. You point it to a model checkpoint (in FP8 format, for which conversion scripts are provided) and it will start a web server on a local port.

./target/release/memra-server --model /path/to/your/model.fp8

Once the server is running, you can interact with it using standard tools like `curl` or any OpenAI-compatible client library, pointing them to the local server address.

Real-World Use Cases

  • AI Research and Development: For researchers experimenting with model architecture or inference techniques, Memra provides a high-performance, verifiable baseline to measure against. The bit-exact guarantee is crucial for reproducible research.
  • Low-Latency API Endpoints: Companies that need to serve LLMs with the lowest possible latency can use Memra to power their API backends, leveraging its specialized optimizations to provide faster responses to users.
  • Agentic Systems: AI agent systems often require many rapid LLM calls. Memra’s speed can significantly reduce the “thinking time” of an agent, leading to a more responsive and interactive experience.
  • Offline and Embedded AI: For applications running on edge devices or workstations with high-end NVIDIA GPUs, Memra provides a powerful inference engine that does not require a cloud connection.
  • Financial and Scientific Modeling: In fields where determinism is a non-negotiable requirement, Memra’s exactness discipline ensures that the same inputs will always produce the same outputs, which is critical for validation and auditing.

Contributing to Memra

Memra is an open-source project that welcomes community contributions. The development process is centered around the GitHub repository. At present, there is no formal `CONTRIBUTING.md` file, so the best way to get involved is by engaging with the project through standard GitHub workflows.

To contribute, you can start by checking the Issues tab for existing bug reports or feature requests. For new ideas, it is best practice to open a new issue to discuss your proposed changes with the maintainer before beginning work. Pull requests should be clear, concise, and linked to a relevant issue if possible.

Community and Support

The central hub for the Memra community is its GitHub repository. All technical discussions, bug reports, and support requests are handled through GitHub Issues.

  • GitHub Issues: The primary channel for reporting bugs, asking questions, and requesting new features.
  • GitHub Discussions: If enabled by the maintainer, this would be a forum for more general conversation, sharing use cases, and community interaction.

Given the highly technical nature of the project, users are expected to have a strong background in Rust, CUDA, and the mechanics of LLM inference. Support is provided by the project’s author and its community of early adopters.

Conclusion

Memra is not just another LLM inference engine; it is a statement about the power of specialization. By intentionally limiting its scope to specific NVIDIA architectures and prioritizing bit-exactness above all else, it achieves a level of performance and reliability that is difficult to attain with a general-purpose tool. It carves out a vital niche for expert developers and researchers who demand the absolute highest speed without compromising on the integrity of the model’s output.

While it may not be the right tool for beginners or those on non-NVIDIA hardware, Memra is an essential project for anyone working on the cutting edge of AI inference. If your work involves low-latency applications, reproducible research, or simply getting the most out of your high-end GPU, you owe it to yourself to clone the Memra repository, run the benchmarks, and experience its performance firsthand.

Resources

What is Memra?

Memra is a high-performance LLM (Large Language Model) inference engine built from scratch using Rust and CUDA. It is specifically optimized for modern NVIDIA GPU architectures, such as Hopper and Blackwell, to deliver extremely fast and efficient model execution while guaranteeing that the results are bit-for-bit identical to standard, non-optimized decoding.

How does Memra compare to llama.cpp?

The main difference is their hardware focus. Memra is specialized for maximum performance on high-end NVIDIA GPUs and does not support other hardware. In contrast, llama.cpp is designed for broad compatibility, offering excellent performance on CPUs and support for various GPU backends like Apple Metal, making it more versatile for consumer hardware.

What does 'exactness-gated' or 'bit-exact' mean in Memra?

This is a core principle of Memra, meaning that all performance optimizations, such as speculative decoding, are guaranteed to produce the exact same sequence of tokens as the simpler, slower plain decoding method. This ensures there is no trade-off between speed and the correctness or reproducibility of the model’s output, which is critical for scientific and financial applications.

How do I install Memra?

Memra is typically installed by building it from the source code available on its GitHub repository. You will need the Rust toolchain and the NVIDIA CUDA toolkit installed. The installation involves cloning the repository and running `cargo build –release`, which will compile binaries optimized for your specific GPU.

Can I use Memra with AMD or Intel GPUs?

No. Memra is explicitly designed and optimized for specific NVIDIA GPU architectures (Hopper and Blackwell). Its hand-written CUDA kernels are not compatible with other GPU ecosystems like ROCm (for AMD) or oneAPI (for Intel). For non-NVIDIA GPUs, alternatives like llama.cpp are a better choice.

Is Memra suitable for production environments?

Memra is a newer, highly specialized tool. While it is built for high performance, its suitability for production depends on the use case. Its guarantee of exactness makes it very reliable, but its limited hardware support and smaller community mean it is best suited for expert teams who can manage a from-source software build and require its specific performance characteristics.

What programming language is Memra written in?

Memra uses a hybrid approach. The main application logic, server, and high-level components are written in Rust, which provides memory safety and modern tooling. The performance-critical computations, known as kernels, are hand-written in NVIDIA’s CUDA C++ to achieve the best possible performance on the GPU.

Does Memra have an API I can use?

Yes, Memra includes a built-in server that exposes an OpenAI-compatible API. This makes it easy to integrate into existing applications, scripts, and workflows that are already designed to communicate with OpenAI’s models. You can simply point your existing client to the local Memra server endpoint.