Introduction

Evaluating the performance of a Large Language Model (LLM) often feels like a guessing game, as traditional benchmarks fail to capture the nuance of human-like conversation. FastChat is an open-source platform designed to solve this by providing a standardized framework for training, serving, and evaluating LLM-based chatbots. By introducing the “LLM-as-a-judge” approach and the MT-bench benchmark, FastChat allows developers to quantify chat quality with a level of precision and scalability that was previously impossible without expensive human review.

What Is FastChat?

FastChat is an open platform for training, serving, and evaluating large language model based chatbots, maintained by the Large Model Systems Organization (LMSYS). It is written primarily in Python and licensed under the Apache License 2.0. The project serves as the foundation for the well-known Vicuna model and the crowdsourced Chatbot Arena, providing the infrastructure needed to move a model from raw weights to a deployed, evaluated chat assistant.

At its core, FastChat provides a distributed multi-model serving system that is fully compatible with the OpenAI API, allowing it to act as a local drop-in replacement for commercial LLM services. This enables researchers and developers to swap models seamlessly while maintaining the same integration layer.

Why FastChat Matters

Before FastChat, evaluating chat assistants was a binary choice between slow, expensive human evaluation and rigid, outdated benchmarks like BLEU or ROUGE, which measure string overlap rather than conversational quality. FastChat fills this gap by implementing the LLM-as-a-judge paradigm, where a strong model (like GPT-4) is used to score the responses of other models based on a set of complex, multi-turn questions.

The project has gained massive traction because it democratizes the ability to run high-quality open-source models locally. By providing the weights for Vicuna and the tools to serve them via a WebUI or REST API, FastChat allows organizations to maintain data privacy and reduce dependency on proprietary APIs while still achieving performance that rivals commercial alternatives.

For the ML engineer, FastChat is essential because it provides a complete lifecycle toolset. Instead of stitching together separate libraries for training, serving, and evaluation, FastChat offers a unified pipeline that ensures consistency across the development process.

Key Features

• LLM-as-a-Judge Evaluation: Uses strong LLMs to automatically score and explain the quality of model responses, approximating human preference with over 80% agreement.
• MT-Bench Integration: A challenging set of multi-turn, open-ended questions designed to test a model’s ability to maintain context and follow complex instructions.
• OpenAI-Compatible API Server: A RESTful API that allows any application built for OpenAI to work with local models without changing a single line of code.
• Distributed Serving System: A scalable architecture consisting of a controller, model workers, and a Gradio-based WebUI for interacting with multiple models simultaneously.
• Vicuna Model Weights: Provides the training code and weights for Vicuna, an open-source chatbot that achieves high quality relative to ChatGPT.
• Multi-Model Support: Compatible with a wide array of models including Llama-2, Mistral, and various Hugging Face models, with a flexible registry for adding new ones.
• Chatbot Arena: The infrastructure that powers the crowdsourced battle platform where models are ranked via Elo ratings based on human preference.
• Flexible Grading Options: Supports single-answer grading, pairwise comparisons, and reference-guided grading to mitigate biases like verbosity or position bias.

How FastChat Compares

FastChat differs from tools like lm-evaluation-harness because it focuses on the conversational aspect of LLMs. While the harness is the gold standard for measuring raw capabilities (like math or coding) on a base model, FastChat is designed for the aligned model—the chatbot. It measures how a model behaves in a multi-turn dialogue, which is the primary way users actually interact with AI.

Compared to DeepEval, FastChat is more of a platform than a testing framework. DeepEval provides a suite of metrics for RAG pipelines and specific application outputs, whereas FastChat provides the entire infrastructure to serve the model and then evaluate it using a standardized benchmark like MT-Bench. FastChat is the right choice when you need to deploy a local LLM and want a standardized way to see how it compares to GPT-4 in terms of general chat quality.

Getting Started: Installation

Method 1: Using Pip

The fastest way to install FastChat is via pip. This is recommended for most users who want to use the serving and evaluation tools.

pip3 install fschat

To ensure all dependencies for model serving are met, it is often recommended to install the accelerate library:

pip install accelerate

Method 2: From Source (Editable Mode)

If you plan to contribute to the project or integrate custom models, install from source. This allows changes to the code to be reflected immediately.

git clone https://github.com/lm-sys/FastChat.git
cd FastChat
pip3 install -e .

Method 3: Specialized Installation for Evaluation

For those specifically targeting the LLM-as-a-judge and MT-bench pipeline, use the following installation command to include the necessary judge modules:

pip install -e ".[model_worker,llm_judge]"

Prerequisites: FastChat requires Python 3.9+ and a GPU with sufficient VRAM to load the models you intend to serve. For Mac users, rust and cmake may be required for certain dependency compilations.

How to Use FastChat

The most common workflow in FastChat is deploying a model and then interacting with it via the WebUI or API. This involves three main components: the controller, the model worker, and the frontend.

First, launch the FastChat controller, which manages the available models:

python3 -m fastchat.serve.controller

Next, launch a model worker to load a specific model (e.g., Vicuna or Llama-2). The worker handles the actual inference:

python3 -m fastchat.serve.model_worker --model-path lmsys/vicuna-7b-v1.5

Finally, launch the Gradio-based WebUI to chat with the model in your browser:

python3 -m fastchat.serve.gradio_web_server

Once these three services are running, you can open your browser and begin testing your model’s responses. If you prefer an API-driven approach, you can launch the OpenAI-compatible API server instead of the Gradio server:

python3 -m fastchat.serve.openai_api_server --host localhost --port 8000

Code Examples

FastChat allows you to interact with your local models using the official OpenAI Python SDK, making it a seamless replacement for commercial APIs.

Example 1: Basic Chat Completion

import openai

# Point the SDK to your local FastChat server
openai.api_key = "EMPTY"
openai.base_url = "http://localhost:8000/v1/"

model = "vicuna-7b-v1.5"
prompt = "Explain the concept of quantum entanglement in two sentences."

completion = openai.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": prompt}]
)

print(completion.choices[0].message.content)

This example demonstrates how to use the openai-python library to send a request to a local Vicuna model served by FastChat, treating it as if it were a commercial API.

Example 2: Generating Model Answers for MT-Bench

python gen_model_answer.py --model-path lmsys/vicuna-7b-v1.5 --model-id vicuna-7b-v1.5

This command uses the internal scripts to generate responses to the MT-bench questions, which can then be passed to an LLM judge for scoring.

Example 3: Running the LLM Judge

export OPENAI_API_KEY=your_key_here
python gen_judgment.py --model-list vicuna-7b-v1.5 --parallel 64

This script uses GPT-4 as a judge to evaluate the answers generated in the previous step, providing a quantitative score for the model’s quality.

Real-World Use Cases

FastChat is particularly powerful in scenarios where data privacy, model customization, and standardized evaluation are critical.

1. Local LLM Deployment for Privacy-Sensitive Data: A healthcare provider can use FastChat to serve a fine-tuned Llama-2 model locally on their own hardware. By using the OpenAI-compatible API, they can build a patient-triage chatbot without sending sensitive medical data to external servers.

2. Benchmarking New Model Iterations: An ML researcher developing a new fine-tuning recipe for a model can use the MT-bench pipeline to quickly quantify the performance gain. Instead of manual review, they can use the LLM-as-a-judge to see if the new version is objectively better at following instructions.

3. Creating a Model Comparison Hub: A company can deploy multiple models (e.g., Mistral-7B, Vicuna-13B, and Llama-3) using FastChat’s distributed serving system. They can then create an internal “Arena” where employees test different models side-by-side to determine which one is best suited for their specific business tasks.

4. Open-Source Model Selection: A developer choosing between several open-source models for a project can use FastChat’s pre-generated judgments and the Chatbot Arena leaderboard to make an informed decision based on human-preference data rather than raw benchmark scores.

Contributing to FastChat

FastChat is an open-source project maintained by LMSYS. Contributions are welcome and encouraged to expand the model support and serving infrastructure. Developers can contribute by adding new model templates to the model_registry.py file or by improving the evaluation scripts.

To contribute, users should first report bugs via GitHub Issues. For new features, it is recommended to open an issue to discuss the design before submitting a pull request. The project follows standard GitHub flow: fork the repository, create a feature branch, and submit a PR. All contributions are licensed under the Apache License 2.0.

Community and Support

FastChat is backed by a massive community of LLM researchers and developers. The primary hub for support is the GitHub Discussions section of the repository, where users can share configurations and troubleshoot installation issues.

Official channels include the LMSYS Discord server and their Twitter/X account for the latest updates on model releases and Chatbot Arena rankings. Detailed documentation is available directly in the /docs folder of the GitHub repository, covering everything from vLLM integration and OpenAI API compatibility.

Conclusion

FastChat is the definitive platform for anyone moving beyond simple API calls to commercial LLMs and into the world of self-hosted, open-source models. It provides the critical infrastructure needed to serve models with a professional API and serve them with a quantitative, research-backed evaluation framework.

Whether you are a researcher benchmarking new models or an engineer deploying a private chatbot, FastChat is the right choice when you need a local, scalable, and OpenAI-compatible serving system. While it requires some GPU resources to run, the trade-off is complete control over your data and your model’s behavior.

Star the repo, try the quickstart, and join the community to start deploying your own high-quality LLM chatbots.

Resources

FastChat GitHub Repository

Chatbot Arena (LMSYS)

FastChat Documentation