Introduction

Managing massive volumes of observability data—logs, traces, and metrics—often leads to a costly trade-off between search speed and infrastructure expenses. Developers typically struggle with the high Total Cost of Ownership (TCO) associated with traditional search engines that require expensive, high-performance disks. Quickwit, a cloud-native search engine with over 11k GitHub stars, solves this by allowing users to index and search petabytes of data directly on cheap cloud object storage like Amazon S3, significantly reducing costs without sacrificing query performance.

What Is Quickwit?

Quickwit is a cloud-native search engine designed specifically for observability data, such as logs and traces, that allows for high-performance searching and analytics directly on cloud object storage. Built in Rust for maximum efficiency and safety, Quickwit separates computing from storage, which enables it to be highly resource-efficient and scalable to petabytes of data. It is an open-source project licensed under the Apache 2.0 license, allowing for flexible deployment and modification.

The project’s core philosophy is to treat cloud object storage as the primary data store, using an inverted index and columnar storage to ensure that search response times remain under a second, even when querying billions of records.

Why Quickwit Matters

Traditional search engines like Elasticsearch or Splunk often require expensive SSDs to maintain performance, which becomes prohibitively expensive as data volumes grow into the terabytes and petabytes. This “storage tax” storage tax often forces teams to reduce their data retention periods or drop logs to save money, which is a critical failure during incident response.

Quickwit changes this paradigm by leveraging the virtually unlimited and cheap capacity of object storage. By optimizing the way indexes are built and queried, Quickwit allows teams to retain more data for longer periods while keeping infrastructure costs low. This makes it a viable alternative for organizations that have outgrown the cost of the ELK stack or are looking for a more modern, serverless-friendly architecture.

With a growing community and adoption by companies at petabyte scale, Quickwit is becoming the standard for cost-efficient, long-term log retention and searchability.

Key Features

• Cloud-Native Storage: Quickwit indexes data directly onto object storage (S3, GCS, Azure Blob), eliminating the need for expensive local disks.
• Separation of Compute and Storage: This architecture allows you to scale your search nodes independently of your data, enabling a truly serverless search experience.
• High Performance: Leveraging Rust and the Tantivy library, Quickwit provides sub-second response times for queries across billions of records.
• Columnar Storage: In addition to the inverted index, Quickwit uses columnar storage for efficient aggregation and analytics, reducing the amount of data read from the cloud.
• Observability Integration: Designed for logs and traces, it integrates seamlessly with tools like Grafana for visualization and alerting.
• Resource Efficiency: Because it doesn’t require a heavy JVM, it uses significantly less RAM and CPU than Java-based search engines.
• Petabyte Scalability: The system is designed from the ground up to handle petabytes of data without the need for complex sharding or manual re-indexing.
• Open Source: Licensed under Apache 2.0, ensuring the community can contribute and extend the function of the search engine.

How Quickwit Compares

Quickwit occupies a unique middle ground between the high-performance, high-cost nature of Elasticsearch and the low-cost, low-performance (for full-text search) nature of Grafana Loki. While Elasticsearch provides the fastest possible search response times by keeping data on local SSDs, it is prohibitively expensive at scale. Loki, on the other hand, only indexes metadata (labels), meaning that searching for a specific string within a log line requires a “grep-like” scan of the data, which can be slow for large time ranges.

Quickwit provides the best of both worlds: the full inverted index of Elasticsearch (allowing for lightning-fast full-text search) and the cost-efficiency of object storage. This makes it the ideal choice for teams that need to search through petabytes of logs without spending a fortune on infrastructure.

Getting Started: Installation

Quickwit can be deployed in several ways depending on your environment. The most common methods are via Docker or as a binary installation.

Docker Installation

The fastest way to get Quickwit running is using the official Docker image. Run the following command to start a single-node cluster:

docker run -d --name quickwit -p 7280:7280 -v $(pwd)/qwdata:/quickwit/qwdata quickwit/quickwit

Binary Installation

For those who prefer to run Quickwit as a system service, you can download the pre-compiled binaries from the GitHub releases page. After downloading and extracting, you can start the search engine with:

./quickwit run

Prerequisites

To use Quickwit’s cloud-native capabilities, you will need an S3-compatible object storage bucket. You must configure your AWS credentials or use a MinIO instance for local development.

How to Use Quickwit

The basic workflow in Quickwit is to define an index configuration, create the index, and then upload your data. Unlike traditional search engines, Quickwit is designed for “batch” indexing, where data is written to object storage and then indexed.

Quickwit provides a REST API for managing indexes and the searcher. Searchers are the components that query the index files stored on S3.

Once your index is created, you can perform searches using the following API call:

curl -X POST "http://localhost:7280/api/v1/search" -H "Content-Type: application/json" -d '{"query": "error", "index": "my-logs"}'

Code Examples

Quickwit provides a comprehensive API for indexing and searching. Below are examples of how to interact with the search engine.

Creating an Index

The following example shows how to create a log index with a specific schema definition for timestamps and fields.

curl -X POST "http://localhost:7280/api/v1/index/my-logs" -H "Content-Type: application/json" -d '{"index_config": {"doc_mapping": {"field_mappings": {"timestamp": {"type": "datetime", "analyzer": "standard"}, "level": {"type": "text", "analyzer": "standard"}, "message": {"type": "text", "analyzer": "standard"} } } } }'

Searching for Logs

Quickwit uses a powerful query language that allows for filtering by time range and time-based partitioning. la

curl -X POST "http://localhost:7280/api/v1/search" -H "Content-Type: application/json" -d '{"query": "level:error AND timestamp:[2023-01-01T00:00:00Z TO 2023-01-02T00:00:00Z]", "index": "my-logs"}'

Real-World Use Cases

Quickwit is most effective when the volume of observability data is so high that traditional search engines become too expensive to maintain.

• Long-Term Log Retention: A company needs to retain logs for 1 year for compliance reasons but doesn’t want to pay for SSDs for an entire year’s worth of data. They use Quickwit to store and search logs on S3, reducing costs by 90% compared to Elasticsearch.
• Security Audit Logs: Security teams use Quickwit to index audit logs from across their entire infrastructure. Because audit logs are rarely searched but must be kept for long periods, Quickwit’s object-storage-first approach is perfect for this use case.
• Cloud-Native Observability: A team using AWS Lambda and other serverless components uses Quickwit to centralize their logs. By deploying Quickwit as a serverless searcher, they maintain a consistent architecture across their entire stack.
• Trace Analysis: Developers use Quickwit to index distributed traces. Because traces are high-volume and high-cardinality data, Quickwit’s columnar storage allows them to quickly filter and aggregate traces by service name or trace ID.

Contributing to Quickwit

Quickwit is an open-source project with a strong commitment to community growth. Contributions are welcome through the following standard GitHub flow:

Contributions start by reporting bugs or suggesting new features via GitHub Issues. If you are looking to contribute code, you can fork the repository, create a feature branch, and then submit a pull request. The maintainers provide a detailed contributing guide in the repository to help new contributors get their environment set up.

The project also adheres to a Code of Conduct to ensure a respectful and collaborative environment for all participants.

Community and Support

Quickwit has a growing ecosystem of tools and plugins. The primary hub for community interaction is the official GitHub Discussions forum, where users can ask questions, share their experiences, and collaborate with the maintainers.

Detailed technical documentation is available at the official Quickwit documentation site, which covers everything from installation to advanced indexing strategies. For real-time support and community help, users often turn to the GitHub Discussions forum and the project’s official social media channels.

Conclusion

Quickwit is the right choice for teams that have reached the “storage tax” limit of traditional search engines. It is the ideal solution for when you need the full-text search capabilities of Elasticsearch but the cost-efficiency of object storage. When you are managing petabytes of data and the need for sub-second search response times remains critical, Quickwit is the most viable alternative to the ELK stack.

While it is a highly efficient engine, it is important to note that it is optimized for observability data (logs and traces) observability data. If your use case requires real-time, high-frequency updates to existing documents, it is not the best tool for the tool for this specific task, as it is designed for append-only log data.

Star the repo, try the quickstart, and join the community to start reducing your observability costs today.

Resources

Explore more about the project through these official links:

• Quickwit GitHub Repository
• Quickwit Community Discussions
• Official Quickwit Documentation