Introduction to TorchServe CPP

TorchServe CPP is an experimental release of the popular TorchServe framework, designed to facilitate the deployment of machine learning models in C++. Although the project is currently in a state of limited maintenance, it offers a robust platform for serving models with high performance.

This guide will walk you through the main features, installation process, and usage examples of TorchServe CPP, enabling you to leverage its capabilities effectively.

Main Features of TorchServe CPP

• High Performance: Built with C++17, TorchServe CPP is optimized for speed and efficiency.
• Custom Handlers: Easily create custom handlers for various model types, including TorchScript, ONNX, and more.
• Docker Support: Simplifies the setup process with Docker containers for both CPU and GPU support.
• Extensive Documentation: Comprehensive guides and examples to help you get started quickly.

Technical Architecture and Implementation

The architecture of TorchServe CPP is designed to be modular and extensible. It allows developers to implement custom handlers that can interact with various model formats. The backend runs as a process similar to the Python backend, supporting torch scripted models by default.

Key components include:

• BaseHandler: The core class for creating custom handlers.
• Model Store: A dedicated directory for storing models.
• API Endpoints: RESTful APIs for model inference and management.

Setup and Installation Process

To get started with TorchServe CPP, follow these steps:

Requirements

• C++17
• GCC version: gcc-9
• cmake version: 3.26.4+
• Linux

Using Docker

For convenience, you can use a Docker container:

cd serve/docker
# For CPU support
./build_image.sh -bt dev -cpp
# For GPU support
./build_image.sh -bt dev -g [-cv cu121|cu118] -cpp

Running the Container

# For CPU support
docker run [-v /path/to/build/dir:/serve/cpp/build] -it pytorch/torchserve:cpp-dev-cpu /bin/bash
# For GPU support
docker run --gpus all [-v /path/to/build/dir:/serve/cpp/build] -it pytorch/torchserve:cpp-dev-gpu /bin/bash

Installing Dependencies

cd serve
python ts_scripts/install_dependencies.py --cpp --environment dev [--cuda=cu121|cu118]

Building the Backend

cd cpp
mkdir build && cd build
cmake ..
make -j && make install

Usage Examples and API Overview

Once installed, you can run TorchServe with the following command:

mkdir model_store
torchserve --ncs --start --model-store model_store

Creating a Custom Handler

To create a custom handler, extend the BaseHandler class:

class CustomHandler(BaseHandler):
  def initialize(self, ctx):
    # Initialization code here

  def preprocess(self, requests) -> List[Any]:
    # Preprocessing code here

  def inference(self, input_batch: List[torch.Tensor]) -> List[torch.Tensor]:
    # Inference code here

  def postprocess(self, inference_output: List[torch.Tensor]) -> List[Any]:
    # Postprocessing code here

For more examples, check the examples directory.

Community and Contribution Aspects

Although TorchServe CPP is in limited maintenance mode, contributions are still welcome. If you are interested in contributing, please follow the guidelines outlined in the contributing guidelines.

For issues, check the issues page for opportunities to contribute.

License and Legal Considerations

TorchServe CPP is licensed under the Apache License 2.0. You can find the full license text in the repository. Ensure compliance with the license terms when using or modifying the code.

Conclusion

TorchServe CPP provides a powerful framework for deploying machine learning models in C++. Despite its limited maintenance status, it remains a valuable tool for developers looking to leverage C++ for model serving.

For more information, visit the official repository.

FAQs