redhat/granite-3-3b-a800m-instruct repository overview

This Docker image is pre-configured to run IBM's Granite 3.0 large language model (LLM), optimized for local large language model operations. The image includes all necessary dependencies, libraries, and runtime environments for seamless deployment and interaction with Granite 3.0.

Key components:

1. Base Image: Starts with a lightweight, base image: ubi9
2. Model Runtime Environment: The image provides llama.cpp as the model serving runtime

To run the image in CLI mode use the command docker run --ipc=host -it redhat/granite-3-3b-a800m-instruct

To run the image as an openAI compatible API server run docker run --network host --ipc=host -it redhat/granite-3-3b-a800m-instruct -s