Please use the form below to provide your feedback. Because your feedback is valuable to us, the information you submit in this form is recorded in our issue tracking system (JIRA), which is publicly available. You can track the status of your feedback using the ticket number displayed in the dialog once you submit the form.

Deploy a Large Language Model (LLM)

  • how-to
    +
    The Capella Model Service can deploy Large Language Models (LLMs) close to your data in Capella to power your AI applications.

    The Capella Model Service offers endpoints for popular Large Language Models (LLMs) and supports features such as keyword filtering, caching, and guardrails.

    Prerequisites

    Procedure

    1. From your organization, go to AI Services  Models.

    2. Click Deploy New Model.

    3. Choose an LLM to deploy:

      1. Click View All Models.

      2. Click Type:All and deselect the Text to Embedding option, or use the search bar to find a specific LLM.

      3. Click the model you want to deploy.

      4. Click Use Selected Model.

    4. (Optional) Change the autogenerated name for the LLM that you’re deploying.

    5. Choose the AWS region where you want to deploy the model.

    6. Choose the compute and GPU size configuration to run the model.

      The minimum supported compute size available for the model in your chosen region is the default.

    7. (Optional) Apply advanced configuration options:

      If you change or enable any advanced configurations, such as value adds or security features, after deployment, your existing Model Service API keys will stop working, and you must create a new API key. For more information, see Value Adds and Security Features.
      Quantization

      Available to select models, reduce the model size and improve inference speed by applying a quantization level. Quantization can lead to a slight decrease in model accuracy, so test if it works well for your application.

      For more information, see Configure LLM Performance.

      Optimization

      Available to select models, use optimization techniques to enhance model performance and reduce latency. Choose from predefined profiles tailored for requirements such as low latency or high throughput.

      For more information, see Configure LLM Performance.

      Guardrails

      Use the guardrails configuration options to set up content moderation and safety features for your LLM.

      For more information, see Configure Guardrails.

      Jailbreak

      Detect and block jailbreak attempts to safeguard your model.

      For more information, see Configure Jailbreak Protection.

      Caching

      The conversation cache stores past conversations to improve the conversational experience by retrieving past or semantically similar conversations from the cache.

      For more information, see Configure Caching.

      Async Processing

      Increase throughput by processing jobs asynchronously when system capacity becomes available. This allows task queuing, where tasks are handled as resources permit, improving overall efficiency.

      For more information, see Configure Async Processing.

      Keyword Filtering

      You can add up to 10 keywords separated by commas to filter from user prompts and responses.

      For more information, see Configure Keyword Filtering.

    8. Click Deploy Model.

    Next Steps

    The Models page opens with your model in a deploying state. Once the model has finished deploying, you can view the model details and manage the model by expanding its listing on the Models page.

    To create API keys for your deployed model, see Generate Model Service API Keys.