GenAI feature considerations¶

When working with generative AI capabilities in DataRobot, consider the following. Note that as the product continues to develop, some considerations may change.

Trial users: See the considerations specific to the DataRobot free trial, including supported LLM base models.

General considerations¶

• If a multilingual dataset exceeds the limit associated with the multilingual model, DataRobot defaults to using the jinaai/jina-embedding-t-en-v1 embedding model.

• There is no support for adding external/custom vector databases or custom LLMs through the UI.

• Deployments that require an association ID for predictions cannot be used as bring-your-own (BYO) models.

• Deployments created from custom models with training data attached that have extra columns cannot be used unless column filtering is disabled on the custom model.

• When using LLMs that are either BYO or deployed from the playground and require a runtime parameter to point to the endpoint associated with their credentials: Be aware of the vendor's model versioning and end-of-life schedules. As a best practice, use only endpoints that are generally available when deploying to production. (Models provided in the playground manage this for you.)

• Note that an API key named [Internal] DR API Access for GenAI Experimentation is created for you when you access the playground or vector database in the UI.

• BYO embeddings functionality is available for self-managed users only. Note that when many users run VDB creation jobs in parallel, if using BYO embeddings, LLM playground functionality may be degraded until VDB creation jobs complete.

• Only one aggregated metric job can run at a time. If an aggregation job is currently running, the Configure aggregation button is disabled and the "Aggregation job in progress; try again when it completes" tooltip appears.

LLM availability¶

The following table describes the availability of LLMs:

* Available for trial users, cluster-dependent.

Sharing and permissions¶

The following table describes GenAI component-related user permissions. All roles (Consumer, Editor, Owner) refer to the user's role in the Use Case; access to various function are based on the Use Case roles:

Playground considerations¶

• Playgrounds can be shared for viewing, and users with editor or owner access can perform additional actions within the shared playground, such as creating blueprints. While non-creators cannot prompt an LLM blueprint in the playground, they can make a copy and submit prompts to that copy.

• You can only prompt LLM blueprints that you created (i.e., in both configuration and comparison view). To see the results of prompting another user’s LLM blueprint in a shared Use Case, copy the blueprint and then you can chat with the same settings applied.

• Each user can submit 5000 LLM prompts per day across all LLMs, where deleted prompts and responses are also counted. However, only successful prompt response pairs are counted and bring-your-own (BYO) LLM calls are not part of the count. Limits for trial users are different, as described here.

Vector database considerations¶

By default, DataRobot uses the Facebook AI Similarity Search (FAISS) Vector Database. When determining the number of contexts to retrieve from the VDB, DataRobot allocates 3/4 of the excess token budget (after system prompt, user prompt, and max completion length) to retrieved documents and the rest to chat history (if applicable).

The following sections describe considerations related to vector databases:

• Supported dataset types
• Dataset limits

Supported dataset types¶

When uploading datasets for creating a vector database, the only supported format is zip. DataRobot then processes the .zip to create a .csv containing text columns with an associated reference ID (file path) column. The reference ID column is created automatically when the zip is uploaded. All files should be either in the root of the archive or in a single folder inside an archive. Using a folder tree hierarchy is not supported.

Regarding file types, DataRobot provides the following support:

• .txt documents

• PDF documents

  • Text-based PDFs are supported.
  • Image-based PDFs are not fully supported. That is, images are generally ignored but do not lead to errors.
  • Documents with mixed image and text content are supported; only the text is parsed.
  • Single documents consisting only of images result in empty documents and are ignored.
  • Datasets consisting of image-only documents (no text) are not processable.

• .docx documents are supported but older .doc format is not supported.

• .md documents, and the .markdown variant, are supported.

• A mix of all supported document types in a single dataset is allowed.

Dataset limits¶

The global 1GB dataset limit is applied during vector database creation, after the text is extracted from the document. Additional dynamic limits are listed below:

• jinaai/jina-embedding-t-en-v1: Supported to the 1GB global limit
• sentence-transformers/all-MiniLM-L6-v2: Supported to the 650MB limit
• cl-nagoya/sup-simcse-ja-base: Supported to the 250MB limit
• Multilingual-e5-base: Supported to the 250MB limit
• E5-base-v2: Supported to the 250 MB limit
• E5-large-v2: Supported to the 100MB limit

Playground deployment considerations¶

Consider the following when registering and deploying LLMs from the playground:

• Setting API keys through the DataRobot credential management system is supported. Those credentials are accessed as environment variables in a deployment.

• Registration and deployment is supported for:

  • All base LLMs in the playground.

  • LLMs with vector databases.

• The creation of a custom model version from an LLM blueprint associated with a large vector database (500+ MB) can be time consuming. You can leave the model workshop while the model is created and will not lose your progress.

Trial user considerations¶

The following considerations apply only to DataRobot free trial users:

• You can create up to 15 vector databases, computed across multiple Use Cases. Deleted vector databases are included in this count.

• You can make 1000 LLM API calls, where deleted prompts and responses are also counted. However, only successful prompt response pairs are counted.

See also the section on LLM availability.