LLM Nodes
Last updated
Last updated
The LLM nodes in Dynamiq allow users to integrate various LLM providers for natural language processing tasks such as text generation, question answering, and language comprehension. Through connections to services like OpenAI, Anthropic, and custom LLMs, users can configure and optimize workflows for various use cases, including customer support automation, content generation, and more.
Dynamiq supports a variety of LLM providers, enabling users to choose the best model for their specific use case. Each provider offers unique models with varying capabilities, costs, and performance characteristics. Below is a list of the LLM providers integrated within Dynamiq:
Each LLM node requires careful configuration to ensure accurate and efficient operation. Follow these steps to set up an LLM node:
Step 1: Connection Configuration
Connection: Each LLM node must be linked to its respective service provider.
API Keys: Obtain API keys or tokens for each provider by following the documentation links.
Step 2: Prompt Configuration
Prompt Library: Dynamiq allows users to select prompts from a library or create inline prompts.
Dynamic Prompting: Prompts can be customized based on input parameters to generate diverse responses.
These parameters allow you to control the behavior and performance of the LLM node, optimizing it for various applications:
JSONPath Selection: Use JSONPath to filter and structure input data, allowing for precise extraction of information.
Prompt Templates: Dynamically create prompts by inserting runtime parameters.
Filtering: Filter responses to retrieve only relevant data.
Structured Outputs: Dynamiq supports different output formats, including plain text, JSON.
Selecting the right model is crucial for balancing cost, speed, and quality:
Complex Tasks: Use more advanced models like GPT-4 or Claude for complex outputs.
Cost-Efficiency: Opt for smaller models like GPT-4o-mini / claude haiku for simpler tasks to reduce expenses.
Provider-Specific Features: Some providers offer unique features like function calling; refer to provider documentation for details.
Effective prompt design can significantly impact model performance:
Clear Instructions: Use specific language to minimize ambiguity.
Contextual Information: Include background details to guide the model's response.
Testing: Test prompts across various inputs to ensure consistency.
To ensure seamless operation and improve resilience, configure error handling mechanisms within your workflow:
Adjust these settings based on workflow requirements and provider limitations to avoid unnecessary delays or costs.
Rate Limit Handling
Consider configuring rate limits to stay compliant with provider-specific quotas. Utilize the backoff rate and interval settings to manage requests dynamically and avoid reaching rate limits, which can lead to throttling or blocked requests.
Optimize your LLM node's performance by following these strategies:
Response Caching: Enable caching to reduce redundant requests and improve speed.
Batching Requests: Group requests to process multiple items simultaneously, improving efficiency.
Token Usage Monitoring: Track token consumption to control costs and manage API quotas.
Dynamiq LLM nodes offer multiple modes to tailor responses to specific needs:
DEFAULT: Standard text-based response generation.
STRUCTURED_OUTPUT: Provides structured outputs in JSON format.
Enable streaming for applications requiring real-time feedback, such as customer support or live content generation.
Dynamiq's custom LLM node allows users to integrate models deployed on their own servers or models compatible with OpenAI's API syntax through OpenRouter. This flexibility provides seamless integration for both proprietary and third-party LLMs.
Add Custom LLM Node: Select the Custom LLM node from the panel.
Choose Model: Enter the model name. Dynamiq supports manual input for models not listed in auto-suggestions.
Prompt Configuration: Define prompts using the inline prompt editor or select from the library.
Connect & Test: Connect to your server or OpenRouter and test the configuration.
Use Custom LLM for experimental models or those deployed within secure environments.
Provider | Description | Models | API Documentation |
---|---|---|---|
Parameter | Description | Example Values |
---|---|---|
Parameter | Description | Example Values |
---|---|---|
Token Type | Example Input | Cost Calculation |
---|---|---|
OpenAI
Advanced language models suitable for complex language tasks.
GPT-4o, GPT-4o-mini
Anthropic
Models like Claude for conversational and generative text applications.
Claude 3.5 Sonnet, Claude 3.5 Haiku
Cohere
Text generation and embedding models, often used for content creation and analysis.
Command R+, Command R
Gemini
Models optimized for information retrieval and summarization tasks.
Gemini Pro 1.5, Gemini Flash 1.5
AWS Bedrock
Cloud-based service offering models optimized for enterprise and general-purpose NLP tasks.
Various proprietary models
Groq
Models optimized for high-performance inference, often used in real-time applications.
Llama 3.1, Llama 3.2
Mistral
Specialized models focusing on efficiency in text generation and comprehension.
Mistral Large, Small, Embed
Together AI
Provides collaborative NLP tools for tasks such as summarization and translation.
Llama, Gemma, Mistral models
Hugging Face
Offers a vast repository of models, from general-purpose transformers to specialized NLP models.
Open-source models
IBM WatsonX
IBM's suite of AI tools for enterprise applications, including data analysis and NLP.
Granite
Azure AI
Microsoft's cloud-based language models suitable for enterprise and developer-focused applications.
OpenAI models
Replicate
Models focused on reproducible AI research, useful for scientific and technical applications.
Open-source models
SambaNova
AI models optimized for enterprise-scale NLP and other machine learning applications.
Llama family
Cerebras
AI models tailored for high-performance NLP and deep learning tasks in large-scale environments.
Open-source models
DeepInfra
Specializes in deploying high-performance AI infrastructure with NLP capabilities.
Open-source models
Custom LLM
Allows integration with any OpenAI-compatible or custom-deployed models, ideal for proprietary setups.
Custom models (OpenRouter compatible, self-hosted)
Model
Specifies the LLM model to use. The model field supports free-text input with auto-suggestions, allowing for immediate access to new models. Ensure the model name matches the provider’s offerings.
gpt-4o, claude-3-haiku
Temperature
Controls the level of randomness in the model’s output. Lower values (close to 0) make responses more deterministic, suitable for tasks requiring precision. Higher values (close to 1) encourage creative responses, ideal for content generation.
0.2
for deterministic, 0.7
for creative
Max Tokens
Sets the maximum number of tokens the model can generate in its response. Useful for limiting the output length to control costs or meet specific response size requirements.
500
, 1000
Streaming
Enables token-by-token streaming of responses, providing real-time feedback. Streaming is recommended for use cases requiring quick insights, such as interactive applications.
Enabled
, Disabled
Interval
Sets the delay (in seconds) before the first retry attempt. Must be greater than 0.
2
seconds
Max Attempts
Specifies the maximum number of retry attempts. Set to 0
or greater to determine retry limits.
3
attempts
Backoff Rate
Multiplier that increases the retry interval for each subsequent attempt. Helps to reduce load on the system progressively. Must be 1
or greater.
2.0
Timeout
Sets a timeout limit (in seconds) for each attempt. If exceeded, the process fails to prevent excessive delays.
10
seconds
Prompt Tokens
250 tokens
(Prompt tokens / 1000) * Cost per 1K tokens
Completion Tokens
500 tokens
(Completion tokens / 1000) * Cost per 1K tokens