Setting up Kestra

Video: Setting up Kestra

Here's how to set up everything you need to run the example flows in this module.

Prerequisites

This module requires Docker with Docker Compose to run Kestra locally. Docker Desktop is the easiest way to get both on Mac and Windows. If you don't have Docker installed, set that up before proceeding.

Step 1: Start Kestra

This module includes a docker-compose.yml with Kestra pre-configured:

cd 03-orchestration
docker compose up -d

Once the container starts, access the Kestra UI at http://localhost:8080.

To shut down Kestra:

docker compose down

Step 2: Obtain API Keys

Gemini API Key (Required)

1. Visit Google AI Studio
2. Sign in with your Google account
3. Click "Create API Key" and copy your key

The free tier is sufficient for light use, but rate limits are relatively low — you may hit quota quickly if you run the agent and multi-agent flows repeatedly. If you run into 429 Resource Exhausted errors, wait a minute before retrying, or consider upgrading to a paid tier.

OpenAI API Key (Required for flow 3)

1. Visit platform.openai.com and sign in or create an account
2. Go to API keys and create a new key

Tavily API Key (Required for web search: flows 3, 5, and 6)

1. Visit Tavily
2. Sign up for the free tier
3. Get your API key from the dashboard

The free tier includes 1,000 searches/month.

Step 3: Configure API Keys in Kestra

Kestra reads secrets from environment variables prefixed with SECRET_ where the value is base64-encoded. Export your keys before starting Kestra:

export GEMINI_API_KEY="your-gemini-api-key-here" # required
export SECRET_GEMINI_API_KEY=$(echo -n $GEMINI_API_KEY | base64) # required
export SECRET_OPENAI_API_KEY=$(echo -n "your-openai-api-key-here" | base64)   # required for flow 3
export SECRET_TAVILY_API_KEY=$(echo -n "your-tavily-api-key-here" | base64)   # optional

Then start (or restart) Kestra:

docker compose up -d

In flows, reference secrets with {{ secret('GEMINI_API_KEY') }} — omit the SECRET_ prefix when calling secret().

[!WARNING] Never commit API keys to Git!

Step 4: Import Example Flows

cd 03-orchestration

# Adjust username and password to match your Kestra setup
curl -X POST -u 'admin@kestra.io:Admin1234!' http://localhost:8080/api/v1/flows/import -F fileUpload=@flows/1_chat_without_rag.yaml
curl -X POST -u 'admin@kestra.io:Admin1234!' http://localhost:8080/api/v1/flows/import -F fileUpload=@flows/2_chat_with_rag.yaml
curl -X POST -u 'admin@kestra.io:Admin1234!' http://localhost:8080/api/v1/flows/import -F fileUpload=@flows/3_rag_with_websearch.yaml
curl -X POST -u 'admin@kestra.io:Admin1234!' http://localhost:8080/api/v1/flows/import -F fileUpload=@flows/4_simple_agent.yaml
curl -X POST -u 'admin@kestra.io:Admin1234!' http://localhost:8080/api/v1/flows/import -F fileUpload=@flows/5_web_research_agent.yaml
curl -X POST -u 'admin@kestra.io:Admin1234!' http://localhost:8080/api/v1/flows/import -F fileUpload=@flows/6_multi_agent_research.yaml

Alternatively, copy-paste the flow YAML directly into Kestra's UI.

Step 5: Run Your First Agent

1. Open Kestra UI at http://localhost:8080
2. Navigate to the zoomcamp namespace
3. Find the 4_simple_agent flow and click "Execute"
4. Leave default inputs or customize them
5. Watch the execution and review the outputs
6. Then run 5_web_research_agent and 6_multi_agent_research and analyze the logs and outputs

Question 1: Context Engineering

Try the following experiment:

1. Open ChatGPT in a private browser window: https://chatgpt.com
2. Enter this prompt: "Create a Kestra flow that loads NYC taxi data from CSV to BigQuery"
3. Then, use Kestra's AI Copilot with the same prompt

After trying the same prompt in ChatGPT vs Kestra's AI Copilot, what is the primary reason AI Copilot generates better Kestra flows?

• AI Copilot uses a more powerful model
• ✅ AI Copilot has access to current Kestra plugin documentation
• AI Copilot uses more tokens
• AI Copilot has internet access

Question 2: RAG vs No RAG

Run both 1_chat_without_rag.yaml and 2_chat_with_rag.yaml in the Kestra UI. Read the execution logs for each.

The non-RAG response about Kestra 1.1 features is best described as:

• ✅ Accurate and specific, matching the actual release notes
• Vague, generic, or fabricated — the model guesses from training data
• Empty — the model refuses to answer without context
• Identical to the RAG version

Question 3: Token usage — short summary

Run 4_simple_agent.yaml with summary_length = short (leave the other inputs as defaults).

Open the execution logs and find the token usage logged by the log_token_usage task.

What is the approximate output token count for multilingual_agent?

• 5-15 tokens
• ✅ 60-100 tokens
• 200-400 tokens
• 500+ tokens

Question 4: Token usage — long summary

Run 4_simple_agent.yaml again with summary_length = long.

Compare the multilingual_agent output token count to your result from Question 3. Roughly how many times more output tokens does the long summary use?

• About the same (within 20%)
• ✅ 2-5x more
• 10-20x more
• 50x more

Question 5: Modifying a flow

Open 4_simple_agent.yaml in the Kestra flow editor. Find the english_brevity task and change its prompt from asking for exactly 1 sentence to asking for exactly 3 sentences.

Save the flow, then run it with summary_length = long.

Compare the english_brevity output token count to the original 1-sentence version (also with summary_length = long). How do they compare?

• About the same (within 20%)
• ✅ 2-4x more
• 5-10x more
• 10x+ more

Question 6: Best Practices

Based on what you learned in this module, for production workflows requiring deterministic, repeatable results with strict compliance requirements (e.g., financial reporting, workflows in highly regulated industries), which approach is most appropriate?

• Always use AI agents for maximum flexibility and adaptation
• ✅ Use traditional task-based workflows for predictability and auditability
• Use only RAG without agents for better performance
• Use web search tools exclusively to ensure current data