56 lines
1.9 KiB
Markdown
56 lines
1.9 KiB
Markdown
---
|
|
name: cost-aware-llm-pipeline
|
|
description: Cost optimization for AI development pipelines — model routing, budget tracking, token optimization. Use when managing multi-step AI workflows, reducing API costs, or selecting models for tasks.
|
|
---
|
|
|
|
# Cost-Aware LLM Pipeline
|
|
|
|
Based on ECC cost-aware-llm-pipeline patterns.
|
|
|
|
## Model Selection Strategy
|
|
|
|
Match model capability to task complexity:
|
|
|
|
| Task Type | Model Tier | Examples |
|
|
|-----------|-----------|----------|
|
|
| Simple (summarize, classify, format) | Small/Fast | Claude Haiku, GPT-4o-mini |
|
|
| Medium (code fixes, analysis, rewrite) | Mid | Claude Sonnet, GPT-4o |
|
|
| Complex (architecture, debugging, planning) | High | Claude Opus, GPT-4o, Gemini Pro |
|
|
| Creative (design, brainstorming) | Variable | Depends on breadth needed |
|
|
|
|
## Token Optimization Techniques
|
|
|
|
1. **Trim system prompts**: Remove redundant instructions. Keep only task-specific rules.
|
|
2. **Compress context**: Use summaries instead of full files. Skip irrelevant code.
|
|
3. **Background processes**: Run long tasks (test suites, builds) async. Don't waste tokens waiting.
|
|
4. **Chunk large files**: Read only relevant sections with offset/limit.
|
|
5. **Avoid loops**: Don't poll in a loop. Use proper timeouts and wait mechanisms.
|
|
|
|
## Budget Tracking
|
|
|
|
```bash
|
|
# Check session cost (if available)
|
|
session_status
|
|
|
|
# Monitor token usage
|
|
# Track: input_tokens, output_tokens, cost_cents
|
|
```
|
|
|
|
## Cost Reduction Rules
|
|
|
|
- Use the cheapest model that still works
|
|
- Batch similar requests together
|
|
- Cache responses when possible
|
|
- Avoid regenerating the same output
|
|
- Set explicit max_tokens for generation
|
|
- Use structured output (JSON) to reduce retries
|
|
- Prefer targeted file reads over broad "read everything"
|
|
|
|
## Pipeline Design
|
|
|
|
When building multi-step AI workflows:
|
|
1. Step 1: Plan/analyze with mid-tier model
|
|
2. Step 2: Implement with appropriate model for code
|
|
3. Step 3: Verify/test with cheapest model
|
|
4. Step 4: Review with high-tier model only if needed
|