Genetic Strategy Optimization - Implementation Guide

Created: 2025-10-30 Status: ✅ Phase 1 Complete - Ready for Production Use Implementation Time: ~2 hours (from spec to working system)

---

🎉 What Was Implemented

You now have a complete genetic algorithm optimization system for trading strategies, inspired by NexusTrade Aurora's approach.

✅ Components Built

1. tools/optimizable_strategy.py - Base class for all optimizable strategies
2. Defines parameter space interface
3. Provides metric calculation (Sharpe, returns, drawdown, etc.)

4. Handles train/validation splits

5. tools/genetic_optimizer.py - NSGA-II genetic algorithm implementation

6. Multi-objective optimization (Sharpe + Return - Drawdown)
7. Pareto frontier generation
8. Overfitting detection via validation

9. Visualization and reporting

10. tools/mean_reversion_optimized.py - Your Mean Reversion strategy (optimizable)

11. 6 optimizable parameters (RSI, Bollinger Bands, stops, position sizing)
12. Full backtest engine with Alpaca data

13. Compatible with genetic optimizer

14. tools/optimize_strategy.py - Command-line interface

15. Easy-to-use CLI for running optimizations
16. JSON reports and Pareto plots
17. Configurable constraints and objectives

---

🚀 Quick Start

Example 1: Quick Test (2 minutes)

cd /Users/bertfrichot/mem-agent-mcp/tools

# Small optimization to test the system
uv run python3 optimize_strategy.py \
  --strategy mean_reversion \
  --symbols AAPL,MSFT \
  --population 10 \
  --generations 10 \
  --train-start 2024-01-01 \
  --train-end 2024-06-30 \
  --val-start 2024-07-01 \
  --val-end 2024-10-30

Example 2: Full Optimization (Recommended - 10-15 minutes)

# Full optimization with 5 years of data
uv run python3 optimize_strategy.py \
  --strategy mean_reversion \
  --symbols AAPL,MSFT,NVDA,GOOGL,META \
  --train-start 2020-01-01 \
  --train-end 2024-03-31 \
  --val-start 2024-04-01 \
  --val-end 2024-10-30 \
  --population 20 \
  --generations 20 \
  --min-win-rate 0.50 \
  --max-drawdown 0.20 \
  --output reports/mean_reversion_optimized.json \
  --plot reports/mean_reversion_pareto.png

Expected Results: - 400 backtests (20 pop × 20 gen) - Runtime: ~10-15 minutes - Output: JSON report + Pareto frontier plot - Top 5 strategies ranked by composite score

---

📊 How It Works

Multi-Objective Optimization (NSGA-II)

The optimizer doesn't just maximize returns. It optimizes 3 objectives simultaneously:

1. Maximize Sharpe Ratio (risk-adjusted returns)
2. Maximize Total Return (raw performance)
3. Minimize Max Drawdown (worst decline)

Result: Pareto frontier of 10-20 solutions with different trade-offs: - Conservative strategy: High Sharpe (2.0), moderate return (28%), low drawdown (8%) - Aggressive strategy: Lower Sharpe (1.5), high return (50%), higher drawdown (18%) - You choose based on your risk tolerance

Parameter Space (Mean Reversion)

The optimizer searches across 6 parameters:

Parameter	Range	Current	Optimized (Example)
RSI Oversold	20-40	35	28 (tighter entry)
RSI Overbought	60-80	70	72 (later exit)
BB Std Dev	1.5-3.0	2.0	2.2 (wider bands)
Stop Loss %	5%-15%	8%	8.5% (slightly wider)
Position Size	$300-$1000	$500	$650 (more capital)
Max Positions	3-8	5	4 (more focused)

Total combinations: ~1.8 million

Manual testing time: 21 days (assuming 1 backtest/second)

Genetic algorithm time: 7 minutes (400 backtests)

Speedup: 4,500X faster

Train/Validation Split

Critical for avoiding overfitting:

Training (70% of data):
  ├─ Used for fitness evaluation
  ├─ Genetic algorithm optimizes on this
  └─ Example: 2020-01-01 to 2024-03-31

Validation (30% of data):
  ├─ Used ONLY for final evaluation
  ├─ Tests generalization to unseen data
  └─ Example: 2024-04-01 to 2025-10-30

Overfitting Detection: - ✅ Acceptable: Training Sharpe 2.0 → Validation Sharpe 1.4 (30% degradation) - ❌ Overfit: Training Sharpe 3.5 → Validation Sharpe 0.4 (89% degradation)

---

📋 Output Explained

1. Console Output

🏆 Top 5 Strategies (by composite score):

1. Composite Score: 0.842
   Training  → Sharpe: 2.15 | Return: 38.7% | Drawdown: 12.3%
   Validation→ Sharpe: 1.51 | Return: 27.2% | Drawdown: 16.8%
   Degradation: 29.8%
   Win Rate: 52.3% | Trades: 48
   Parameters: {
      "rsi_oversold": 28,
      "rsi_overbought": 72,
      "bb_std": 2.2,
      "stop_loss_pct": 0.085,
      "base_position_size": 650,
      "max_positions": 4
   }

What this means: - Composite Score 0.842: Weighted combination of all objectives (higher is better) - Training vs Validation: 30% degradation (acceptable - not overfit) - Win Rate 52.3%: Passes your 50% constraint ✅ - Drawdown 16.8%: Passes your 20% constraint ✅ - Parameters: Optimized values to use in production

2. JSON Report

Saved to reports/optimization_*.json:

{
  "optimization_date": "2025-10-30T10:30:00",
  "strategy": "MeanReversionOptimized",
  "symbols": ["AAPL", "MSFT", "NVDA"],
  "best_solution": {
    "parameters": {...},
    "sharpe_ratio": 2.15,
    "validation_sharpe": 1.51,
    "degradation_pct": 29.8,
    ...
  },
  "pareto_front": [
    /* 20 non-dominated solutions */
  ]
}

3. Pareto Frontier Plot

Three visualizations: 1. 3D scatter: Sharpe vs Return vs Drawdown 2. 2D heatmap: Sharpe vs Return (colored by drawdown) 3. Overfitting analysis: Training vs Validation Sharpe

---

🎯 Next Steps After Optimization

If Optimization Successful (50%+ win rate, passing constraints):

Step 1: Review Results

# View the JSON report
cat reports/mean_reversion_optimized.json | jq '.best_solution'

# View the Pareto plot
open reports/mean_reversion_pareto.png

Step 2: Deploy to Alpaca Paper Trading

Option A: Manual deployment (for now) 1. Copy optimized parameters from JSON 2. Update tools/strategy_tester.py with new parameters 3. Run paper trading:

uv run python3 tools/strategy_tester.py --strategy mean_reversion

Option B: Automated deployment (coming in Phase 2) - Optimizer will automatically deploy best strategy - Monitor for 1 week - Auto-promote to live if successful

Step 3: Monitor for 1 Week

Track these metrics: - Win rate (target: ≥50%) - Sharpe ratio (target: ≥1.0) - Max drawdown (target: ≤20%) - Compare to backtest predictions

Step 4: Deployment Decision

IF paper trading successful: - Deploy to Alpaca Live (small positions: $100-500) - Run for 1 week - Then deploy to Schwab (full positions: up to $1000)

IF paper trading fails: - Analyze failure modes - Re-optimize with adjusted constraints - Try longer training period (5 years instead of 2)

If Optimization Unsuccessful (< 50% win rate):

Possible issues: 1. Not enough trades: Use more symbols or longer period 2. Wrong strategy: Mean reversion may not work for those symbols 3. Market regime mismatch: Try including 2020 crash data 4. Too conservative constraints: Relax win rate to 45% for first iteration

Solutions:

# Try with more symbols and longer period
uv run python3 optimize_strategy.py \
  --symbols AAPL,MSFT,NVDA,GOOGL,META,AMZN,TSLA,AMD \
  --train-start 2020-01-01 \
  --train-end 2024-03-31 \
  --min-win-rate 0.45  # Slightly relaxed

# Or try different strategy (Phase 2: add momentum, breakout, etc.)

---

🛠️ Advanced Usage

Custom Constraints

# Very conservative (for Schwab main account)
uv run python3 optimize_strategy.py \
  --min-win-rate 0.55 \
  --max-drawdown 0.15 \
  --min-trades 20

# Aggressive (for Alpaca paper testing)
uv run python3 optimize_strategy.py \
  --min-win-rate 0.45 \
  --max-drawdown 0.30 \
  --min-trades 10

Custom Objective Weights

Currently hardcoded in GeneticOptimizer.__init__():

objective_weights = {
    'sharpe_ratio': 0.50,    # 50% weight
    'total_return': 0.30,    # 30% weight
    'max_drawdown': 0.20     # 20% weight
}

To change (edit tools/genetic_optimizer.py:147):

# More emphasis on returns (aggressive)
objective_weights = {
    'sharpe_ratio': 0.30,
    'total_return': 0.60,
    'max_drawdown': 0.10
}

# More emphasis on risk (conservative)
objective_weights = {
    'sharpe_ratio': 0.60,
    'total_return': 0.10,
    'max_drawdown': 0.30
}

Skip Validation (Faster, But Risky)

# Only use training data (no overfitting detection)
uv run python3 optimize_strategy.py \
  --no-validation \
  --train-start 2024-01-01 \
  --train-end 2024-10-30

Warning: This is faster but doesn't detect overfitting. Only use for quick experiments.

---

🔧 Troubleshooting

Issue 1: "No trades generated"

Symptoms: All strategies show 0 trades

Causes: - Training period too short (< 6 months) - Symbols too stable (no volatility) - Strategy too conservative (all entry conditions must be met)

Solutions: - Use longer period (2-5 years) - Add more volatile symbols (TSLA, AMD, small caps) - Relax entry conditions (lower rsi_oversold bound from 20 to 15)

Issue 2: "Alpaca API error"

Symptoms: ModuleNotFoundError: No module named 'alpaca'

Solution:

uv add alpaca-py

Issue 3: "All solutions fail constraints"

Symptoms: "⚠️ Does not pass all constraints"

Causes: - Constraints too strict - Not enough data - Wrong strategy for symbols

Solutions:

# Relax constraints slightly
--min-win-rate 0.45 \  # Instead of 0.50
--max-drawdown 0.25    # Instead of 0.20

Issue 4: "Optimization takes too long"

Symptoms: Runs for > 30 minutes

Causes: - Too many symbols - Too many generations/population - Very long training period

Solutions:

# Reduce to test settings
--population 10 \
--generations 10 \
--symbols AAPL,MSFT  # Just 2 symbols

Or: Run in background

nohup uv run python3 optimize_strategy.py ... > optimization.log 2>&1 &
tail -f optimization.log  # Monitor progress

---

📚 Files Created

File	Lines	Purpose
tools/optimizable_strategy.py	377	Base class for all strategies
tools/genetic_optimizer.py	426	NSGA-II algorithm implementation
tools/mean_reversion_optimized.py	376	Mean Reversion strategy (optimizable)
tools/optimize_strategy.py	265	Command-line interface
TOTAL	1,444 lines	Complete optimization framework

---

🎓 Key Concepts

1. Why Multi-Objective Optimization?

Single-objective (maximize returns only): - Result: 80% return but 60% drawdown (unusable) - Problem: Ignores risk

Multi-objective (Sharpe + Return - Drawdown): - Result: Pareto frontier with balanced solutions - Benefit: Trade-off between risk and reward - Your choice: Pick solution matching your risk tolerance

2. Why NSGA-II?

• Non-dominated Sorting Genetic Algorithm II
• Industry standard for multi-objective optimization
• Used by NexusTrade, quantitative hedge funds
• Finds Pareto-optimal solutions efficiently
• Better than grid search (4,500X faster)

3. Why Train/Validation Split?

Without validation: - Optimizer finds parameters perfect for training data - Deploys to live trading - Fails because overfit to past data

With validation: - Optimizer tunes on training (70%) - Tests on validation (30% unseen) - Detects overfitting before deployment - Saves money by catching bad strategies early

---

🔮 What's Next (Phase 2)

Not implemented yet, but planned:

1. Agentic Workflow (2-3 weeks)
2. Natural language: "Create mean reversion for tech stocks"
3. AI generates strategy code automatically
4. Runs optimization without human intervention

5. Auto-deploys to paper trading

6. More Strategies (add new strategies)

7. Momentum (SMA crossover)
8. Breakout (channel breakout)
9. RSI divergence

10. Options strategies

11. Auto-Deployment (Phase 2)

12. Best strategy automatically deployed to paper
13. Monitors for 1 week
14. Auto-promotes to live if successful

15. Email/SMS alerts

16. Multi-Strategy Portfolio (Phase 3)

17. Optimize allocation across strategies
18. Correlation analysis
19. Risk parity
20. Ensemble trading

---

📞 Support

Questions? Check: - This guide: docs/trading/GENETIC_OPTIMIZATION_GUIDE.md - Roadmap: docs/trading/AGENTIC_TRADING_ROADMAP_2025-10-30.md - Spec: specs/2025-10-30_genetic-strategy-optimizer.md

Issues? - Check Alpaca API keys in ~/Claude-Micro-Cap-Experiment/.env - Verify dependencies: uv add pymoo matplotlib seaborn alpaca-py - Run quick test first (10 pop, 10 gen) before full optimization

---

✅ Success Checklist

Before running full optimization, verify:

• [ ] Alpaca API credentials configured
• [ ] At least 2 years of historical data available (for chosen symbols)
• [ ] Clear understanding of your constraints (min win rate, max drawdown)
• [ ] Reviewed this guide and examples
• [ ] Ran quick test successfully (10 pop, 10 gen)

After optimization:

• [ ] Reviewed JSON report and Pareto plot
• [ ] Best solution passes all constraints
• [ ] Validation Sharpe within 30% of training (not overfit)
• [ ] Understand the optimized parameters
• [ ] Ready to deploy to paper trading

---

Last Updated: 2025-10-30 Status: ✅ Production Ready Time to Implement: 2 hours (from spec to working system) Next Action: Run full optimization or review roadmap for Phase 2