Milestone 12 - Learning Agent (Basic)

Summary

Implement basic learning agent to collect metrics and provide optimization recommendations.

Timeline

Weeks 25-26

Goal

Add metrics collection and basic optimization capabilities.

Deliverables

Learning Agent (Basic)

• ✅ Agent Versioning:
  • Reads active version from AgentVersionDeployment table during initialization
  • Includes agentVersion field in all outputs
  • Version information stored in AgentVersion table
• ✅ Agent Registration:
  • Registers agent type metadata via Agent Registry API (POST /api/registry/register/)
  • Registers instance via Agent Registry API (POST /api/register/) when spawned by orchestrator
  • Reports heartbeat via Agent Registry API (POST /api/heartbeat/) with current load and status
• ✅ Orchestrator-Triggered Execution:
  • Schedule created by admin via admin interface (stored in Schedule table)
  • Example: Daily at midnight schedule
  • Orchestrator invokes agent HTTP endpoint with job parameters
  • Operates independently - analyzes all metrics from previous day
• ✅ Metrics Collection:
  • Engagement metrics (opens, clicks, time spent)
  • User feedback collection (ratings, comments)
  • Section-level feedback analysis (analyzes feedback collected in Milestone 9 - Like/Dislike, Useful/Irrelevant per section)
  • Newsletter performance tracking
  • Agent performance metrics
  • Version performance metrics (tracks performance per agent version)
• ✅ Basic Analysis:
  • Engagement rate calculations
  • Trend detection
  • Top/low performing content identification
  • Section feedback analysis (analyzes feedback data collected in Milestone 9):
    • Identify sections with high/low feedback scores
    • Correlate feedback with content types and topics
    • Track feedback trends over time
  • Version Performance Analysis:
    • Compare performance across agent versions
    • Identify best-performing versions
    • Detect version-related performance regressions
    • Correlate version changes with outcome improvements
• ✅ Version Optimization:
  • Analyze version performance data from AgentExperiment table
  • Recommend version promotions based on performance metrics
  • Suggest version rollbacks if performance degrades
  • Track version impact on key metrics (engagement, quality, accuracy)
• ✅ Optimization Recommendations:
  • Basic config optimization suggestions
  • Simple A/B test analysis
  • Performance improvement recommendations
  • Updates agent configs in database (AgentConfig table - hot-reloadable)
  • Version-specific optimization recommendations
• ✅ Agent Data API Integration:
  • Writes learning results via POST /api/learning/ endpoint
  • Includes agentVersion in all outputs
  • Writes version performance analysis to database

Database Updates

• ✅ User feedback table (if not exists)
• ✅ Section feedback analytics views (aggregates section-level feedback)
• ✅ Agent metrics table (if not exists)
• ✅ Engagement tracking

Web Dashboard Updates

• ✅ Feedback collection UI (ratings, comments)
• ✅ Basic analytics dashboard
• ✅ Section feedback analytics visualization
• ✅ Admin metrics view

Delivery Agent (Update)

• ✅ Email tracking (open tracking, click tracking)
• ✅ Engagement metrics collection

Task Timeline

Limitations (Acceptable for This Milestone)

• Basic metrics only (no advanced analytics)
• Simple recommendations (no automated updates)
• No statistical significance testing for A/B tests
• Manual review of recommendations

Success Criteria

• ✅ Agent versioning is functional for Learning Agent (agent reads active version, includes in outputs)
• ✅ Agent registration is functional (agent registers type and instance, reports heartbeat)
• ✅ Collects engagement metrics accurately
• ✅ Tracks user feedback (including section-level feedback)
• ✅ Analyzes section feedback patterns
• ✅ Version performance analysis works correctly (compares versions, identifies best performers)
• ✅ Version optimization recommendations are generated
• ✅ Generates basic optimization recommendations
• ✅ A/B test results are tracked
• ✅ Metrics dashboard displays data correctly
• ✅ All outputs include agentVersion field
• ✅ Learning results written via Agent Data API

Next Steps

After this milestone, system collects metrics. Milestone 13 will focus on production hardening.