Milestone 2 - First Working Newsletter

Summary

Replace placeholder agents with minimal working versions to produce the first real newsletter.

Timeline

Weeks 5-6

Goal

Create the first end-to-end working newsletter using hardcoded/simple data sources, basic content generation, and database delivery.

Deliverables

Data Collection Agent (Minimal)

• ✅ Basic Data Collection Agent (apps/data-collection/) that:
  • Exposes HTTP endpoint (POST) for job execution
  • Accepts ticker as input via HTTP request
  • Returns hardcoded sample data (3-5 news articles)
  • Writes data to DataSource table via Agent Data API (POST /api/data-collection/)
  • Includes agentVersion field in all outputs (read from AgentVersionDeployment table)
  • Registers agent type metadata via Agent Registry API (POST /api/registry/register/)
  • Registers instance via Agent Registry API (POST /api/register/) when spawned
  • Reports heartbeat via Agent Registry API (POST /api/heartbeat/)
• ✅ Simple data structure:
  • News articles (title, url, snippet)
• ✅ No external API calls yet (uses mock data)
• ✅ DataSource table schema:
  • Basic fields: id, tickerId, type (e.g., 'news'), title, url, snippet, collectedAt, createdAt, agentVersion

Content Generation Agent (Minimal)

• ✅ Basic Content Generation Agent (apps/content-generation/) that:
  • Exposes HTTP endpoint (POST) for job execution
  • Accepts collected data (reads from DataSource table) and user preferences via HTTP request
  • Uses OpenAI API to generate simple newsletter:
    • Executive summary (2-3 sentences)
    • Top 3 news items with brief summaries
  • Writes newsletter to database via Agent Data API (POST /api/content-generation/):
    • Creates Newsletter record with userId and tickerId
    • Creates NewsletterContent record with generated HTML content
  • Includes agentVersion field in all outputs (read from AgentVersionDeployment table)
  • Registers agent type metadata via Agent Registry API (POST /api/registry/register/)
  • Registers instance via Agent Registry API (POST /api/register/) when spawned
  • Reports heartbeat via Agent Registry API (POST /api/heartbeat/)
• ✅ Simple prompt template (no personalization yet)
• ✅ Basic HTML email template (text-only, no styling)
• ✅ AI integration package (packages/ai/) with:
  • OpenAI client wrapper
  • Basic retry logic
  • Token usage tracking

Delivery Agent (Minimal)

• ✅ Basic Delivery Agent (apps/delivery/) that:
  • Exposes HTTP endpoint (POST) for job execution
  • Verifies newsletter is properly associated with user (ensures Newsletter.userId is set correctly)
  • Ensures newsletter is ready for dashboard display (content exists in NewsletterContent)
  • Writes delivery status via Agent Data API (POST /api/delivery/)
  • Includes agentVersion field in all outputs (read from AgentVersionDeployment table)
  • Registers agent type metadata via Agent Registry API (POST /api/registry/register/)
  • Registers instance via Agent Registry API (POST /api/register/) when spawned
  • Reports heartbeat via Agent Registry API (POST /api/heartbeat/)
  • No email sending yet (only dashboard delivery)
  • No status field needed yet (newsletters are immediately available once created)
• ✅ Newsletter accessible via dashboard (users can view newsletters filtered by their userId)

Hermes Orchestrator (Update)

• ✅ Newsletter pipeline triggered by orchestrator:
  • Pipeline defined in Pipeline table (admin-configurable)
  • Schedule created in Schedule table (admin-configurable timing: cron, interval, or on-demand)
  • User-defined schedules supported (per newsletter via UserTicker configuration)
  • Orchestrator invokes agent HTTP endpoints (not direct calls)
  • Agents authenticate using API keys from Agent Auth API
  • Checks data freshness before processing:
    • Checks DataSource.collectedAt timestamp for the ticker (data freshness check)
    • If data is stale (older than 4 hours), triggers immediate data collection via HTTP endpoint
    • Also checks if newsletter was already generated recently for this user-ticker to avoid duplicates
    • If data is fresh and no recent newsletter exists, proceeds with pipeline
  • Tracks job execution in AgentJobExecution table

Web Dashboard Updates

• ✅ Newsletter detail page (/newsletters/[id])
• ✅ Newsletter list on dashboard
• ✅ Basic newsletter display (HTML rendering)

Task Timeline

Limitations (Acceptable for This Milestone)

• Uses hardcoded/mock data (no real data collection from external APIs)
• No analysis stage (just summarizes news directly)
• No personalization (same content for all users, ignores user preferences)
• No email delivery (dashboard only - newsletters stored in database but not sent)
• No quality assurance stage (content generated directly without validation)
• Simple content structure only (executive summary + top 3 news items)
• DataSource table has minimal schema (only essential fields for mock data)

Success Criteria

• ✅ DataSource table created and migrations run successfully
• ✅ All agents expose HTTP endpoints and can be invoked by orchestrator
• ✅ All agents register themselves via Agent Registry API
• ✅ All agents include agentVersion in outputs
• ✅ All agents write outputs via Agent Data API (not directly to database)
• ✅ Orchestrator can invoke agent HTTP endpoints and execute pipeline
• ✅ Orchestrator runs and produces a newsletter end-to-end
• ✅ Newsletter contains real AI-generated content (not placeholders)
• ✅ Newsletter is saved to database with proper structure (Newsletter and NewsletterContent tables)
• ✅ Data flows correctly: Orchestrator → Data Collection (HTTP) → Agent Data API → DataSource → Orchestrator → Content Generation (HTTP) → Agent Data API → Newsletter/NewsletterContent → Orchestrator → Delivery (HTTP) → Agent Data API
• ✅ User can view newsletter in dashboard (filtered by userId)
• ✅ Content is readable and relevant (even if from mock data)
• ✅ System can generate newsletters for multiple users/tickers

Next Steps

After this milestone, we have a working newsletter system using mock data. Milestone 3 will replace mock data with real data collection.