Daily Reddit Startup Signal Engine

An end-to-end AI engineering portfolio project that turns daily Reddit discussion into startup ideas and a newsletter draft.

The app runs with Python 3.11, FastAPI, Uvicorn, and SQLite. Live Reddit, OpenAI, Gmail draft, SMTP, and Resend test-email integrations are optional and activated through environment variables. Without credentials, the app runs against realistic sample Reddit data and saves a local email draft.

What it does

• Ingests Reddit posts from startup, SaaS, business, AI, and productivity communities.
• Cleans, deduplicates, and clusters discussions into market signals.
• Extracts pain points and generates startup ideas.
• Scores ideas by evidence volume, engagement, pain intensity, feasibility, and novelty.
• Renders a Markdown and HTML newsletter.
• Saves a Gmail draft when authorized, otherwise writes a local .eml draft.
• Provides a FastAPI-backed review dashboard for running the workflow, editing newsletter drafts, managing test subscribers, and inspecting metrics.

Quick start

python3 -m signal_engine run --sample --db work/startup_signal.db
python3 -m signal_engine serve --db work/startup_signal.db --port 8000

Open http://127.0.0.1:8000 after starting the server.

For ASGI hosting, use main:app. The root main.py loads .env from the project directory and resolves relative APP_DB_PATH / OUTPUT_DIR against that directory.

On PythonAnywhere Free, external API calls must use the platform proxy. Set this in .env:

OUTBOUND_HTTP_PROXY=http://proxy.server:3128

Run tests:

python3 -m unittest discover -s tests

Live mode

Copy .env.example to .env, add credentials, then run:

python3 -m signal_engine run --live --db work/startup_signal.db

The app automatically reads a local .env file when present. Keep real secrets in .env, not .env.example.

Live Apify ingestion requires:

• REDDIT_SOURCE=apify
• APIFY_TOKEN
• APIFY_ACTOR_ID=harshmaur~reddit-scraper
• MAX_REDDIT_ITEMS=20
• optional APIFY_SEARCH_URLS as |-separated Reddit search URLs
• optional APIFY_SEARCH_TERMS as |-separated fallback pain-signal queries

The default Apify actor is harshmaur~reddit-scraper, run against scoped Reddit search URLs for pain-signal queries instead of noisy subreddit hot feeds. The adapter sends startUrls, disables comment crawling, sets maxPostsCount=MAX_REDDIT_ITEMS, and caps stored items at MAX_REDDIT_ITEMS. If APIFY_SEARCH_URLS is disabled in code, the same adapter can fall back to Harshmaur's searchTerms input.

Live official Reddit API ingestion requires:

• REDDIT_CLIENT_ID
• REDDIT_CLIENT_SECRET
• REDDIT_USER_AGENT

OpenAI generation is optional. If OPENAI_API_KEY is not set, the project uses deterministic heuristic generation.

Gmail draft creation is optional. If GMAIL_ACCESS_TOKEN and NEWSLETTER_TO are not set, the project writes an .eml draft under outputs/drafts.

SMTP test sending is optional and intentionally test-only in this version:

• EMAIL_PROVIDER=smtp
• SMTP_HOST
• SMTP_PORT=587
• SMTP_USERNAME
• SMTP_PASSWORD
• SMTP_FROM_EMAIL
• optional SMTP_FROM_NAME
• optional EMAIL_REPLY_TO
• EMAIL_SEND_MODE=test
• EMAIL_DAILY_LIMIT=20

For Gmail, enable 2-Step Verification, create a Google App Password, then use smtp.gmail.com, port 587, your Gmail address as SMTP_USERNAME, the app password as SMTP_PASSWORD, and your Gmail address as SMTP_FROM_EMAIL.

Resend test sending remains available if you prefer it:

• EMAIL_PROVIDER=resend
• RESEND_API_KEY
• RESEND_FROM_EMAIL
• optional EMAIL_REPLY_TO or RESEND_REPLY_TO
• EMAIL_SEND_MODE=test
• EMAIL_DAILY_LIMIT=20

The dashboard only sends to active subscribers marked as test recipients. If the selected email provider settings are missing, the send-test API returns a readiness error instead of silently simulating delivery.

API

• GET /api/health
• GET /api/latest
• GET /api/runs
• POST /api/run
• GET /api/metrics
• GET /api/subscribers
• POST /api/subscribers
• PATCH /api/subscribers/{subscriber_id}
• DELETE /api/subscribers/{subscriber_id}
• PATCH /api/newsletter/{run_id}
• POST /api/newsletter/{run_id}/send-test

Architecture

flowchart LR
  A["CLI / daily scheduler"] --> B["Reddit ingestion"]
  B --> C["Filter + dedupe"]
  C --> D["Topic clustering"]
  D --> E["Idea generation"]
  E --> F["Scoring"]
  F --> G["Newsletter renderer"]
  G --> H["Gmail or local draft"]
  G --> I["Editable newsletter issue"]
  I --> J["FastAPI dashboard API"]
  J --> K["SQLite subscribers + send logs"]
  J --> L["SMTP or Resend test email"]

Loading

Core modules:

• signal_engine.ingestion: sample and live Reddit clients.
• signal_engine.clustering: lightweight text clustering.
• signal_engine.idea_generation: heuristic and optional OpenAI structured generation.
• signal_engine.scoring: deterministic idea ranking.
• signal_engine.newsletter: Markdown and HTML newsletter rendering.
• signal_engine.email_draft: Gmail draft API and local .eml fallback.
• signal_engine.smtp_email: SMTP test-email sending via Python smtplib.
• signal_engine.resend_email: Optional Resend test-email sending.
• signal_engine.server: FastAPI dashboard and API.

Resume framing

Built a production-shaped AI workflow that mines social discussion, clusters noisy market signals, generates structured startup concepts, ranks them with deterministic and model-assisted scoring, and publishes a human-reviewed newsletter draft through Gmail.

Highlights:

• API integration under external rate-limit and auth constraints.
• Structured LLM outputs with schema validation and fallback generation.
• Evidence-backed generation to reduce hallucinated ideas.
• Human-in-the-loop review UI.
• Integration and unit tests with fixture-based data.