Productivity Market Intelligence

People regularly discuss their productivity problems, current tools and buying decisions in public online communities. Those conversations can improve product decisions and reveal time-sensitive opportunities, but only if the relevant ones can be found without spending the day scanning feeds.

Useful product feedback and sales conversations were spread across more than 30 Reddit communities. Checking them by hand took too long. Urgent posts, such as a person asking for a recommendation, could become stale within hours.

I defined the source and keyword strategy, built the collection and ranking pipeline, designed the structured language-model output, created the immediate-alert and digest formats and deployed the unattended job on a Linux server.

I also added the unglamorous parts that make scheduled work dependable: deduplication, short-lived caches, file locks, retries, validation, structured logs and failure alerts.

1. Poll more than 30 subreddits through the Reddit API. Focused communities are read broadly. General communities are filtered using competitor and category terms.
2. Load each candidate post and its full public comment tree so the ranking uses the surrounding discussion.
3. Remove posts already processed and discard cached items after 24 hours.
4. Ask a language model for a strict JSON result containing High, Medium or Low relevance, an urgency flag and one short reason. Validate the returned fields before use.
5. Combine that result with upvotes and comment count to produce a clear ranking score.
6. Send urgent recommendation requests as immediate alerts. Cache the remaining items for digest emails at 9 AM and 9 PM UTC.
7. Build a readable HTML email with source links, the reason each item was selected and response ideas for manual review.
8. Protect unattended runs with file locks, exponential retry, structured logs and a failure email if the job stops.
9. Deploy through a repeatable shell script that sets up an Ubuntu server, Python environment, restricted credentials and cron schedule.

A mention of a productivity term does not necessarily contain a useful signal. The system therefore reads the public comment tree around each candidate and combines a validated relevance classification with engagement data.

Urgency and importance are handled separately. Recommendation requests can trigger an immediate alert, while routine research is deliberately batched into two predictable reviews so the monitor does not create another source of distraction.

Urgent findings arrive as individual alerts while they are still timely. Everything else is grouped into a concise HTML digest containing the source, the reason it was selected and draft talking points for manual review.

Scanning and sorting more than 30 communities would take five to seven hours each week. The two daily digests reduce that to about 20 minutes of review a day, saving roughly four hours each week.

Urgent recommendation requests arrive within one polling cycle. Routine findings are grouped into two predictable reviews instead of interrupting the day.

Collecting every possible mention would have produced a larger but less useful inbox. The combination of community-specific search rules, contextual ranking, deduplication and digest timing mattered more than raw collection volume.

Strict structured output also made the language-model step operationally useful. A small validated schema was easier to rank, log and recover than an open-ended narrative response.

This project shows how I turn an interrupt-driven research task into a reliable operating rhythm: collect broadly, rank transparently, escalate only what is urgent and keep a person responsible for the final response.