All posts

Why I Killed 11,000 Daily DB Polls with BullMQ

How we replaced ~11,000 daily MongoDB polls with BullMQ delayed jobs - and why 'check periodically' is often the wrong question.

nodejsbullmqmongodbbackendperformance

At ABSS, our Teams platform had a reminder system that quietly polled MongoDB every minute, checking every entity in the database for "does anything need a reminder right now?" It worked. It was also expensive, dumb, and getting worse as data grew.

The Problem

The reminder system had one job: notify users when something needed their attention - an upcoming meeting, a pending task, a deadline. The original implementation ran a cron-style job every minute that queried the entire collection, checked timestamps against the current time, and fired notifications for anything that matched.

Do the math on that: 1,440 checks a day, each one scanning across potentially thousands of documents that had nothing to do with reminders right now. We were sitting at roughly 11,000 unnecessary DB polling calls a day - and this number only goes up as more reminders get created. It's a pattern that looks fine in a demo and quietly becomes a scaling problem in production.

The deeper issue wasn't just load - it was architecture. We were asking "has anything changed?" on a fixed interval instead of asking "what already know needs to happen, and when?"

The Approach

The fix was to flip from polling to event-driven scheduling using BullMQ (Redis-backed job queues for Node.js).

Instead of scanning the whole collection every minute, we schedule a delayed job at the moment the entity is created - the exact moment we already know when the reminder should fire. If a task is created with a deadline three days out, we don't wait around checking every minute for three days. We schedule one job, delayed by exactly the right number of milliseconds, and BullMQ handles firing it at the right time.

A few things that mattered in the implementation:

  • Delay calculation lives at creation time, not at check time. The job carries its own "when," so there's nothing to repeatedly evaluate.
  • Job IDs tied to entity IDs, so if the underlying entity is updated or deleted before the reminder fires, we can find and cancel/reschedule the exact job instead of hunting for it.
  • Redis as the backing store for the queue means the reminder state survives process restarts - a concern that made the "just use setTimeout in memory" shortcut a non-starter.

The Result

Zero polling. The 11,000 daily DB reads for reminder-checking dropped to essentially nothing - we only touch the database when a job actually fires, or when a reminder needs to be rescheduled/cancelled. CPU and DB load from this feature became a non-issue instead of a slow-growing one.

The bigger lesson for me wasn't really about BullMQ specifically - it was noticing that "we need to check for X periodically" is often a sign that X was actually knowable in advance. If you can calculate the future event at creation time, you very likely don't need a poller at all - you need a scheduler.

What I'd do differently next time: build the "cancel/reschedule on entity update" path in from day one rather than retrofitting it. It's the part of event-driven systems that's easy to forget until an edge case (someone edits a deadline five minutes before the old one would've fired) forces you to build it anyway.