Sidequest.js Sidequest.js

Sidequest.js: Safe, Efficient, and Scalable Job Execution for Node.js

If you’re looking for a powerful yet simple way to handle background jobs in Node.js, check out Sidequest.js:

Modern Node.js applications often rely on task schedulers like node-cron, cron, and node-schedule to automate recurring jobs. These libraries are simple, effective, and widely used, but only up to a certain scale.

In distributed environments, where multiple instances of the same application are running, things get complicated. These traditional schedulers operate purely at the process level and lack visibility across nodes. If you deploy a task scheduled to run every 10 minutes on five different servers, it will run five times unless you implement your own distributed lock mechanism. That shifts the burden of coordination and infrastructure onto the developer, detracting from the core logic of the application.

Another major limitation is that these schedulers run tasks in the main thread. If you’re using something like an Express API and one of these tasks performs a blocking I/O operation, the whole process could be blocked until the task finishes. During that time, your API becomes unresponsive.

Sidequest.js was built to solve these problems head-on. It’s a distributed background job system that’s robust, scalable, and ready for production. Instead of requiring an external queueing service or complex setup, Sidequest.js leverages a database you’re likely already using, like PostgreSQL, MySQL and MongoDB, coordinate job execution across multiple nodes. Each task is guaranteed to run only once, even in a multi-instance setup.

Heavy jobs are executed in worker threads, isolating them from the main application thread. This ensures your API or application remains responsive, even when processing demanding background tasks.

More than just solving infrastructure pain points, Sidequest.js focuses on developer experience. Its API is intuitive and expressive: define a class that extends Job, call enqueue, and you’re done. Distributed execution, retries with exponential backoff, and job management are handled for you, with minimal configuration.

Example: Creating a Background Job with Sidequest.js

import { Job } from "sidequest";

// Define your job class
export class SendEmailJob extends Job {
  async run(to: string, subject: string, body: string) {
    // Simulate sending an email
    console.log(`Sending email to ${to}: ${subject}`);
    // await emailService.send(to, subject, body);
  }
}

// Enqueue the job
await Sidequest.build(SendEmailJob).enqueue("user@example.com", "Welcome!", "Thanks for signing up!");

That’s it. Sidequest.js handles everything else: persisting the job, distributing it across nodes, retrying on failure, and isolating the execution from your main app thread.

Why Sidequest.js instead of Airflow, RabbitMQ or SQS?

While there are many tools in the background job ecosystem, each comes with trade-offs, especially when used in modern Node.js environments.

Apache Airflow is powerful for complex DAG-based workflows, but it introduces a heavy operational footprint. It’s Python-centric, requires separate infrastructure, and is often overkill for simple or mid-tier background tasks that could easily live inside your application.

RabbitMQ and other message brokers like Kafka are great at decoupling producers and consumers, but they require maintaining a separate queueing infrastructure. They’re also optimized for message throughput, not for job durability, retries with backoff, or task deduplication. This adds complexity when all you need is reliable job execution.

Cloud-native solutions like AWS SQS seem appealing at first, but they introduce vendor lock-in and can behave unpredictably in distributed environments. SQS is eventually consistent, and handling distributed locking or deduplication correctly often requires glue code with services like DynamoDB or Lambda. The simplicity quickly fades when you need stronger delivery guarantees or full control over job behavior.

Sidequest.js takes a different approach.

It’s designed to run within your application stack, using the database you’re likely already running in production. That means:

By executing jobs in worker threads, isolating them from your main application logic, and managing distribution through your own database, Sidequest.js gives you the power of a job system like Airflow, with the simplicity of a local library, and without the operational cost of RabbitMQ or the opacity of SQS.

Ready to scale background tasks the right way?

Have you run into scaling issues with background jobs in Node.js? Try Sidequest.js and experience a cleaner, safer, and more scalable approach to job processing.