[FIXED] How does a worker know which queue to get tasks from?

Issue

Am working on a nodejs project where i need to implement task queueing. I have picked the bullMq with redis packages for this. Following the documentation here

  import { Queue, Worker } from 'bullmq'

// Create a new connection in every instance
const myQueue = new Queue('myqueue', { connection: {
  host: "myredis.taskforce.run",
  port: 32856
}});

const myWorker = new Worker('myworker', async (job)=>{}, { connection: {
  host: "myredis.taskforce.run",
  port: 32856
}});

After digging deeper in the documentation, i ended up asking some questions:

1. Do i need one worker and queue instance for the whole app? (I think this depends on the kind of tasks and operations you need) I need a task queue that process payments. Another task queue to work on marketing emails. I cant figure out how this would work if we had only one instance of worker and queue. It would but it requires setting up identifiers for every kind of operation and acting on each accordingly.

2. If i were to have many queues and worker instances,how would a worker know from which queue it should listen for tasks. From the code sample above, the worker seems to have be named myworker and the queue is called myqueue. How are these two connected? How does the worker know it should listen to jobs from that specific queue without colliding with other queues and workers?

Am quite new in tasks and queues, any help will be appreciated.

Solution

How does a worker know which queue to get tasks from?

The first argument to the Worker is supposed to be the name of the queue that you want it to pull messages from. The code you show is not doing that properly. But, the doc here explains that.

Do i need one worker and queue instance for the whole app? (I think this depends on the kind of tasks and operations you need) I need a task queue that process payments. Another task queue to work on marketing emails. I cant figure out how this would work if we had only one instance of worker and queue. It would but it requires setting up identifiers for every kind of operation and acting on each accordingly.

This really depends upon your design. You could have one queue that holds multiple types of things and one worker that processes whatever it finds in the queue.

Or, if want jobs to be processed concurrently, you can create more than one worker and those additional workers can even be in different processes.

If i were to have many queues and worker instances,how would a worker know from which queue it should listen for tasks. From the code sample above, the worker seems to have be named myworker and the queue is called myqueue. How are these two connected? How does the worker know it should listen to jobs from that specific queue without colliding with other queues and workers?

As explained above, the first argument to the Worker is supposed to be the name of the queue that you want it to pull messages from.

Answered By - jfriend00

Answer Checked By - Robin (PHPFixing Admin)