README
A simple high-performance Redis message queue for Node.js.
RedisSMQ is a Node.js library for queuing messages (aka jobs) and processing them asynchronously with consumers. Backed by Redis, it allows scaling up your application with ease of use.
Features
- High-performance message processing
- Multi-Queue Producers & Multi-Queue Consumers: Offering very flexible models which make RedisSMQ an ideal message broker for your microservices.
- Supporting both at-least-once/at-most-once delivery: In case of failures, while delivering or processing a message, RedisSMQ can guaranty that the message will be not lost and redelivered again. When configured to do so, RedisSMQ can also ensure that the message is delivered at-most-once.
- Message Expiration: A message will not be delivered if it has been in a queue for longer than a given amount of time, called TTL (time-to-live).
- Message Consumption Timeout: Timeout for consuming messages.
- Queue Rate Limiting: Allowing you to control the rate at which the messages are consumed from a given queue.
- Scheduling Messages: Messages can be configured to be delayed, delivered for N times with an optional period between deliveries, and to be scheduled using CRON expressions.
- Reliable Priority Queues: Supports priority messaging.
- HTTP API: an HTTP interface is provided to interact with the MQ.
- Web UI: RedisSMQ can be managed also from your web browser.
- Logging: Comes with a built-in JSON logger. But you can also use your own logger instance.
- Configurable: Many options and features can be configured.
- Both redis & ioredis clients are supported: RedisSMQ can be configured to use either
redisorioredisto connect to Redis server. - Rigorously tested: With 100+ tests and code coverage no less than 80%.
- Highly optimized: Strongly-typed and implemented using pure callbacks, with small memory footprint and no memory leaks. See callbacks vs promises vs async/await benchmarks.
RedisSMQ Use Case: Multi-Queue Producers & Multi-Queue Consumers
Table of Content
What's new?
2022.02.08
- :rocket: Release v6 is finally ready. This release includes new features such as multi-queue consumers and multi-queue producers, message rate time series, complete integration with the Web UI, as well as many improvements and bug fixes. If you are upgrading your installation, take a look at the migration guide before proceeding.
See CHANGELOG for more details.
Installation
npm install redis-smq --save
Considerations:
- RedisSMQ is targeted to be used in production environments. Therefore, only active LTS and maintenance LTS Node.js releases (v12, v14, and v16) are supported. The latest stable Node.js version is recommended.
- Minimal Redis server version is 2.6.12. The latest stable Redis version is recommended.
Configuration
See Configuration for more details.
Usage
Basics
RedisSMQ provides 3 classes in order to work with the message queue: Message, Producer, and Consumer.
Message Class
Message class is responsible for creating messages that may be published.
A message can carry your application data, sometimes referred to as message payload, which may be delivered to a consumer to be processed asynchronously.
The message payload can be of any valid JSON data type. It may be a simple text message like Hello world or a complex data type like {hello: 'world'}.
const { Message } = require('redis-smq');
const message = new Message();
message
.setBody({hello: 'world'})
.setTTL(3600000) // in millis
.setQueue('test_queue');
let messageTTL = message.getTTL();
The Message class provides many methods for setting up different message parameters such as message body, message priority, message TTL, etc.
See Message Reference for more details.
Producer Class
Producer class allows you to publish a message to a queue.
You can use a single Producer instance to produce messages, including messages with priority, to multiple queues.
Before publishing a message do not forget to set the destination queue of the message using the setQueue() method, otherwise an error will be returned.
// filename: ./examples/javascript/producer.js
'use strict';
const {Message, Producer} = require('redis-smq');
const message = new Message();
message
.setBody({hello: 'world'})
.setTTL(3600000) // in millis
.setQueue('test_queue');
message.getId() // null
const producer = new Producer();
producer.produce(message, (err) => {
if (err) console.log(err);
else {
const msgId = message.getId(); // string
console.log('Successfully produced. Message ID is ', msgId);
}
});
See Producer Reference for more details.
Consumer Class
Consumer class can be used to receive and consume messages from a queue.
Similarly to a Producer instance, a Consumer instance can consume messages from multiple queues.
For consuming messages from a queue, the Consumer class provides the consume() method which allows you to register a message handler.
A message handler is a function which get called once a message is received.
Message handlers can be registered at any time, before or after you have started your consumer.
A consumer can be started using the run() method.
To shut down and remove a given message handler from your consumer, use the cancel() method.
To shut down completely your consumer and tear down all message handlers, use the shutdown() method.
// filename: ./examples/javascript/consumer.js
'use strict';
const { Consumer } = require('redis-smq');
const consumer = new Consumer();
const messageHandler = (msg, cb) => {
const payload = msg.getBody();
console.log('Message payload', payload);
cb(); // acknowledging the message
};
// The second parameter is for enabling priority queuing for the message handler
// If you are willing to consume messages with priority, do not forget to set a priority for your messages
// See Reliable Priority Queues for more details
consumer.consume('test_queue', false, messageHandler, (err, isRunning) => {
if (err) console.error(err);
// the message handler will be started only if the consumer is running
else console.log(`Message handler has been registered. Running status: ${isRunning}`); // isRunning === false
});
const anotherMessageHandler = (msg, cb) => {
const payload = msg.getBody();
// ...
cb();
};
consumer.consume('another_queue', false, anotherMessageHandler, (err, isRunning) => {
if (err) console.error(err);
});
consumer.run();
Once a message is received, to acknowledge it, you invoke the callback function without arguments, as shown in the example above.
Message acknowledgment informs the MQ that a message has been successfully consumed.
If an error occurred while processing a message, you can unacknowledge the message by passing the error to the callback function.
By default, unacknowledged messages are re-queued and delivered again unless message retry threshold is exceeded. Then the messages are moved to a dead-letter queue (DLQ).
A dead-letter queue is a system generated queue that holds all messages that couldn't be processed or can not be delivered to consumers.
By default, RedisSMQ does not store acknowledged and dead-lettered messages for saving disk and memory space, and also to increase message processing performance. If you need such feature, you can enable it from your configuration object.
See Consumer Reference for more details.
Advanced Topics
RedisSMQ Architecture
Performance
See Performance for more details.
Contributing
So you are interested in contributing to this project? Please see CONTRIBUTING.md.