SMQ/S Broker Interactions
The SMQ/S Broker uses a named pipe on the local filesystem to rapidly buffer messages for relay to an SMQ/S instance.
The SMQ/S Broker uses an SQLite3 file-backed database to store a local queue consisting of message references requested by the Synapse Consumer.
The SMQ/S Broker uses the same SQLite3 database to store a table representing the pool of participating SMQ/S instances, their addresses, and their availability ratings.
Tenta Producer
Receives messages from the Tenta Producer as HTTP POST requests whose payloads consist of a JSON object, with a String field for the topic, and an array field whose elements represent messages for that topic to queue. After receipt, the SMQ/S thread processing the POST request breaks this into individual message objects and buffers each message object in a named pipe demarcated by newline.
For the duration of the SMQ/S Broker service’s lifecycle, a user-configured number of SMQ/S Broker threads pull message objects from the named pipe and relay these message objects to the SMQ/S instance in the pool with the highest availability rating. After this is done, the thread then forms a JSON object with fields representing the target SMQ/S instance ID, the message index, and topic. That object is a message reference. The message reference is added persistently to a local queue Failures are logged and raw message objects that failed to process are dumped to a separate file.
SMQ/S
Receives an HTTP POST request from SMQ/S to join the SMQ/S pool, with a payload of a JSON object consisting of a single field representing that SMQ/S instance’s instance ID. The SMQ/S Broker thread handling the request will then update a local table that contains rows representing SMQ/S instances, their network addresses, and their availability ratings.
A background thread will periodically poll the pool of SMQ/S instances and update availability ratings at user-defined intervals.
Receives an HTTP POST request from SMQ/S to drop reference messages with a payload of a JSON object consisting of an array of strings representing message indices.
SMQ/S Broker
Receives an HTTP POST request from other SMQ/S Brokers to join the SMQ/S Broker pool, with a payload of a JSON object consisting of a single field representing that SMQ/S Broker’s instance ID. The SMQ/S Broker
Receives an HTTP GET request from other SMQ/S Brokers as part of a syncing mechanism. It will respond with a queue dump of the reference messages as a JSON object consisting of an array of strings representing reference messages.
Periodically makes an HTTP GET request to all other SMQ/S Brokers in the SMQ/S Broker pool as part of a syncing mechanism. As it requests all the reference messages in each pool member’s local queue, it….
