Consistency and Reliability Concerns within the Message Broker System

1. Consistency in Message Broker

A message broker system provides data consistency, ensuring that all system components have a reliable and up-to-date view of the data.

1.1 Consistent message counts

Ensuring uniform message delivery and reception is a widespread concern. Network or system failures can disrupt this consistency, causing messages to be missed or duplicated. To tackle this problem, numerous message broker systems implement features like:

• Acknowledgments (ACKs): The system requires the recipient to send back a confirmation after receiving the message. This helps ensure that the message has been successfully received.
• Retries: The system provides mechanisms to automatically resend messages that have failed to be sent or have been lost.

1.2 Data consistency

Data consistency among multiple brokers in distributed systems can be difficult to achieve, especially in highly concurrent environments. Several techniques have been employed to solve this issue, including:

• Replication: Creating multiple copies of data across different brokers to ensure that if one broker fails, the data can still be accessed from the other brokers.
• Consensus Protocols: Employing consensus protocols such as Paxos or Raft to guarantee that all brokers agree on the system's state.

2. Reliability in Message Brokers

The reliability of a message broker system pertains to its ability to continue operating normally even in the face of failures. Factors influencing reliability include:

2.1. Self-Healing Capabilities

A reliable system should be able to self-heal after encountering a failure. Some ways to improve self-healing capabilities include:

• Failover Mechanisms: Automatically switching to a standby broker if the primary broker fails.
• Health Checks: Monitoring the broker's status to detect and address issues before they impact the system.

2.2. Ensuring Scalability

To accommodate growing demands without compromising reliability, the system must be designed with scalability in mind. This can involve:

• Horizontal Scaling: Adding more brokers to the system to distribute the load and improve performance.
• Load Balancing: Distributing messages evenly among brokers to avoid overloading.

3. Repeated message within the Message Broker

3.1 Context

I have a Google Pub/Sub system, configured as follows.

gcloud pubsub topics create $TOPIC_ID --project=$PROJECT_ID

gcloud alpha pubsub subscriptions create $SUBSCRIPTION_ID --topic=$TOPIC_ID --project=$PROJECT_ID 
        --dead-letter-topic=$DEAD_LETTER_TOPIC_ID 
        --dead-letter-topic-project=$PROJECT_ID 
        --max-delivery-attempts=5 
        --ack-deadline=90 
        --message-retention-duration="7d" 
        --min-retry-delay="10s" 
        --max-retry-delay="20s" 
        --expiration-period="never"

And a subscriber has a configuration as follows:

public  Subscriber.Builder configSubscriberBuilder(Subscriber.Builder builder){
	FlowControlSettings flowControlSettings =
            FlowControlSettings.newBuilder()
                    .setMaxOutstandingElementCount(500L)
                    .setMaxOutstandingRequestBytes(50L * 1024L * 1024L)
                    .build();

    int cores = Runtime.getRuntime().availableProcessors();
    int pullCount = 1;
    int cpuUsed = cores/pullCount;
    ExecutorProvider executorProvider =
            InstantiatingExecutorProvider.newBuilder().setExecutorThreadCount(cpuUsed).build();  
    builder.setFlowControlSettings(flowControlSettings);
    builder.setParallelPullCount(pullCount);
    builder.setMaxAckExtensionPeriod(Duration.ofSeconds(30));
    builder.setExecutorProvider(executorProvider);
    
    return builder;
}

The above parameter means:

• Use the number of threads equal to the number of physical threads to process messages.
• The number of messages pulled will satisfy one of the following conditions:
• Condition A: A maximum of 500 messages will be pulled.
• Condition B: The total size of all pulled messages must be less than 50MB.
• ack-deadline=90 means the maximum allowed time for an ACK message is 90 seconds.

Let me elaborate more on the Pub/Sub system in Google. Once a message is sent to a Topic, it will be fetched and stored by the Subscriptions. When an application connects to a Subscription, it will pull messages based on the configuration of the Subscription and Subscriber and hold those messages in a Message Container for gradual processing. Once they are fully processed, it will continue to pull more messages and do the same.

The processing inside service B will happen as follows:

For instance, the messages undergo processing through calls to services D, E, and F. Once this processing is complete, service B sends an acknowledgment (ACK) to the subscription to indicate that the message has been successfully processed.

This means that the messages must be completed before the ACK timeout expires.

3.2 Reason

As mentioned above, if all messages cannot be processed within 90 seconds, then

The remaining 150 messages will be retried on service B. Because they've exceeded their timeout, they cannot be acknowledged and will be returned to the subscription to be processed again, at least once.

3.3 Some ways to handle

I propose the following solutions:

• To optimize resource utilization, we can perform load testing on the functions responsible for processing messages. By determining the maximum throughput of these functions, we can configure a subscriber to pull a suitable number of messages. This will prevent overwhelming the system with more messages than it can handle.
• To guarantee the system's idempotency, it is crucial to implement a mechanism for checking the existence of records within the functions called by services D, E, and F. This check will ensure that an action is not executed multiple times for the same request, even if the request is repeated.

4. Conclusion

Consistency and reliability are paramount in guaranteeing the performance and steadfastness of a message broker system. By comprehending and resolving consistency and reliability concerns, you can construct robust and dependable distributed systems that align with the requisites of contemporary applications. Let's implement best practices and techniques to ensure that your system consistently functions efficiently and steadily.

Should you have any inquiries, kindly leave a comment beneath the chat box for further discussion.

Thank you for your time, and we look forward to connecting with you again.