SQS Message Acknowledgment Strategy¶

The Problem: Data Loss Risk¶

Original Implementation (Flawed)¶

# ❌ PROBLEMATIC: Automatic acknowledgment
def lambda_handler(event, context):
    for record in event["Records"]:
        # Process message
        process_event(record["body"])
        # Lambda completes → SQS automatically acknowledges
        # If Lambda dies here, message is lost!

The Risk¶

Lambda receives SQS message (visibility timeout starts)
Message processed successfully
Lambda dies before completion (crash, timeout, etc.)
SQS automatically acknowledges the message
Message is lost forever - no retry possible

The Solution: Explicit Message Acknowledgment¶

Corrected Implementation¶

# ✅ CORRECT: Explicit acknowledgment
def lambda_handler(event, context):
    message_infos = []

    # Parse and track all messages
    for record in event["Records"]:
        message_info = SQSMessageInfo(
            receipt_handle=record["receiptHandle"],
            message_id=record["messageId"],
            event=parse_event(record["body"])
        )
        message_infos.append(message_info)

    # Process all events
    results = process_events_batch(message_infos)

    # Only acknowledge successfully processed messages
    acknowledge_messages(results["acknowledged_messages"])

Message Flow Architecture¶

1. Message Ingestion¶

SQS Queue → Lambda Trigger → Parse Messages → Track Receipt Handles

2. Processing Phase¶

Process Events → Update Status → Mark for Acknowledgment

3. Acknowledgment Phase¶

Successful Events → Explicit Delete → SQS Removes Message
Failed Events → Keep in Queue → Retry on Next Invocation

Key Components¶

SQSMessageInfo Class¶

@dataclass
class SQSMessageInfo:
    receipt_handle: str      # SQS receipt handle for deletion
    message_id: str          # SQS message ID for tracking
    event: Event            # Parsed event data
    processed: bool = False  # Processing status flag

Acknowledgment Logic¶

def acknowledge_messages(self, message_infos: List[SQSMessageInfo]) -> None:
    """Acknowledge successfully processed SQS messages."""
    for message_info in message_infos:
        if message_info.processed:
            try:
                self.sqs_client.delete_message(
                    queue_url=self.settings.main_queue_url,
                    receipt_handle=message_info.receipt_handle
                )
                self.logger.info(f"Acknowledged message {message_info.message_id}")
            except Exception as e:
                self.logger.error(f"Failed to acknowledge message: {str(e)}")
                # Don't raise - we don't want to fail the entire batch

Failure Scenarios and Handling¶

Scenario 1: Lambda Dies During Processing¶

1. Lambda receives 10 messages
2. Processes 5 successfully, 5 fail
3. Lambda crashes before acknowledgment
4. All 10 messages return to queue (visibility timeout)
5. Next Lambda invocation retries all 10
6. Idempotency prevents duplicate processing of the 5 successful ones

Scenario 2: Lambda Dies After Processing, Before Acknowledgment¶

1. Lambda processes all 10 messages successfully
2. Lambda crashes before calling acknowledge_messages()
3. All 10 messages return to queue
4. Next invocation: idempotency check skips all 10 (already processed)
5. Messages are acknowledged and removed from queue

Scenario 3: Acknowledgment Failure¶

1. Lambda processes all messages successfully
2. DynamoDB acknowledgment fails for some messages
3. Failed acknowledgments remain in queue
4. Next invocation: idempotency prevents reprocessing
5. Messages are acknowledged and removed

Idempotency + Acknowledgment = Data Safety¶

The Safety Net¶

# Step 1: Check if already processed
if self.check_idempotency(event.event_id, idempotency_key):
    message_info.processed = True  # Mark for acknowledgment
    continue  # Skip processing

# Step 2: Process if not already done
success = self.process_single_event_resilient(event, idempotency_key)
if success:
    message_info.processed = True  # Mark for acknowledgment

# Step 3: Acknowledge only processed messages
acknowledge_messages([mi for mi in message_infos if mi.processed])

Benefits¶

No Data Loss: Failed messages stay in queue
No Duplicates: Idempotency prevents double processing
Efficient Retries: Only failed messages are retried
Audit Trail: Complete processing history

Configuration Considerations¶

SQS Visibility Timeout¶

# Set visibility timeout longer than Lambda timeout
visibility_timeout = lambda_timeout + buffer_time
# Example: Lambda timeout 5 minutes, visibility timeout 6 minutes

Batch Size Limits¶

# Process messages in batches to avoid timeout
max_batch_size = 1000  # Adjust based on processing time

Error Handling¶

# Don't fail entire batch for acknowledgment failures
try:
    acknowledge_messages(processed_messages)
except Exception as e:
    logger.error(f"Acknowledgment failed: {str(e)}")
    # Messages will be retried on next invocation

Monitoring and Alerting¶

Key Metrics¶

Messages Acknowledged: Count of successfully acknowledged messages
Acknowledgment Failures: Count of failed acknowledgments
Processing Success Rate: Percentage of messages processed successfully
Queue Depth: Number of messages waiting for processing

Alerts¶

High Queue Depth: Too many messages waiting
Low Success Rate: Too many processing failures
Acknowledgment Failures: Issues with message removal

Testing Strategy¶

Unit Tests¶

def test_message_acknowledgment():
    # Test that only processed messages are acknowledged
    # Test acknowledgment failure handling
    # Test idempotency with acknowledgment

Integration Tests¶

def test_end_to_end_message_flow():
    # Test complete message processing and acknowledgment
    # Test failure scenarios and recovery
    # Test batch processing with mixed success/failure

Chaos Engineering¶

def test_lambda_crash_scenarios():
    # Simulate Lambda crashes during processing
    # Verify no data loss
    # Verify proper retry behavior

Best Practices¶

1. Always Use Explicit Acknowledgment¶

Never rely on automatic acknowledgment
Track receipt handles for all messages
Acknowledge only after successful processing

2. Implement Idempotency¶

Generate unique keys for each event
Check for existing processing before starting
Handle duplicate events gracefully

3. Handle Acknowledgment Failures¶

Don't fail the entire batch for ack failures
Log failures for monitoring
Let failed acknowledgments retry naturally

4. Monitor and Alert¶

Track acknowledgment success rates
Alert on high failure rates
Monitor queue depth and processing latency

5. Test Failure Scenarios¶

Test Lambda crashes during processing
Test acknowledgment failures
Verify no data loss in all scenarios

Summary¶

The explicit message acknowledgment strategy ensures:

No Data Loss: Failed messages remain in queue for retry
Exactly-Once Processing: Idempotency prevents duplicates
Efficient Processing: Only failed messages are retried
Reliable Recovery: System recovers from any failure scenario
Complete Audit Trail: Full processing history maintained

This approach provides at-least-once delivery with exactly-once processing, which is the gold standard for financial transaction systems.