[FIXED] How to guarantee to process Kinesis event stream serially when using paralleization factor?

Issue

Kinesis stream has only 1 shard and when creating Lambda, concurrent batches per shard for Kinesis stream source has been set as 10. When there is a spike in stream data, it will increase the concurrencies to 10. That means we will have 10 lambdas working in parallel. My question in this case is, how we can guarantee to process event stream serailly? It seems to me that it is impossible to do that because we can't control concurrencies. Can anyone have an idea for this? I can't get my head round.

Solution

AWS Lambda supports concurrent batch processing per shard and serial event processing, as long as all events in the Kinesis stream have the same partition key.

From AWS documentation:

You can also increase concurrency by processing multiple batches from each shard in parallel. Lambda can process up to 10 batches in each shard simultaneously. If you increase the number of concurrent batches per shard, Lambda still ensures in-order processing at the partition-key level.

References:

1. Using AWS Lambda with Amazon Kinesis (AWS)
2. Partition Key (Amazon Kinesis Data Streams Terminology and Concepts)

Answered By - Andrew Nguonly

Answer Checked By - Mildred Charles (PHPFixing Admin)

Read More

[FIXED] How to verify that Amazon Kinesis Python client is working

Issue

I'm trying to build an Amazon Kinesis Python consumer using the KCL library for Python (https://github.com/awslabs/amazon-kinesis-client-python). I started by checking the sample code. I was able to run both producer and consumer scripts parts of the sample code, but I'm unable to verify if the data from my kinesis stream (with one shard) is being pushed to the sample Python consumer script, sample_kclpy_app.py.

I used the amazon_kclpy_helper.py to generate the Java command that would call the Python script through the sample.properties file. I ran the Java command, and I can see from the terminal output that the data from the Kinesis stream is being read. I added a print statement in the process_record function of the Python consumer script to check if the data was being pushed to it. But it doesn't show up in the terminal output.

I also tried using logging to generate STDOUT messages, as well as write to a file. I also added assert 0 line in the Python code to force fail the script and see an exception would appear in the log output. Then, I purposely added a syntax error in the Python code. However, it seems that all of these were not detected by the Java MultiLangDaemon, which just kept going and churning out INFO log messages.

What could be the problem? And is there a better way to check if the data is indeed being sent to the process_record function of the Python consumer script?

Solution

After numerous trial and error, I figured out that the Python script assigned to executableName should be visible from the Java KCL process that is calling it.

The quick fix is to set the PATH environmental variable to the directory of the script and make the script an executable file (e.g. chmod +x)

Answered By - Carlos

Answer Checked By - Gilberto Lyons (PHPFixing Admin)

Read More