[FIXED] How to verify that Amazon Kinesis Python client is working

Issue

I'm trying to build an Amazon Kinesis Python consumer using the KCL library for Python (https://github.com/awslabs/amazon-kinesis-client-python). I started by checking the sample code. I was able to run both producer and consumer scripts parts of the sample code, but I'm unable to verify if the data from my kinesis stream (with one shard) is being pushed to the sample Python consumer script, sample_kclpy_app.py.

I used the amazon_kclpy_helper.py to generate the Java command that would call the Python script through the sample.properties file. I ran the Java command, and I can see from the terminal output that the data from the Kinesis stream is being read. I added a print statement in the process_record function of the Python consumer script to check if the data was being pushed to it. But it doesn't show up in the terminal output.

I also tried using logging to generate STDOUT messages, as well as write to a file. I also added assert 0 line in the Python code to force fail the script and see an exception would appear in the log output. Then, I purposely added a syntax error in the Python code. However, it seems that all of these were not detected by the Java MultiLangDaemon, which just kept going and churning out INFO log messages.

What could be the problem? And is there a better way to check if the data is indeed being sent to the process_record function of the Python consumer script?

Solution

After numerous trial and error, I figured out that the Python script assigned to executableName should be visible from the Java KCL process that is calling it.

The quick fix is to set the PATH environmental variable to the directory of the script and make the script an executable file (e.g. chmod +x)

Answered By - Carlos

Answer Checked By - Gilberto Lyons (PHPFixing Admin)