[FIXED] AWS-EC2 Redis-server RDB snapshot write error

Issue

I have a web application running on Laravel5.2 framework, with session driver set to redis with following AWS setup.

• Instance-1: Running web application, with Redis configurations in .env file as follow

  1. Redis-host: aws-private-ip-of-instance-2
  2. Redis-password: NULL
  3. Redis-port: 6379

• Instance-2: Redis-server running with following configuration

  1. Bind aws-private-ip-of-instance-2 and 127.0.0.1
  2. Working directory /var/lib/redis with 775 permission, and ower-group is redis.
  3. RDB snapshot name dump.rdb with 660 permission, and ower-group is redis.

NOTE: In AWS inbound rule for port 6379 is configured for Instance-2.

Everything works fine, until redis tries to write the data on the RDB file. Following error shows on front-end.

MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.

While in the logs of Redis server i got following data.

4873:M 23 Sep 10:08:15.028 * 1 changes in 900 seconds. Saving...
4873:M 23 Sep 10:08:15.028 * Background saving started by pid 7392
7392:C 23 Sep 10:08:15.028 # Failed opening .rdb for saving: Read-only file system
4873:M 23 Sep 10:08:15.128 # Background saving error

Things I have tried

• Add vm.overcommit_memory = 1 to /etc/sysctl.conf, as suggested in Redis-administraition-blog
• Change path to dump.rdb file to tmp folder and change permissions to 777.

Solution

Well this is very embarrassing to post answer of own question, which was a really stupid mistake. But hope new folks here learns from my mistake too.

• So first thing I have done is enable detail logs for redis-server in /etc/redis/redis.conf file by changing log_level option to debug.

• Observe the logs and understand that my redis port 6379 was open for everyone on internet.

So from logs I observe that someone else's server is spoofing into my redis server and making it slave of it. And as my redis server is configure in a way that slave is read-only, when i try to access my redis-server it throw error of read-only.

• After applying the fire-wall for redis server port, I have not encounter this issue anymore.

Answered By - Jay Pandya