[FIXED] Why is appengine memcache not storing my data for the requested period of time?

Issue

I have my employees stored in appengine ndb and I'm running a cron job via the taskque to generate a list of dictionaries containing the email address of each employee. The resulting list looks something like this:

[{"text":"john@mycompany.com"},{"text":"mary@mycompany.com"},{"text":"paul@mycompany.com"}]

The list is used as source data for varing angular components such as ngTags ngAutocomplete etc. I want to store the list in memcache so the Angular http calls will run faster.

The problem I'm having is that the values stored in memcache never last for more than a few minutes even though I've set it to last 26 hours. I'm aware that the actual value stored can not be over 1mb so as an experiment I hardcoded the list of employees to contain only three values and the problem still persists.

The appengine console is telling me the job ran successfully and if I run the job manually it will load the values into memcache but they'll only stay there for a few minutes. I've done this many times before with far greater amount of data so I can't understand what's going wrong. I have billing enabled and I'm not over quota.

Here is an example of the function used to load the data into memcache:

def update_employee_list():
try:
    # Get all 3000+ employees and generate a list of dictionaries
    fresh_emp_list = [{"text":"john@mycompany.com"},{"text":"mary@mycompany.com"},{"text":"paul@mycompany.com"}]

    the_cache_key = 'my_emp_list'
    emp_data = memcache.get(the_cache_key)

    # Kill the memcache packet so we can rebuild it.
    if emp_data is not None:
        memcache.delete(the_cache_key)

    # Rebuild the memcache packet      
    memcache.add(the_cache_key, fresh_emp_list, 93600) # this should last for 26 hours  

except Exception as e:
    logging.info('ERROR!!!...A failure occured while trying to setup the memcache packet: %s'%e.message)
    raise deferred.PermanentTaskFailure() 

Here is an example of the function the angular components use to get the data from memcache:

@route
def get_emails(self):
    self.meta.change_view('json')
    emp_emails = memcache.get('my_emp_list')
    if emp_emails is not None:
        self.context['data'] = emp_emails
    else:
        self.context['data'] = [] 

Here is an example of the cron setting in cron.yaml:

- url: /cron/lookups/update_employee_list
  description: Daily rebuild of Employee Data
  schedule: every day 06:00
  timezone: America/New_York 

Why can't appengine memcache hold on to a list of three dictionaries for more than a few minutes?

Any ideas are appreciated. Thanks

Solution

Unless you are using dedicated memcache (paid service) the cached values can and will be evicted at any time.

What you tell memcache by specifying a lifetime is when your value becomes invalid and can therefor be removed from memcache. That however does not guarantee that your value will stay that long in the memcache, it's just a capping to a cache value's maximum lifetime.

Note: The more you put in memcache the more it is likely that other values will get dropped. Therefor you should carefully consider what data you put in your cache. You should definitely not put every value you come across in the memcache.

On a sidenote: In the projects i recently worked in, we had a - sort of - maximum cache lifetime of about a day. No cache value ever lasted longer that that, even if the desired lifetime was much higher. Interestingly enough though the cache got cleared out at about the same time every day, even including very new values.

Thus: Never rely on memcache. Always use a persistent storage and memcache for performance boosts with high volume traffic.

Answered By - konqi

Answer Checked By - Willingham (PHPFixing Volunteer)