[FIXED] How to cache paged queries using memcache in gae, ndb?

Issue

I'm having a really difficult time figuring out how can I cache a paged query.

I'm building a Forum with ndb and gae. The front page is the default forum with a limited amount of posts and a Next button to load more posts.

This posts are retrieved with ndb fetch_page and I found that is really common for the users to navigate through pages. So instead of querying the datastore for every user request I would like to store each page in memcache and let it do the work.

I tried to do it and I can only cache the next pages, but not the previous ones.

Also, I don't know how to invalidate all cached results when a user creates a new post.

Anyone can help?

EDIT:

The following code returns 10 records of any class given a bookmark (a urlsafe of a cursor).

How can y cache every page considering that must work going forward and backwards and you can invalidate all the queries cached for a given query for example when the user puts a new record.

def return_query_page(cls, size=10, bookmark=None, is_prev=None, equality_filters=None, orders=None):
    """
    Generate a paginated result on any class
    Param cls: The ndb model class to query
    Param size: The size of the results
    Param bokkmark: The urlsafe cursor of the previous queries. First time will be None
    Param is_prev: If your requesting for a next result or the previous ones
    Param equal_filters: a dictionary of {'property': value} to apply equality filters only
    Param orders: a dictionary of {'property': '-' or ''} to order the results like .order(cls.property)
    Return: a tuple (list of results, Previous cursor bookmark, Next cursor bookmark)
    """
    if bookmark:
        cursor = ndb.Cursor(urlsafe=bookmark)
    else:
        is_prev = None
        cursor = None

    q = cls.query()
    try:
        for prop, value in equality_filters.iteritems():
            q = q.filter(cls._properties[prop] == value)

        q_forward = q.filter()
        q_reverse = q.filter()

        for prop, value in orders.iteritems():
            if value == '-':
                q_forward = q_forward.order(-cls._properties[prop])
                q_reverse = q_reverse.order(cls._properties[prop])
            else:
                q_forward = q_forward.order(cls._properties[prop])
                q_reverse = q_reverse.order(-cls._properties[prop])
    except:
        return None, None, None
    if is_prev:
        qry = q_reverse
        new_cursor = cursor.reversed() if cursor else None
    else:
        qry = q_forward
        new_cursor = cursor if cursor else None

    results, new_cursor, more = qry.fetch_page(size, start_cursor=new_cursor)
    if more and new_cursor:
        more = True
    else:
        more = False

    if is_prev:
        prev_bookmark = new_cursor.reversed().urlsafe() if more else None
        next_bookmark = bookmark
        results.reverse()
    else:
        prev_bookmark = bookmark
        next_bookmark = new_cursor.urlsafe() if more else None

    return results, prev_bookmark, next_bookmark

Solution

You might be better off using a keys_only query and then doing a key.get() on each returned key.

That way, memcache will be used for each post.

Example (assuming Post is the model):

keys, cursor, more = Post.query().fetch_page(30, start_cursor=cursor, keys_only=True)
entities = ndb.get_multi(keys)

Answered By - David Bennett

Answer Checked By - Marie Seifert (PHPFixing Admin)