[FIXED] How to pickle a defaultdict which uses a lambda function?

Issue

How do I create pickle file of function (defaultDict)? The error I get is can't pickle function objects

from collections import defaultdict
dtree = lambda: defaultdict(tree)

try:    import cPickle as pickle
except: import pickle

#Create defaultdict  object:
hapPkl = dtree()

#Create Pickle file
f  = open("hapP.pkl","wb")
pickle.dump(hapPkl,f)
f.close()

StackTrace:

TypeError                                 Traceback (most recent call last)
<ipython-input-287-376cac3b4f0d> in <module>()
      1 f  = open("hapP.pkl","wb")
----> 2 pickle.dump(hapPkl,f)
      3 f.close()

/usr/lib64/python2.7/copy_reg.pyc in _reduce_ex(self, proto)
     68     else:
     69         if base is self.__class__:
---> 70             raise TypeError, "can't pickle %s objects" % base.__name__
     71         state = base(self)
     72     args = (self.__class__, base, state)

TypeError: can't pickle function objects

Solution

A simple workaround would be to implement your tree data-structure differently, without defaultdict:

class DTree(dict):
    def __missing__(self, key):
        value = self[key] = type(self)()
        return value

try:    import cPickle as pickle
except: import pickle

#Create dtree object:
hapPkl = DTree()

#Create Pickle file
with open("hapP.pkl", "wb") as f:
    pickle.dump(hapPkl, f)

Answered By - martineau

Answer Checked By - David Marino (PHPFixing Volunteer)

Read More

[FIXED] How to load a pickle file from S3 to use in AWS Lambda?

Issue

I am currently trying to load a pickled file from S3 into AWS lambda and store it to a list (the pickle is a list).

Here is my code:

import pickle
import boto3

s3 = boto3.resource('s3')
with open('oldscreenurls.pkl', 'rb') as data:
    old_list = s3.Bucket("pythonpickles").download_fileobj("oldscreenurls.pkl", data)

I get the following error even though the file exists:

FileNotFoundError: [Errno 2] No such file or directory: 'oldscreenurls.pkl'

Any ideas?

Solution

As shown in the documentation for download_fileobj, you need to open the file in binary write mode and save to the file first. Once the file is downloaded, you can open it for reading and unpickle.

import pickle
import boto3

s3 = boto3.resource('s3')
with open('oldscreenurls.pkl', 'wb') as data:
    s3.Bucket("pythonpickles").download_fileobj("oldscreenurls.pkl", data)

with open('oldscreenurls.pkl', 'rb') as data:
    old_list = pickle.load(data)

download_fileobj takes the name of an object in S3 plus a handle to a local file, and saves the contents of that object to the file. There is also a version of this function called download_file that takes a filename instead of an open file handle and handles opening it for you.

In this case it would probably be better to use S3Client.get_object though, to avoid having to write and then immediately read a file. You could also write to an in-memory BytesIO object, which acts like a file but doesn't actually touch a disk. That would look something like this:

import pickle
import boto3
from io import BytesIO

s3 = boto3.resource('s3')
with BytesIO() as data:
    s3.Bucket("pythonpickles").download_fileobj("oldscreenurls.pkl", data)
    data.seek(0)    # move back to the beginning after writing
    old_list = pickle.load(data)

Answered By - avigil

Answer Checked By - Cary Denson (PHPFixing Admin)

Read More