[FIXED] Why does a result() from concurrent.futures.as_completed() occasionally return None?

Issue

I've been writing a fairly complex series of modules to help test our poorly documented networking gear, this one focused on trying the various passwords used across the company. I've been using the concurrent.futures module to speed things along by testing many devices in parallel but am running in to a problem where occasionally a result comes back as None and it's a mystery to me as why. It revolves around this chunk of code:

def process_device(dev):
    check = CheckPass(creds, inv_list)
    dev = switch.Switch(**dev)
    online, net_device = check.device_online(dev)
    if online == True:
        conn, net_device = check.try_passwords(dev, return_conn=True)
        if conn != False and conn != "none":
            if conn != True:
                hostname,model,serial_num,version,date = net_device.get_id_details(conn)
                net_device.hostname = hostname
                net_device.model = model
                net_device.serial_num = serial_num
                net_device.version = version
                net_device.date = date
                conn.disconnect()
            conf_dev = check.confirm_device(net_device)
            check.dev_confirmed.append(conf_dev)
            return dev.hostname, dev.ip        

with concurrent.futures.ThreadPoolExecutor(max_workers=120) as executor:
    threads = {executor.submit(process_device, dev): dev for dev in inv_list}
    for future in concurrent.futures.as_completed(threads):
        name, ip = future.result()

At unpredictable intervals future.result() will have a result of None and I can't reliably reproduce it. The error occurs with different devices every time, I've checked the logs and fed it an inventory list containing just the device that it had processed and running that device by itself will be fine. I have tried using fewer workers and more. Clearly I'm not understanding something about how future.result() works. dev.hostname and dev.ip always have values so process_device() should always return them (barring unhandled exceptions, which haven't occurred) yet I always end up with TypeError: Cannot unpack non-iterable NoneType object referencing the name, ip = future.results() command.

Solution

It is not problem with future but with your function process_device() which sometimes can return None.

When online is False or when if conn != False and conn != "none": gives False then process_device() will run default return None

def process_device(dev):

    # ... your code ...

    return None  # default behavior

You should filter results

result = future.result()
if result:
    name, ip = result

Answered By - furas

Answer Checked By - Cary Denson (PHPFixing Admin)

Read More

[FIXED] Why is the file saved showing None while console shows the correct response?

Issue

When I try to save the responses to a file, the actual response is not saved even though it shows in the console. The result that is saved in the file is None. See examples below

from concurrent.futures import ThreadPoolExecutor
import requests
#from timer import timer


#########  create test file

URLsTest = '''
https://en.wikipedia.org/wiki/NBA
https://en.wikipedia.org/wiki/NFL
'''.strip()

with open('input.txt', 'w') as f:
    f.write(URLsTest)
    
####################

with open('input.txt', 'r') as f:
    urls=f.read().split('\n')    # url list

def fetch(tt):  # received tuple
    session, url = tt
    print('Processing')
    with session.get(url) as response:
        print(response.text)

#@timer(1, 5)
def main():
    with ThreadPoolExecutor(max_workers=100) as executor:
        with requests.Session() as session:  # for now, just one session
            results = executor.map(fetch, [(session, u) for u in urls])  # tuple list (session, url), each tuple passed to function
            executor.shutdown(wait=True)
    # write all results to text file
    with open('output.txt', 'w') as f2:
        for r in results:  # tuple (url, html)
            f2.write("%s\n" % r)
            
main()

Response file - output.txt

None    
None

Solution

First of all, you could avoid printing the html since you are saving that output to a file. That way you can avoid using resources to print the results.

Then, your fetch is not returning anything for the results. Therefore you should change your print for a return So instead of printing return the response.text

# print(response.text)
return response.text

Answered By - kachus22

Answer Checked By - Katrina (PHPFixing Volunteer)

Read More