[FIXED] How to properly call method asynchronous without blocking main thread?

Issue

I have a service with some kind of main-loop. In this service I need to send request to other service with some metadata about operation. I need to do it asynchronous without blocking main thread.

Here is my code:

def process_response(self, request, response, spider):
    if self.enabled:
        Thread(target=self._send_request_to_information_supplier,
               args=(response,)).start()
        # I don't want this flow to be blocked by above thread.
        return response
    return response


def _send_request_to_information_supplier(self, html_response):
    some_metadata = InformationSupplierMetadata(url=html_response.url,
                                                html=html_response.text).convert_to_dict()
    try:
        request_response = requests.post(url=self.url,
                                         data=some_metadata,
                                         headers={'Content-Type': 'application/json; charset=utf-8'},
                                         timeout=self.timeout,
                                         verify=False)
        self.logger.debug('Request was sent and end up with status {0}'.format(request_response.status_code))
    except ConnectTimeout:
        self.logger.debug('Request timed out')

I am beginner in multi-threading and I am not sure this is the correct way to do it. Maybe someone could suggest better approach?

Solution

you can use the ThreadPoolExecutor. Here is an example that may can help.

from concurrent.futures import ThreadPoolExecutor

executor = ThreadPoolExecutor(24)

def process_response(self, request, response, spider):
    if self.enabled:
        executor.submit(self._send_request_to_information_supplier, response)

        # I don't want this flow to be blocked by above thread.
        return response
    return response


def _send_request_to_information_supplier(self, html_response):
    ...

this way the executor will having 24 maximum number of threads, and the submit will run the method in one thread if any available.

Answered By - Fran N.J.

Answer Checked By - Mary Flores (PHPFixing Volunteer)