[FIXED] How to run multiple(>200) http requests in node.js without getting ENOTFOUND error

Issue

I am trying to scrape a big list of pages from a website. Here is the axios request which I make :

 Promise.all(urls.map((url, index) => axios(url))).then(function (values) {
        ........
 });

The urls object just have a bunch of urls > 13000. If i keep that number to 200 the whole code works fine. But when I put in the entire object i get this:

errno: 'ENOTFOUND', code: 'ENOTFOUND', syscall: 'getaddrinfo',

I have read all the existing answers but nothing helped so please!

Solution

This you need to hit the server in chunks of request like 100 at a time. Send as every time you try to call API it goes to the DNS server and due to so. many requests it either sometimes discard or not able to get your host address in the registry. So try using dnscache from npm. https://www.npmjs.com/package/dnscache.

Also for more context check https://github.com/request/request/issues/2536

Do require('dnscache')({ enable: true }) in a proxy file or where your code lies of calling API. This will make DNS package the first check-in local cache for the host and then go to DNS if not found locally.

Answered By - Himanshu Joshi

Answer Checked By - Dawn Plyler (PHPFixing Volunteer)