javascript - Using setInterval() with a .map method in array and returning as a promise

Question

I have a function that takes a list of IDs, converts them into an array URLS, then uses the map function to fire a fetch request. It works great but it fires too fast and the provider throws errors because we hit the API too often. I need to set an interval for the request but everytime I do, it doesn't really work. Ideas?

async function getReports(reportIDs) {
    const urls = reportIDs.map(id => `https://api.data.com/api/v1/report/${id}/?include_datasets=true`);
    const requests = urls.map(url => fetch(url, {
        method: 'GET',
        headers: { 'api-key': key }
    }).then(res => res.json()));
    
    const responses = await Promise.all(requests).catch(err => console.error(err));
    return responses;
}

I use a promise so I can await the results of the function inside another function to transform the datasets.

Ideas?

Answer 1

“Simplicity is a great virtue but it requires hard work to achieve it and education to appreciate it. And to make matters worse: complexity sells better.” — Edsger W. Dijkstra

The accepted "lightweight" solution is nearly 20,000 lines of code and depends on both CoffeeScript and Lua. What if you could trade all of that for just 50 lines of JavaScript?

Let's say we have some job that takes some amount of time to compute some result -

async function job(x) {
  // job consumes some time
  await sleep(rand(5000))
  // job computes a result
  return x * 10
}

Promise.all([1,2,3,4,5,6,7,8,9,10,11,12].map(job))
  .then(console.log, console.error)

[10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120]

This runs all twelve (12) jobs at once. If these were requests to a remote, some connections could be rejected because you are flooding the server with too much simultaneous traffic. By modeling a Pool of threads, we control the flow of the parallelized jobs -

// my pool with four threads
const pool = new Pool(4)

async function jobQueued(x) {
  // wait for pool thread
  const close = await pool.open()
  // run the job and close the thread upon completion
  return job(x).then(close)
}

Promise.all([1,2,3,4,5,6,7,8,9,10,11,12].map(jobQueued))
  .then(console.log, console.error)

[10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120]

Functions should be small and do just one thing. This makes it easier to write individual features and promotes a higher degree of reusability, allowing you to combine several simiple features into more sosphisticated ones. Above you already saw rand and sleep -

const rand = x =>
  Math.random() * x

const sleep = ms =>
  new Promise(r => setTimeout(r, ms))

If we want to throttle each job, we can specialize sleep to ensure a minimum runtime -

const throttle = (p, ms) =>
  Promise.all([ p, sleep(ms) ]).then(([ value, _ ]) => value)

async function jobQueued(x) {
  const close = await pool.open()
  // ensure job takes at least 3 seconds before freeing thread
  return throttle(job(x), 3000).then(close)
}

Promise.all([1,2,3,4,5,6,7,8,9,10,11,12].map(jobQueued))
  .then(console.log, console.error)

[10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120]

We can add some console.log messages to ensure things are running properly. And we will add a random sleep at the beginning of the job to show that the tasks can queue in any order without affecting the order of the result -

async function jobQueued(x) {
  await sleep(rand(5000))
  console.log("queueing", x)
  const close = await pool.open()
  console.log("  sending", x)
  const result = await throttle(job(x), 3000).then(close)
  console.log("    received", result)
  return result
}

Promise.all([1,2,3,4,5,6,7,8,9,10,11,12].map(jobQueued))
  .then(console.log, console.error)

console.log	thread 1	thread 2	thread 3	thread 4
queueing 12	?	?	?	?
???sending 12	open	?	?	?
queueing 9	↓	?	?	?
???sending 9	↓	open	?	?
queueing 8	↓	↓	?	?
???sending 8	↓	↓	open	?
queueing 4	↓	↓	↓	?
???sending 4	↓	↓	↓	open
queueing 10	↓	↓	↓	↓
queueing 6	↓	↓	↓	↓
queueing 7	↓	↓	↓	↓
queueing 2	↓	↓	↓	↓
queueing 11	↓	↓	↓	↓
??????received 120	closed	↓	↓	↓
???sending 11	open	↓	↓	↓
queueing 3	↓	↓	↓	↓
queueing 5	↓	↓	↓	↓
queueing 1	↓	↓	↓	↓
??????received 80	↓	↓	closed	↓
???sending 1	↓	↓	open	↓
??????received 90	↓	closed	↓	↓
???sending 5	↓	open	↓	↓
??????received 110	closed	↓	↓	↓
???sending 3	open	↓	↓	↓
??????received 40	↓	↓	↓	closed
???sending 2	↓	↓	↓	open
??????received 10	↓	↓	closed	↓
???sending 7	↓	↓	open	↓
??????received 50	↓	closed	↓	↓
???sending 6	↓	open	↓	↓
??????received 20	↓	↓	↓	closed
???sending 10	↓	↓	↓	open
??????received 30	closed	↓	↓	↓
??????received 70	?	↓	closed	↓
??????received 60	?	closed	?	↓
??????received 100	?	?	?	closed