python - Multiprocessing.Pool runs only one process

Question

I have a Django project and I must fill/update the database with data from an external API.

For every user, I must update the values for every minute of every day fetched by the external API. As you understand this is a time-consuming process if I pass multiple months.

Please note that I make a call for each day, and process the data of the fetched json. After a specific number of requests, I must wait one hour.

With all that in mind, I prepared a script that did all the work but it was taking ages to complete so I decided to proceed with multiprocessing.

There is a method user_sync(day_list, user) that handles everything and works perfectly.

and here are the lines of code that initiate multiprocessing.

user_list = User.objects.filter(username__startswith='US')

start_date = datetime.datetime(2020, 6, 28, 0, 0, tzinfo=pytz.UTC)
end_date = datetime.datetime(2021, 1, 20, 0, 0, tzinfo=pytz.UTC)

day_list = create_day_list(start_date, end_date)
pool = multiprocessing.Pool()
try:
    part = partial(user_sync, day_list)
    pool.map(part, user_list)
    pool.join()
except Exception as e:
    print(e)
    pool.close()
finally:
    pool.close()
    print('FINISHED')

To my understanding, it should run multiple user_sync methods for every user in parallel but it doesn't

Any help or point to the right direction would be much appreciated.

question from:https://stackoverflow.com/questions/65834964/multiprocessing-pool-runs-only-one-process

Answer 1

Waitting for answers