python 3.x - How to make a group from tasks with bind = True?

Question

Task with bind=True

I have a celery task that runs a computation lasting a few seconds.

from celery import states

@celery.task(name="crunch.task", bind=True)
def crunch(self, data):
    try:
        pass
        # ... run computation with data here
    except Exception as exc:
        self.update_state(
            state=states.FAILURE,
            meta={"details": "error details here"}
        )
        raise exc

The important feature here is I'm using bind=True which passes the task into the function as the self parameter. This allows access to the task's update_state method which is great for error handling.

Group job

I now want to run this task in a batch job using celery.group.

@celery.task(name="batch.task", bind=True)
def batch(self, data_list):
    try:
        # HERE! IT SEEM WRONG TO PASS `SELF` INTO THE CHILD TASKS
        job = group([crunch(self, data) for data in data_list])  # <-- here `self` should be created by celery!
        job.async_apply()
    except Exception as exc:
        self.update_state(
            state=states.FAILURE,
            meta={"details": "error details here"}
        )
        raise exc

How can I make a celery.group composed of tasks with bind=True?

question from:https://stackoverflow.com/questions/65891987/how-to-make-a-group-from-tasks-with-bind-true

Answer 1

You need to work with signatures,

crunch.si(data)