Azure Data Factory Performance Issue - Until Activity Loop Exist consumes Lot of Time

Question

I have created a pipeline where I have used an until activity to iterate through the files and load the data into a SQL table. The logic used in the until activity expression is compared to the number of files available in the data lake with an incremental variable value. If the variable value is equal to or greater than the total number of files the loop will exit.

If we have less than 5 files in the data lake then the loop may exit within 1 minute but if we have more than 25 it takes nearly 8 minutes to exit the loop. AS the value of the count of files increases the time taken to exit the loop also increases.

For Eg: We have 35 Files, all the files got processed and even the last activity inside the loop also successfully got executed, but the "until" activity again runs for another 12-14 minutes without any further activity to get executed.

Any help to improve the performance of the until activity loop exit would be really helpful.

NB: Our pipeline requires sorting/failure exit/and other logical implementation methods that's why we did not use the for-each loop activity.

Regards, Sandeep

question from:https://stackoverflow.com/questions/65933538/azure-data-factory-performance-issue-until-activity-loop-exist-consumes-lot-of

Answer 1

According to the documentation, we can scale up the DIU to improve the performance.
We can increase the Computer type and Core count when creating the IR.