Multi threading C# application with SQL Server database calls

Question

I have a SQL Server database with 500,000 records in table main. There are also three other tables called child1, child2, and child3. The many to many relationships between child1, child2, child3, and main are implemented via the three relationship tables: main_child1_relationship, main_child2_relationship, and main_child3_relationship. I need to read the records in main, update main, and also insert into the relationship tables new rows as well as insert new records in the child tables. The records in the child tables have uniqueness constraints, so the pseudo-code for the actual calculation (CalculateDetails) would be something like:

for each record in main
{
   find its child1 like qualities
   for each one of its child1 qualities
   {
      find the record in child1 that matches that quality
      if found
      {
          add a record to main_child1_relationship to connect the two records
      }
      else
      {
          create a new record in child1 for the quality mentioned
          add a record to main_child1_relationship to connect the two records
      }
   }
   ...repeat the above for child2
   ...repeat the above for child3 
}

This works fine as a single threaded app. But it is too slow. The processing in C# is pretty heavy duty and takes too long. I want to turn this into a multi-threaded app.

What is the best way to do this? We are using Linq to Sql.

So far my approach has been to create a new DataContext object for each batch of records from main and use ThreadPool.QueueUserWorkItem to process it. However these batches are stepping on each other's toes because one thread adds a record and then the next thread tries to add the same one and ... I am getting all kinds of interesting SQL Server dead locks.

Here is the code:

    int skip = 0;
    List<int> thisBatch;
    Queue<List<int>> allBatches = new Queue<List<int>>();
    do
    {
        thisBatch = allIds
                .Skip(skip)
                .Take(numberOfRecordsToPullFromDBAtATime).ToList();
        allBatches.Enqueue(thisBatch);
        skip += numberOfRecordsToPullFromDBAtATime;

    } while (thisBatch.Count() > 0);

    while (allBatches.Count() > 0)
    {
        RRDataContext rrdc = new RRDataContext();

        var currentBatch = allBatches.Dequeue();
        lock (locker)  
        {
            runningTasks++;
        }
        System.Threading.ThreadPool.QueueUserWorkItem(x =>
                    ProcessBatch(currentBatch, rrdc));

        lock (locker) 
        {
            while (runningTasks > MAX_NUMBER_OF_THREADS)
            {
                 Monitor.Wait(locker);
                 UpdateGUI();
            }
        }
    }

And here is ProcessBatch:

    private static void ProcessBatch( 
        List<int> currentBatch, RRDataContext rrdc)
    {
        var topRecords = GetTopRecords(rrdc, currentBatch);
        CalculateDetails(rrdc, topRecords);
        rrdc.Dispose();

        lock (locker)
        {
            runningTasks--;
            Monitor.Pulse(locker);
        };
    }

And

    private static List<Record> GetTopRecords(RecipeRelationshipsDataContext rrdc, 
                                              List<int> thisBatch)
    {
        List<Record> topRecords;

        topRecords = rrdc.Records
                    .Where(x => thisBatch.Contains(x.Id))
                    .OrderBy(x => x.OrderByMe).ToList();
        return topRecords;
    }

CalculateDetails is best explained by the pseudo-code at the top.

I think there must be a better way to do this. Please help. Many thanks!

Answer 1

Here's my take on the problem:

• When using multiple threads to insert/update/query data in SQL Server, or any database, then deadlocks are a fact of life. You have to assume they will occur and handle them appropriately.

• That's not so say we shouldn't attempt to limit the occurence of deadlocks. However, it's easy to read up on the basic causes of deadlocks and take steps to prevent them, but SQL Server will always surprise you :-)

Some reason for deadlocks:

• Too many threads - try to limit the number of threads to a minimum, but of course we want more threads for maximum performance.

• Not enough indexes. If selects and updates aren't selective enough SQL will take out larger range locks than is healthy. Try to specify appropriate indexes.

• Too many indexes. Updating indexes causes deadlocks, so try to reduce indexes to the minimum required.

• Transaction isolational level too high. The default isolation level when using .NET is 'Serializable', whereas the default using SQL Server is 'Read Committed'. Reducing the isolation level can help a lot (if appropriate of course).

This is how I might tackle your problem:

• I wouldn't roll my own threading solution, I would use the TaskParallel library. My main method would look something like this:

  using (var dc = new TestDataContext())
  {
      // Get all the ids of interest.
      // I assume you mark successfully updated rows in some way
      // in the update transaction.
      List<int> ids = dc.TestItems.Where(...).Select(item => item.Id).ToList();
  
      var problematicIds = new List<ErrorType>();
  
      // Either allow the TaskParallel library to select what it considers
      // as the optimum degree of parallelism by omitting the 
      // ParallelOptions parameter, or specify what you want.
      Parallel.ForEach(ids, new ParallelOptions {MaxDegreeOfParallelism = 8},
                          id => CalculateDetails(id, problematicIds));
  }
  

• Execute the CalculateDetails method with retries for deadlock failures

  private static void CalculateDetails(int id, List<ErrorType> problematicIds)
  {
      try
      {
          // Handle deadlocks
          DeadlockRetryHelper.Execute(() => CalculateDetails(id));
      }
      catch (Exception e)
      {
          // Too many deadlock retries (or other exception). 
          // Record so we can diagnose problem or retry later
          problematicIds.Add(new ErrorType(id, e));
      }
  }
  

• The core CalculateDetails method

  private static void CalculateDetails(int id)
  {
      // Creating a new DeviceContext is not expensive.
      // No need to create outside of this method.
      using (var dc = new TestDataContext())
      {
          // TODO: adjust IsolationLevel to minimize deadlocks
          // If you don't need to change the isolation level 
          // then you can remove the TransactionScope altogether
          using (var scope = new TransactionScope(
              TransactionScopeOption.Required,
              new TransactionOptions {IsolationLevel = IsolationLevel.Serializable}))
          {
              TestItem item = dc.TestItems.Single(i => i.Id == id);
  
              // work done here
  
              dc.SubmitChanges();
              scope.Complete();
          }
      }
  }
  

• And of course my implementation of a deadlock retry helper

  public static class DeadlockRetryHelper
  {
      private const int MaxRetries = 4;
      private const int SqlDeadlock = 1205;
  
      public static void Execute(Action action, int maxRetries = MaxRetries)
      {
          if (HasAmbientTransaction())
          {
              // Deadlock blows out containing transaction
              // so no point retrying if already in tx.
              action();
          }
  
          int retries = 0;
  
          while (retries < maxRetries)
          {
              try
              {
                  action();
                  return;
              }
              catch (Exception e)
              {
                  if (IsSqlDeadlock(e))
                  {
                      retries++;
                      // Delay subsequent retries - not sure if this helps or not
                      Thread.Sleep(100 * retries);
                  }
                  else
                  {
                      throw;
                  }
              }
          }
  
          action();
      }
  
      private static bool HasAmbientTransaction()
      {
          return Transaction.Current != null;
      }
  
      private static bool IsSqlDeadlock(Exception exception)
      {
          if (exception == null)
          {
              return false;
          }
  
          var sqlException = exception as SqlException;
  
          if (sqlException != null && sqlException.Number == SqlDeadlock)
          {
              return true;
          }
  
          if (exception.InnerException != null)
          {
              return IsSqlDeadlock(exception.InnerException);
          }
  
          return false;
      }
  }
  

• One further possibility is to use a partitioning strategy

If your tables can naturally be partitioned into several distinct sets of data, then you can either use SQL Server partitioned tables and indexes, or you could manually split your existing tables into several sets of tables. I would recommend using SQL Server's partitioning, since the second option would be messy. Also built-in partitioning is only available on SQL Enterprise Edition.

If partitioning is possible for you, you could choose a partion scheme that broke you data in lets say 8 distinct sets. Now you could use your original single threaded code, but have 8 threads each targetting a separate partition. Now there won't be any (or at least a minimum number of) deadlocks.

I hope that makes sense.