【问题标题】:Limit the number of parallel threads in C#限制C#中的并行线程数
【发布时间】:2012-01-13 16:32:56
【问题描述】:

我正在编写一个 C# 程序来通过 FTP 生成和上传 50 万个文件。我想并行处理 4 个文件,因为机器有 4 个内核并且文件生成需要更长的时间。是否可以将以下 Powershell 示例转换为 C#?或者有没有更好的框架,比如C#中的Actor框架(比如F# MailboxProcessor)?

Powershell example

$maxConcurrentJobs = 3;

# Read the input and queue it up
$jobInput = get-content .\input.txt
$queue = [System.Collections.Queue]::Synchronized( (New-Object System.Collections.Queue) )
foreach($item in $jobInput)
{
    $queue.Enqueue($item)
}

# Function that pops input off the queue and starts a job with it
function RunJobFromQueue
{
    if( $queue.Count -gt 0)
    {
        $j = Start-Job -ScriptBlock {param($x); Get-WinEvent -LogName $x} -ArgumentList $queue.Dequeue()
        Register-ObjectEvent -InputObject $j -EventName StateChanged -Action { RunJobFromQueue; Unregister-Event $eventsubscriber.SourceIdentifier; Remove-Job $eventsubscriber.SourceIdentifier } | Out-Null
    }
}

# Start up to the max number of concurrent jobs
# Each job will take care of running the rest
for( $i = 0; $i -lt $maxConcurrentJobs; $i++ )
{
    RunJobFromQueue
}

更新:
与远程 FTP 服务器的连接可能很慢,所以我想限制 FTP 上传处理。

【问题讨论】:

  • 如果要限制并行任务的数量,为什么不使用TPL?
  • 线程池应该足够智能来为你处理这个问题。为什么要尝试自己管理?
  • 您可以使用PLINQ 并相应地设置WithDegreeOfParallelism
  • @M.Babcock 我已经更新了问题 - 我想限制慢速 ftp 连接的上传任务。
  • @M.Babcock - SetMaxThreads 不过是个坏主意。别管它。

标签: c# c#-4.0 parallel-processing


【解决方案1】:

假设您使用 TPL 构建它,您可以将 ParallelOptions.MaxDegreesOfParallelism 设置为您想要的任何值。

Parallel.For 获取代码示例。

【讨论】:

    【解决方案2】:

    Task Parallel Library 是您的朋友。请参阅this 链接,该链接描述了您可以使用的内容。基本上框架 4 附带了它,它将这些基本上后台线程池线程优化为运行机器上的处理器数量。

    大概是这样的:

    ParallelOptions options = new ParallelOptions();
    
    options.MaxDegreeOfParallelism = 4;
    

    然后在你的循环中类似:

    Parallel.Invoke(options,
     () => new WebClient().Upload("http://www.linqpad.net", "lp.html"),
     () => new WebClient().Upload("http://www.jaoo.dk", "jaoo.html"));
    

    【讨论】:

      【解决方案3】:

      如果您使用的是 .Net 4.0,则可以使用 Parallel library

      假设您正在迭代 50 万个文件,您可以使用 Parallel Foreach for instancehave a look to PLinq“并行”迭代 这里是comparison between the two

      【讨论】:

      • 这个问题用 C#-4.0 标记,很明显他既熟悉扩展又使用 .NET 4。一句话不能回答他的问题。
      • 很明显他在使用 C#4.0,但他对 Parallel 库并不明显,否则他不会问任何问题。此外,我的回复包含或多或少与另一个相同的信息。请证明 -1 的合理性。
      【解决方案4】:

      基本上,您需要为要上传的每个文件创建一个操作或任务,将它们放在一个列表中,然后处理该列表,从而限制可以并行处理的数量。

      My blog post 展示了如何使用任务和操作来执行此操作,并提供了一个示例项目,您可以下载并运行以查看两者的实际效果。

      有动作

      如果使用 Actions,您可以使用内置的 .Net Parallel.Invoke 函数。这里我们限制它最多并行运行 4 个线程。

      var listOfActions = new List<Action>();
      foreach (var file in files)
      {
          var localFile = file;
          // Note that we create the Task here, but do not start it.
          listOfTasks.Add(new Task(() => UploadFile(localFile)));
      }
      
      var options = new ParallelOptions {MaxDegreeOfParallelism = 4};
      Parallel.Invoke(options, listOfActions.ToArray());
      

      这个选项虽然不支持异步,我假设你的 FileUpload 函数会,所以你可能想使用下面的任务示例。

      有任务

      Tasks 没有内置函数。但是,您可以使用我在博客上提供的那个。

          /// <summary>
          /// Starts the given tasks and waits for them to complete. This will run, at most, the specified number of tasks in parallel.
          /// <para>NOTE: If one of the given tasks has already been started, an exception will be thrown.</para>
          /// </summary>
          /// <param name="tasksToRun">The tasks to run.</param>
          /// <param name="maxTasksToRunInParallel">The maximum number of tasks to run in parallel.</param>
          /// <param name="cancellationToken">The cancellation token.</param>
          public static async Task StartAndWaitAllThrottledAsync(IEnumerable<Task> tasksToRun, int maxTasksToRunInParallel, CancellationToken cancellationToken = new CancellationToken())
          {
              await StartAndWaitAllThrottledAsync(tasksToRun, maxTasksToRunInParallel, -1, cancellationToken);
          }
      
          /// <summary>
          /// Starts the given tasks and waits for them to complete. This will run the specified number of tasks in parallel.
          /// <para>NOTE: If a timeout is reached before the Task completes, another Task may be started, potentially running more than the specified maximum allowed.</para>
          /// <para>NOTE: If one of the given tasks has already been started, an exception will be thrown.</para>
          /// </summary>
          /// <param name="tasksToRun">The tasks to run.</param>
          /// <param name="maxTasksToRunInParallel">The maximum number of tasks to run in parallel.</param>
          /// <param name="timeoutInMilliseconds">The maximum milliseconds we should allow the max tasks to run in parallel before allowing another task to start. Specify -1 to wait indefinitely.</param>
          /// <param name="cancellationToken">The cancellation token.</param>
          public static async Task StartAndWaitAllThrottledAsync(IEnumerable<Task> tasksToRun, int maxTasksToRunInParallel, int timeoutInMilliseconds, CancellationToken cancellationToken = new CancellationToken())
          {
              // Convert to a list of tasks so that we don't enumerate over it multiple times needlessly.
              var tasks = tasksToRun.ToList();
      
              using (var throttler = new SemaphoreSlim(maxTasksToRunInParallel))
              {
                  var postTaskTasks = new List<Task>();
      
                  // Have each task notify the throttler when it completes so that it decrements the number of tasks currently running.
                  tasks.ForEach(t => postTaskTasks.Add(t.ContinueWith(tsk => throttler.Release())));
      
                  // Start running each task.
                  foreach (var task in tasks)
                  {
                      // Increment the number of tasks currently running and wait if too many are running.
                      await throttler.WaitAsync(timeoutInMilliseconds, cancellationToken);
      
                      cancellationToken.ThrowIfCancellationRequested();
                      task.Start();
                  }
      
                  // Wait for all of the provided tasks to complete.
                  // We wait on the list of "post" tasks instead of the original tasks, otherwise there is a potential race condition where the throttler's using block is exited before some Tasks have had their "post" action completed, which references the throttler, resulting in an exception due to accessing a disposed object.
                  await Task.WhenAll(postTaskTasks.ToArray());
              }
          }
      

      然后创建任务列表并调用函数让它们运行,例如一次最多同时运行 4 个,您可以这样做:

      var listOfTasks = new List<Task>();
      foreach (var file in files)
      {
          var localFile = file;
          // Note that we create the Task here, but do not start it.
          listOfTasks.Add(new Task(async () => await UploadFile(localFile)));
      }
      await Tasks.StartAndWaitAllThrottledAsync(listOfTasks, 4);
      

      另外,由于此方法支持异步,它不会像使用 Parallel.Invoke 或 Parallel.ForEach 那样阻塞 UI 线程。

      【讨论】:

        【解决方案5】:

        我编写了以下技术,我使用 BlockingCollection 作为线程计数管理器。实现和处理这项工作非常简单。 它简单地接受任务对象并将一个整数值添加到阻塞列表中,将运行线程计数增加 1。当线程完成时,它使对象出列并在添加操作时为即将执行的任务释放块。

                public class BlockingTaskQueue
                {
                    private BlockingCollection<int> threadManager { get; set; } = null;
                    public bool IsWorking
                    {
                        get
                        {
                            return threadManager.Count > 0 ? true : false;
                        }
                    }
        
                    public BlockingTaskQueue(int maxThread)
                    {
                        threadManager = new BlockingCollection<int>(maxThread);
                    }
        
                    public async Task AddTask(Task task)
                    {
                        Task.Run(() =>
                        {
                            Run(task);
                        });
                    }
        
                    private bool Run(Task task)
                    {
                        try
                        {
                            threadManager.Add(1);
                            task.Start();
                            task.Wait();
                            return true;
        
                        }
                        catch (Exception ex)
                        {
                            return false;
                        }
                        finally
                        {
                            threadManager.Take();
                        }
        
                    }
        
                }
        

        【讨论】:

          猜你喜欢
          • 1970-01-01
          • 1970-01-01
          • 2017-12-23
          • 1970-01-01
          • 2014-05-31
          • 1970-01-01
          • 1970-01-01
          • 1970-01-01
          • 2012-05-07
          相关资源
          最近更新 更多