限制任务并行库中的线程数

Question

我有数百个文件需要上传到 Azure Blob 存储。
我想使用并行任务库。
但是，不是运行所有 100 个线程以在文件列表上的 foreach 中上传，而是如何限制它可以使用并并行完成工作的最大线程数。 还是它会自动平衡事物？

Answer 1

您根本不应该为此使用线程。为此有一个基于Task 的API，它自然是异步的：CloudBlockBlob.UploadFromFileAsync。将它与async/await 和SemaphoreSlim 一起使用来限制并行上传的数量。

示例（未经测试）：

const MAX_PARALLEL_UPLOADS = 5;

async Task UploadFiles()
{
    var files = new List<string>();
    // ... add files to the list

    // init the blob block and
    // upload files asynchronously
    using (var blobBlock = new CloudBlockBlob(url, credentials))
    using (var semaphore = new SemaphoreSlim(MAX_PARALLEL_UPLOADS))
    {
        var tasks = files.Select(async(filename) => 
        {
            await semaphore.WaitAsync();
            try
            {
                await blobBlock.UploadFromFileAsync(filename, FileMode.Create);
            }
            finally
            {
                semaphore.Release();
            }
        }).ToArray();

        await Task.WhenAll(tasks);
    }
}

Answer 2

您是否尝试过使用 MaxDegreeOfParallelism？像这样：

System.Threading.Tasks.Parallel.Invoke(
new Tasks.ParallelOptions {MaxDegreeOfParallelism =  5 }, actionsArray)

Answer 3

基本上，您将要为要上传的每个文件创建一个操作或任务，将它们放在一个列表中，然后处理该列表，从而限制可以并行处理的数量。

My blog post 展示了如何使用任务和操作来执行此操作，并提供了一个示例项目，您可以下载并运行以查看两者的实际效果。

有动作

如果使用 Actions，您可以使用内置的 .Net Parallel.Invoke 函数。这里我们限制它最多并行运行 5 个线程。

var listOfActions = new List<Action>();
foreach (var file in files)
{
    var localFile = file;
    // Note that we create the Task here, but do not start it.
    listOfTasks.Add(new Task(() => blobBlock.UploadFromFileAsync(localFile, FileMode.Create)));
}

var options = new ParallelOptions {MaxDegreeOfParallelism = 5};
Parallel.Invoke(options, listOfActions.ToArray());

这个选项没有利用 UploadFromFileAsync 的异步特性，所以你可能想使用下面的任务示例。

有任务

Tasks 没有内置函数。不过，您可以使用我在博客上提供的那个。

    /// <summary>
    /// Starts the given tasks and waits for them to complete. This will run, at most, the specified number of tasks in parallel.
    /// <para>NOTE: If one of the given tasks has already been started, an exception will be thrown.</para>
    /// </summary>
    /// <param name="tasksToRun">The tasks to run.</param>
    /// <param name="maxTasksToRunInParallel">The maximum number of tasks to run in parallel.</param>
    /// <param name="cancellationToken">The cancellation token.</param>
    public static async Task StartAndWaitAllThrottledAsync(IEnumerable<Task> tasksToRun, int maxTasksToRunInParallel, CancellationToken cancellationToken = new CancellationToken())
    {
        await StartAndWaitAllThrottledAsync(tasksToRun, maxTasksToRunInParallel, -1, cancellationToken);
    }

    /// <summary>
    /// Starts the given tasks and waits for them to complete. This will run the specified number of tasks in parallel.
    /// <para>NOTE: If a timeout is reached before the Task completes, another Task may be started, potentially running more than the specified maximum allowed.</para>
    /// <para>NOTE: If one of the given tasks has already been started, an exception will be thrown.</para>
    /// </summary>
    /// <param name="tasksToRun">The tasks to run.</param>
    /// <param name="maxTasksToRunInParallel">The maximum number of tasks to run in parallel.</param>
    /// <param name="timeoutInMilliseconds">The maximum milliseconds we should allow the max tasks to run in parallel before allowing another task to start. Specify -1 to wait indefinitely.</param>
    /// <param name="cancellationToken">The cancellation token.</param>
    public static async Task StartAndWaitAllThrottledAsync(IEnumerable<Task> tasksToRun, int maxTasksToRunInParallel, int timeoutInMilliseconds, CancellationToken cancellationToken = new CancellationToken())
    {
        // Convert to a list of tasks so that we don't enumerate over it multiple times needlessly.
        var tasks = tasksToRun.ToList();

        using (var throttler = new SemaphoreSlim(maxTasksToRunInParallel))
        {
            var postTaskTasks = new List<Task>();

            // Have each task notify the throttler when it completes so that it decrements the number of tasks currently running.
            tasks.ForEach(t => postTaskTasks.Add(t.ContinueWith(tsk => throttler.Release())));

            // Start running each task.
            foreach (var task in tasks)
            {
                // Increment the number of tasks currently running and wait if too many are running.
                await throttler.WaitAsync(timeoutInMilliseconds, cancellationToken);

                cancellationToken.ThrowIfCancellationRequested();
                task.Start();
            }

            // Wait for all of the provided tasks to complete.
            // We wait on the list of "post" tasks instead of the original tasks, otherwise there is a potential race condition where the throttler's using block is exited before some Tasks have had their "post" action completed, which references the throttler, resulting in an exception due to accessing a disposed object.
            await Task.WhenAll(postTaskTasks.ToArray());
        }
    }

然后创建任务列表并调用函数让它们运行，例如一次最多同时运行 5 个，您可以这样做：

var listOfTasks = new List<Task>();
foreach (var file in files)
{
    var localFile = file;
    // Note that we create the Task here, but do not start it.
    listOfTasks.Add(new Task(async () => await blobBlock.UploadFromFileAsync(localFile, FileMode.Create)));
}
await Tasks.StartAndWaitAllThrottledAsync(listOfTasks, 5);

Answer 4

你可以通过运行这个找到：

class Program
{
    static void Main(string[] args)
    {
        var list = new List<int>();

        for (int i = 0; i < 100; i++)
        {
            list.Add(i);
        }

        var runningIndex = 0;

        Task.Factory.StartNew(() => Action(ref runningIndex));

        Parallel.ForEach(list, i =>
        {
            runningIndex ++;
            Console.WriteLine(i);
            Thread.Sleep(3000);
        });

        Console.ReadKey();
    }

    private static void Action(ref int number)
    {
        while (true)
        {
            Console.WriteLine("worked through {0}", number);
            Thread.Sleep(2900);
        }
    }
}

如您所见，并行数在开始时较小，逐渐变大，最后逐渐变小。所以肯定会进行某种自动优化。