.NET Core 3.1.4 XUnit 集成测试随机失败答案

【问题标题】：.NET Core 3.1.4 XUnit Integration tests failing randomly.NET Core 3.1.4 XUnit 集成测试随机失败
【发布时间】：2020-10-27 13:08:37
【问题描述】：

我正在尝试集成测试示例 API，并希望使用实际数据库的副本来执行此操作。一次运行测试一个文件会导致全部成功。执行 dotnet 测试 或使用 .NET 测试资源管理器运行所有测试会导致某些测试随机失败，但通常是 Theories。

信息：

数据库：PostgreSql-docker 映像
API：.NET Core 3.1.4
测试框架：XUnit 2.4.0

试过了：

减少总测试量[相同结果]
将 httpClient 设为静态 [相同结果]
使 ConfigureWebHost 异步 [无法这样做]
未运行 EnsureDeleted() 方法 [除删除测试外的所有测试都有效（意外惊喜）]
在运行测试之前手动删除测试数据库 [Works]

如您所见，我找到了一种解决方法，但这还不够。我希望能够从构建服务器运行 dotnet test 而无需创建变通方法脚本。我的理论是多个夹具尝试多次删除和创建数据库并失败，因为其他测试正在使用它？

CustomWebApplicationFactory：

using System;
using System.Linq;
using System.Threading;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Mvc.Testing;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Api;
using DataAccess;
using Microsoft.Extensions.Configuration;

namespace Tests.Api.Controllers
{
    public class CustomWebApplicationFactory<TStartup> : WebApplicationFactory<Startup>
    {
        protected override void ConfigureWebHost(IWebHostBuilder builder)
        {
            builder.ConfigureServices(services =>
            {
                RemoveOriginalDbContextService(services);
                SetDbContextOptions(services);

                var serviceProvider = services.BuildServiceProvider();

                // Create a scope to obtain a reference to the database context
                using (var scope = serviceProvider.CreateScope())
                {
                    var scopedServices = scope.ServiceProvider;
                    var logger = scopedServices.GetRequiredService<ILogger<CustomWebApplicationFactory<TStartup>>>();

                    try
                    {
                        using (var dbContext = scopedServices.GetRequiredService<TestDatabaseContext>())
                        {
                            dbContext.Database.EnsureDeleted(); //This will make random tests fail
                            dbContext.Database.EnsureCreated();
                        }
                    }
                    catch (Exception ex)
                    {
                        logger.LogError(ex, "An error occurred while initializing " +
                                            "database. Error: {ex.Message}");
                    }
                }
            });
        }

        private void RemoveOriginalDbContextService(IServiceCollection services)
        {
            var descriptor = services.SingleOrDefault(
                d => d.ServiceType == typeof(DbContextOptions<DatabaseContext>));

            if (descriptor != null) services.Remove(descriptor);
        }

        private void SetDbContextOptions(IServiceCollection services)
        {
            /***
            * Gets the configuration from the appsettings.json placed in the Api folder.
            ***/
            IConfiguration configuration = new ConfigurationBuilder()
                .AddJsonFile("appsettings.json")
                .Build();

            /***
            * This is to register the test database context as a service.
            * Not having it results in: "No service for type 'TestDatabaseContext' has been registered."
            ***/
            services.AddDbContext<TestDatabaseContext>();

            /***
            * These settings will be used to instansiate the TestDatabaseContext, 
            * as it takes DbContextOptions<DatabaseContext> as a parameter in its constructor.
            * Not having it results in InvalidArgumentException when trying to instansiate the TestDatabaseContext (Dependency Injection)
            ***/
            services.AddDbContext<DatabaseContext>(options =>
                options.UseNpgsql(configuration.GetConnectionString("TestingDatabase")));
        }
    }
}

示例测试类：

using System;
using System.Text;
using System.Net;
using System.Net.Http;
using System.Collections.Generic;
using Newtonsoft.Json;
using Domain.DataModels;
using Xunit;
using Api;

namespace Tests.Api.Controllers
{
    public class JobTests : IClassFixture<CustomWebApplicationFactory<Startup>>, IDisposable
    {
        private readonly HttpClient _client;
        private const string JOBURI = "/jobs";

        public JobTests(CustomWebApplicationFactory<Startup> factory)
        {
            _client = factory.CreateClient();
        }

        public void Dispose()
        {
            _client.Dispose();
        }

        [Fact]
        public async void Insert_WithValidTitle_ShouldReturnOkAndTheNewlyCreatedJob()
        {
            //Given
            //Seeded project exist
            var projectId = 1011;
            string title = "Test";
            Job job = new Job
            {
                ProjectId = projectId,
                Title = title
            };

            //When
            var httpResponse = await _client.PostAsync(JOBURI, ContentHelper.GetStringContent(job));

            //Then
            httpResponse.EnsureSuccessStatusCode();
            var newJob = await TestUtilities.GetDeserializedResponseBody<Job>(httpResponse);
            Assert.Equal(title, newJob.Title);
        }

        [Fact]
        public async void Insert_WithoutProjectId_ShouldReturnBadRequest()
        {
            //Given
            Job job = new Job { Title = "Test" };

            //When
            var httpResponse = await _client.PostAsync(JOBURI, ContentHelper.GetStringContent(job));

            //Then
            Assert.Equal(HttpStatusCode.BadRequest, httpResponse.StatusCode);
        }

        [Fact]
        public async void Insert_WithNonExistingProjectId_ShouldReturnBadRequest()
        {
            //Given
            Job job = new Job
            {
                ProjectId = 777777,
                Title = "Test"
            };

            //When
            var httpResponse = await _client.PostAsync(JOBURI, ContentHelper.GetStringContent(job));

            //Then
            Assert.Equal(HttpStatusCode.BadRequest, httpResponse.StatusCode);
        }

        [Fact]
        public async void Insert_WithoutTitle_ShouldReturnBadRequest()
        {
            //Given
            //Seeded project exist
            Job job = new Job { ProjectId = 1011 };

            //When
            var httpResponse = await _client.PostAsync(JOBURI, ContentHelper.GetStringContent(job));

            //Then
            Assert.Equal(HttpStatusCode.BadRequest, httpResponse.StatusCode);
        }

        [Fact]
        public async void Insert_WithInvalidJson_ShouldReturnBadRequest()
        {
            //Given
            var invalidBody = new StringContent("{\"title\":\"Test\"", Encoding.Default, "application/json");

            //When
            var httpResponse = await _client.PostAsync(JOBURI, invalidBody);

            //Then
            Assert.Equal(HttpStatusCode.BadRequest, httpResponse.StatusCode);
        }

        [Fact]
        public async void GetByProjectId_WithValidProjectId_ShouldReturnAListOfSeededJobs()
        {
            //Given
            //Seeded job exist
            var projectId = 1011;
            var jobUri = JOBURI + "/byprojectId/" + projectId;

            //When
            var httpResponse = await _client.GetAsync(jobUri);

            //Then
            httpResponse.EnsureSuccessStatusCode();
            var jobs = await TestUtilities.GetDeserializedResponseBody<List<Job>>(httpResponse);
            Assert.NotEmpty(jobs);
        }

        [Fact]
        public async void GetByProjectId_WithInvalidProjectId_ShouldReturnEmptyList()
        {
            //Given
            //Seeded job exist
            var projectId = 7777;
            var jobUri = JOBURI + "/byprojectId/" + projectId;

            //When
            var httpResponse = await _client.GetAsync(jobUri);

            //Then
            httpResponse.EnsureSuccessStatusCode();
            var jobs = await TestUtilities.GetDeserializedResponseBody<List<Job>>(httpResponse);
            Assert.Empty(jobs);
        }

        [Fact]
        public async void GetById_WithValidId_ShouldReturnSeededJob()
        {
            //Given
            //Seeded job exist
            var jobId = 1001;
            var jobUri = JOBURI + "/" + jobId;

            //When
            var httpResponse = await _client.GetAsync(jobUri);

            //Then
            httpResponse.EnsureSuccessStatusCode();
            var job = await TestUtilities.GetDeserializedResponseBody<Job>(httpResponse);
            Assert.NotNull(job);
        }

        [Theory]
        [InlineData(-1)]
        [InlineData(0)]
        [InlineData(99999999)]
        public async void GetById_WithInvalidNumericId_ShouldReturnNoContent(int invalidId)
        {
            //Given
            var jobUri = JOBURI + "/" + invalidId;

            //When
            var httpResponse = await _client.GetAsync(jobUri);

            //Then
            Assert.Equal(HttpStatusCode.NoContent, httpResponse.StatusCode);
        }

        [Fact]
        public async void GetById_WithInvalidAlphabeticId_ShouldReturnBadRequest()
        {
            //Given
            var invalidId = "invalidId";
            var jobUri = JOBURI + "/" + invalidId;

            //When
            var httpResponse = await _client.GetAsync(jobUri);

            //Then
            Assert.Equal(HttpStatusCode.BadRequest, httpResponse.StatusCode);
        }

        [Fact]
        public async void Update_WithNewTitle_ShouldReturnUpdatedJob()
        {
            //Given
            //Seeded job exist
            var newJobTitle = "NewJobTitle";
            var jobId = 1002;
            var jobUri = JOBURI + "/" + jobId;

            //When
            var jobResponse = await _client.GetAsync(jobUri);
            var job = await TestUtilities.GetDeserializedResponseBody<Job>(jobResponse);
            job.Title = newJobTitle;

            var httpResponse = await _client.PutAsync(jobUri, ContentHelper.GetStringContent(job));

            //Then
            httpResponse.EnsureSuccessStatusCode();
            var responseBody = await httpResponse.Content.ReadAsStringAsync();
            var updatedJob = JsonConvert.DeserializeObject<Job>(responseBody);
            Assert.Equal(newJobTitle, updatedJob.Title);
        }

        [Fact]
        public async void Update_WithNullTitle_ShouldReturnBadRequest()
        {
            //Given
            //Seeded job exist
            var jobId = 1002;
            var jobUri = JOBURI + "/" + jobId;

            //When
            var jobResponse = await _client.GetAsync(jobUri);
            var job = await TestUtilities.GetDeserializedResponseBody<Job>(jobResponse);
            job.Title = null;

            var httpResponse = await _client.PutAsync(jobUri, ContentHelper.GetStringContent(job));

            //Then
            Assert.Equal(HttpStatusCode.BadRequest, httpResponse.StatusCode);
        }

        [Fact]
        public async void Delete_WithValidId_ShouldReturnOk()
        {
            //Given
            //Seeded job exist
            var jobId = 1003;
            var jobUri = JOBURI + "/" + jobId;

            //When
            var httpResponse = await _client.DeleteAsync(jobUri);

            //Then
            httpResponse.EnsureSuccessStatusCode();
        }

        [Theory]
        [InlineData(-1)]
        [InlineData(0)]
        [InlineData(999999)]
        public async void Delete_WithInvalidId_ShouldReturnBadRequest(int invalidId)
        {
            //Given
            var jobUri = JOBURI + "/" + invalidId;

            //When
            var httpResponse = await _client.DeleteAsync(jobUri);

            //Then
            Assert.Equal(HttpStatusCode.NotFound, httpResponse.StatusCode);
        }

        [Fact]
        public async void FullCRUD()
        {
            //Given
            //Seeded project exist
            var job = new Job
            {
                Title = "TEST_JOB",
                ProjectId = 1012
            };

            //When
            var postHttpResponse = await _client.PostAsync(JOBURI, ContentHelper.GetStringContent(job));
            var newJob = await TestUtilities.GetDeserializedResponseBody<Job>(postHttpResponse);

            var jobUri = JOBURI + "/" + newJob.Id;
            var getHttpResponse = await _client.GetAsync(jobUri);
            getHttpResponse.EnsureSuccessStatusCode();

            newJob.Title = "TEST_JOB_Updated";
            var putHttpResponse = await _client.PutAsync(jobUri, ContentHelper.GetStringContent(newJob));
            putHttpResponse.EnsureSuccessStatusCode();

            var deleteHttpResponse = await _client.DeleteAsync(jobUri);

            //Then
            deleteHttpResponse.EnsureSuccessStatusCode();
        }
    }
}

【问题讨论】：

标签： asp.net-core .net-core integration-testing asp.net-core-webapi xunit

【解决方案1】：

我一直在努力解决类似的问题。尝试了各种方法，例如 classfixtures，每次都为内存数据库名称分配一个新的 guid，以使测试彼此分开等。

最终解决这个问题的方法是将我所有的集成测试放在一个集合中。

[Collection("Our Test Collection #1")]
public class TestClass1
{
 ...
}

[Collection("Our Test Collection #1")]
public class TestClass2
{
 ...
}

这可确保它们不会彼此并行运行。就我而言，它在性能上并没有产生太大的影响，并且解决了失败的测试批次。

（供参考：https://xunit.net/docs/running-tests-in-parallel）

【讨论】：