AWS Device Farm 忽略 TestNG 优先级注释

Question

我创建了一个测试套件类，其中setUp() 和tearDown() 方法在@BeforeClass 和@AfterClass 注释下声明。

我在这个类中包含了 6 个测试用例，每个测试用例都有@Test(priority = #)。我在本地机器上执行这个类，它工作正常，但是当我将它上传到 AWS Device Farm 上时，它只执行一个 (priority=1) 的测试用例，而对于所有其他测试用例，设备场初始化 appium 服务器。

如何克服这个障碍？

Answer 1

您使用的是自定义环境吗？在标准环境中可以预料到这种限制。

https://aws.amazon.com/premiumsupport/knowledge-center/xml-file-tests-jar-file-device-farm/

注意：在 Device Farm 的标准环境中，testng.xml 文件仅支持部分功能。如果项目需要优先级、包含标记、排除标记、复杂分组或使用 testng.xml 文件中的参数，则使用自定义环境。

[编辑]

我使用示例应用和测试测试了优先级

git clone https://github.com/aws-samples/aws-device-farm-appium-tests-for-sample-app.git
cd aws-device-farm-appium-tests-for-sample-app/
mkdir ./src/test/resources
// I used VS code here but any text editor will work
code ./src/test/resources/testng.xml

然后我在上面的支持链接中添加了一些代码，这样我就不必运行所有的测试了：

testng.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Default Suite">
    <test name="test">
        <classes>
            <class name="Tests.LoginTest"/> <!--Package.ClassName-->
        </classes>
    </test>
</suite>

然后我修改了 LoginTest 以使用 TestNG 的priority feature。

LoginTest.java

/*
 * Copyright 2014-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
 *
 * Licensed under the Apache License, Version 2.0 (the "License").
 * You may not use this file except in compliance with the License.
 * A copy of the License is located at
 *
 * http://aws.amazon.com/apache2.0
 *
 * or in the "license" file accompanying this file. This file is distributed
 * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
 * express or implied. See the License for the specific language governing
 * permissions and limitations under the License.
 */

package Tests;

import Pages.LoginPage;
import Tests.AbstractBaseTests.TestBase;
import org.testng.Assert;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeTest;
import org.testng.annotations.Test;

/**
 * Tests for a login page
 */
public class LoginTest extends TestBase {
    private static final String LOGIN_SUCCESS_MESSAGE = "You are logged on as admin";
    private static final String LOGIN_FAIL_MESSAGE = "You gave me the wrong username and password";
    private static final String CORRECT_USER_NAME = "admin";
    private static final String CORRECT_PASSWORD = "password";
    private static final String FAIL_USER_NAME = "Wrong User";
    private static final String FAIL_PASSWORD = "12345";
    private static final String BAD_TEXT_ENTRY_MSG = "Username sent to text field incorrectly";

    private LoginPage loginPage;

    @Override
    public String getName() {
        return "Login Page";
    }

    /**
     * Creates a login
     */
    @BeforeTest
    @Override
    public void setUpPage() {
        loginPage = new LoginPage(driver);
    }

    /**
     * Tests logging in with valid credentials by verifying if the login message is correct
     */
    @Test(priority = 1)
    public void loginSuccess() throws InterruptedException {
        Assert.assertTrue(loginPage.login(CORRECT_USER_NAME, CORRECT_PASSWORD), BAD_TEXT_ENTRY_MSG);
        Assert.assertEquals(loginPage.getMessage(), LOGIN_SUCCESS_MESSAGE);
    }

    /**
     * Tests logging in with invalid credentials by verifying if the error message is correct
     */
    @Test(priority = 2)
    public void loginFail() throws InterruptedException {
        Assert.assertTrue(loginPage.login(FAIL_USER_NAME, FAIL_PASSWORD), BAD_TEXT_ENTRY_MSG);
        Assert.assertEquals(loginPage.getMessage(), LOGIN_FAIL_MESSAGE);
    }

    /**
     * After each test method, logout or try again
     */
    @AfterMethod
    public void logOut() {
        loginPage.pressAltButton();
        Assert.assertTrue(loginPage.checkIfBackAtLogin());
    }
}

我使用命令mvn clean package -DskipTests=true打包了测试

在设备场中运行它产生了以下结果：

视频：https://imgur.com/YHnq08X

所以我无法重现该问题。您能否提供 Device Farm 的自定义环境未能优先执行的证据？