统一社会信用代码测试_统一的测试运行器和测试分析

统一社会信用代码测试

嗨，我是Yan，在过去的两年中，我一直是Unity的一名工具匠。 最近我们增长了很多，我们的测试套件以及不稳定测试，慢速测试和失败的数量也无法在本地重现。 在这篇博客文章中，我将讨论我们正在做些什么，但是首先让我向您介绍一下我们的自动化环境-只是为了让您更好地了解我们正在处理的挑战。 (Hi, I’m Yan and for the past two years I’ve been a Toolsmith at Unity. We have grown quite a lot recently and so have our test suites and the number of unstable tests, slow tests and failures which can not be reproduced locally. In this blog post, I’ll talk about what we’re doing about it, but first let me tell you a little about our automation environment – just to give you a better understanding of what challenges we are dealing with.)

At Unity, we have many different kinds of test frameworks (Figure 1) and test suites:

在Unity，我们有许多不同种类的测试框架(图1)和测试套件：

• Runtime Tests are verifying Unity’s public runtime API on all of Unity’s supported platforms.

  运行时测试 正在验证 所有Unity支持的平台 上 的Unity 公共运行时API 。

• Integration Tests allow testing things that are not easily expressed as runtime tests – they can test Unity Editor features as well as integration to components like the Cache Server and Bug Reporter.

  集成测试允许测试不容易表示为运行时测试的事物–它们可以测试Unity编辑器功能以及与诸如Cache Server和Bug Reporter之类的组件的集成。

• Native C++ Tests are focused on testing native code directly without going through the scripting layer.

  本机C ++测试专注于直接测试本机代码，而无需通过脚本层。

• Graphics Tests are testing rendering features by comparing a resulting image with a reference image, which is considered “correct”.

  图形测试通过将生成的图像与参考图像(被认为是“正确的”)进行比较来测试渲染功能。

• Many others (Performance Tests, Load Tests, IMGUI Tests, etc.).

  其他许多(性能测试，负载测试，IMGUI测试等)。

Figure 1. Testing frameworks at Unity

图1. Unity的测试框架

On the highest level, all tests are grouped in different subsets based on test framework. However, they are further divided based on platform, run frequency, execution time and some other criterias. Those divisions produce an enormous amount of testing points. We’ll discuss these numbers later on.

在*别上，所有测试均基于测试框架分为不同的子集。 但是，它们根据平台，运行频率，执行时间和其他一些条件进一步划分。 这些部门产生了大量的测试点。 我们稍后将讨论这些数字。

Having so many frameworks and runners is not easy, so about a year ago we started working on a Unified Test Runner (UTR): a single entry point for running all tests. It serves as a facade (see Figure 2) for all testing runners/frameworks. This enables anyone to run any of our tests suites from command line.

拥有如此多的框架和运行器并不容易，因此大约一年前，我们开始研究统一测试运行器(UTR)：运行所有测试的单个入口点。 它充当所有测试运行器/框架的立面(参见图2)。 这使任何人都可以从命令行运行我们的任何测试套件。

Figure 2. Unified Test Runner Facade

图2.统一测试运行器外观

All the artifacts that are produced by a test run are copied into the same place and are grouped and organized according to the same conventions everywhere. UTR also provides other services:

测试运行产生的所有工件都被复制到同一位置，并根据各地相同的约定进行分组和组织。 UTR还提供其他服务：

• tests can be filtered the same way everywhere with -testfilter=TestName

  可以在任何地方使用 -testfilter = TestName 以相同的方式过滤测试

• execution progress is reported the same way for all the test suites

  所有测试套件的执行进度都以相同的方式报告

Initially, UTR was mostly used to run tests locally. Then we switched focus to our Build Farm configurations. We wanted to use the Unified Test Runner there as well. Our goal was to run tests the same way locally and on the build farm. Or in other words: if something failed on the Build Farm – it should be easy to reproduce it locally.

最初，UTR主要用于在本地运行测试。 然后，我们将重点转移到 了构建场 配置。 我们也想在那里使用Unified Test Runner。 我们的目标是在本地和构建场上以相同的方式运行测试。 换句话说：如果Build Farm发生故障，则应该很容易在本地复制它。

Slowly but surely UTR has become the single entry point which we are using to run tests in Unity. That’s what made it a perfect candidate for another task: collecting test execution data, both from local and Build Farm test runs. Whenever a test run is finished, UTR reports data to the Web service. That is how our test data analytics solution, Hoarder, was born. Hoarder’s responsibility is to collect, store and provide access to test execution data. It can present aggregated statistics with a possibility to drill down to the individual test runs. See Figure 3.

但可以肯定的是，UTR逐渐成为我们用于在Unity中运行测试的唯一入口点。 这就是使其成为另一项任务的理想人选：从本地和Build Farm测试运行中收集测试执行数据的过程。 每当测试运行完成时，UTR都会向 Web服务 报告数据 。 这就是我们的测试数据分析解决方案Hoarder诞生的方式。 Hoarder的责任是收集，存储并提供对测试执行数据的访问。 它可以提供汇总的统计信息，并可能深入到各个测试运行。 参见图3。

Figure 3. Build agents and humans submit data to Hoarder Web Service. Analytics application fetching it.

图3.构建代理和人员向Hoarder Web Service提交数据。 Analytics应用程序正在获取它。

We discovered a lot of interesting things in the data, which led to a few important decisions. I’m going to talk about how we make informed decisions based on this data in the next blog post.

我们在数据中发现了很多有趣的东西，这导致了一些重要的决定。 我将在下一篇博客文章中讨论如何根据这些数据做出明智的决定。

翻译自: https://blogs.unity3d.com/2015/10/07/unified-test-runner-test-analytics/

统一社会信用代码测试