有效地报告自动测试结果

原链接

https://www.stickyminds.com/article/reporting-automated-test-results-effectively

The modern iterative software development lifecycle has developers checking in code to version control systems frequently, with continuous integration handling building and running automated tests at an almost equally fast rate. Depending on the size of the development team, this can generate an enormous amount of test results data. What should you do with it?

现代迭代软件开发生命周期，使开发人员经常更新代码到版本控制系统，持续集成以几乎同样快的速度，处理构建和运行自动化测试。 根据开发团队的规模，这可以生成大量的测试结果数据。 你应该怎么做？

I have been responsible for automated test infrastructure, including reporting of results, across large teams in the video game industry. I have found that when reported effectively, test results data generated from automation can help teams ship faster; find, isolate and fix bugs faster; prevent regression; and even track performance improvements.

我一直负责视频游戏行业大型团队的自动化测试基础设施，包括结果报告。 我发现，当有效地进行报告时，自动化生成的测试结果数据可以帮助团队更快地迭代; 更快地找到，隔离和修复bug; 防止后退; 甚至跟踪性能改进。

Here’s what you can do to ensure you are reporting results effectively across your team.

以下是您可以采取的措施，以确保您在整个团队中有效地报告结果。

Storage Is Key 存储是关键

Results data should not be ethereal. It’s common for results data to immediately get thrown away after it’s generated, but that is a mistake. This is the first thing to get right: Store all of your test results for at least a couple of weeks, or preferably a month.

结果数据不应该是飘渺的。 结果数据在生成后立即被抛弃是很常见的，但这是一个错误。 这是第一件要做的事：将所有测试结果存储至少几周，或者最好是一个月。

Looking at the results of a single test run in isolation is better than nothing, but keeping a history of all your test runs can help you perform comparisons in order to track progress, identify regression, and even find flaky tests, which can help uncover issues that are not always highly reproducible, such as memory allocation issues or race conditions.

单独查看单个测试的运行结果总比没有好，但保留所有测试运行的历史记录，可以帮助您执行比较以跟踪进度，识别回归，甚至找到不稳定测试，这可以帮助发现问题，这些问题并不总是高度可重复的，例如，内存分配问题或竞争条件。

Most importantly, storing results data allows analytics to be carried out on the data. You can find out what changes were responsible for introducing a failure. By storing the latest revision number or build number with a test run, you can have a developer narrow down exactly what changes introduced an issue. I have seen developers save hours of their time thanks to this. You can also quickly find out what change fixed an issue and separate new failures from existing failures.

最重要的是，存储结果数据可以对数据进行分析。 您可以找出导致故障的更改。 通过在测试运行中存储最新版本号或内部版本号，您可以让开发人员精确地缩小什么更改引起了问题。 由于这个原因，我看到开发人员节省了数小时的时间。 您还可以快速找出什么更改修复了一个问题，并将新故障与现有故障分开。

Historical results trends can even help improve process. I had a team leader decide to change a process after having a look at our results trends for one team that showed patterns of result dips after check-ins later at certain times of the week.

历史结果趋势甚至可以帮助改善过程。 在看了一个团队的结果趋势之后，我有一个团队负责人决定改变一个过程，该团队在一周的某些时间稍后检查后显示结果下降了。

More Than Pass or Fail Results 不仅仅是成功或失败的结果

Files and artifacts generated by automated tests, such as logs, screen captures, and performance data files, should also be stored with the results data. Together, all of this data should be easily accessible by everyone on your team, from the last test run to this time three days ago. Too often the only way to find data is for a build engineer to remote into a server somewhere. Do not make your team rely on only one or two people to get at this information; it should be available to everyone.

自动化测试生成的文件和工件（如日志，屏幕捕获和性能数据文件）也应与结果数据一起存储。 从最后一次测试到三天前的这个时间，团队中的每个人都可以轻松访问所有这些数据。 通常，查找数据的唯一方法是让构建工程师远程进入某个服务器。 不要让你的团队只依靠一或两个人来获取这些信息; 它应该对每个人都能获取到的。

Logs should be detailed enough to provide precise information for developers to fix issues faster. Ensuring call stack information and verbose details of step-by-step execution are posted is an important factor. This way, even system-level and integration tests can highlight the exact source of a problem. This is critical if your team has a low number of unit tests (or no unit tests at all).

日志应足够详细，以便为开发人员提供更准确的信息，以便更快地解决问题。 确保调用堆栈信息和逐步执行的详细信息是一个重要因素。 这样，即使系统级和集成测试也可以突出显示问题的确切来源。 如果您的团队的单元测试数量较少（或根本没有单元测试），这一点至关重要。

Often, teams use catch-all integration or end-to-end tests to get more value from less time-intensive test authoring. Solid reporting of where these tests failed is absolutely critical to make these test failures actionable. Without it, a developer would need to go through a long process to reproduce the issue.

通常，团队使用全面集成或端到端测试，来从较少时间密集的测试创作中获得更多价值。 牢固地报告这些测试失败的位置，对于使这些测试失败容易解决是非常关键的。 没有它，开发人员需要经历一个漫长的过程来重现这个问题。

If your tests store performance data, you can plot trends to see how things are progressing over time. This is one of the most useful bonus outcomes of automated tests and storage of results data.

如果您的测试存储了性能数据，您可以绘制趋势图以了解事物的进展情况。 这是自动测试和结果数据存储最有用的奖励结果之一。

The Hidden Benefit from Visible Data 可见数据的隐藏好处

Making automated test results more visibly prominent has an interesting hidden benefit. You will not realize until you actually do it that something special starts happening: Reporting automated tests effectively has a cyclical effect of making your automated tests better, increasing coverage, rapidly increasing quality, and driving the writing of more and better tests.

使自动化测试结果更加明显突出，有一个有趣的隐藏优势。 直到你真正做到，特别事情开始发生，你才会意识到：有效报告自动化测试具有周期性的效果，可以使你的自动化测试更好，增加覆盖范围，快速提高质量，并推动编写更多更好的测试。

I did not see this coming when I was implementing reporting for my team a few years ago. I just needed a way to report results, so I put something together to do it. Suddenly there was a massive interest in the automated test results for the first time throughout the development team, especially leads.

几年前，当我为我的团队实施报告时，我没有看到这一点。 我只需要一种方法来报告结果，所以我把一些东西放在一起来做。 突然之间，整个开发团队首次对自动化测试结果产生了浓厚的兴趣，尤其是领导。

These tests had existed for a while; I’d just made them more visible, so now people could see how often tests were running, what they were testing, and what they were not. There was an uptake in the number of new tests being written, existing tests were improved to make them more reliable, and management and leads even started to get more invested in automation and asked for more of it. Actioning of failures was made a priority. Getting to a 100 percent pass rate across all branches and build flavors—and maintaining it—became something of a sport in our team.

这些测试已经存在了一段时间; 我只是让它们更加明显，所以现在人们可以看到测试运行的频率，测试的内容以及没有测试的内容。 正在编写的新测试数量有所增加，现有测试得到了改进，使其更加可靠，管理和领导甚至开始在自动化方面投入更多资源并要求更多。 对失败采取行动是一个优先事项。 在所有分支和构建场景中实现100％的通过率并维护它 - 在我们的团队中成为了一项类似运动的事情。

I have reflected on why this sudden change in culture occurred, when there had been a general apathy toward automated tests and continuous integration across the wider team before. I think it came down to three reasons:

我已经反思了为什么文化突然发生了变化，何时对自动化测试以及之前对更广泛的团队的持续整合有了普遍关注。 我认为这归结为三个原因：

• Visibility increases mindshare 可见性增加了思想共享
• There is greater clarity around return on investment 投资回报率更加清晰
• People like to be rewarded for their hard work人们喜欢因努力工作而获得奖励

We started putting our results on an office TV screen, and they were also made accessible to everyone via their browsers. While the same tests had existed before, they has been written to a console log with files in a temp directory on a build server somewhere that no one other than the build engineer could access. They were out of sight and out of mind, so no one understood the importance and value of the automation.

我们开始将我们的结果放在办公室电视屏幕上，并且每个人都可以通过他们的浏览器访问它们。 虽然以前存在相同的测试，但它们已被写入控制台日志，其中的文件位于构建服务器上的临时目录中，除了构建工程师之外，其他任何人都无法访问。 它们不在视线之内，因此没有人理解自动化的重要性和价值。

The return on investment was also unclear without visibility. Now we could see how many bugs we were finding, and we discovered that with better reporting, we could quickly identify and narrow down the causes and fixes. This helped improve reliability of tests, too, because constantly seeing unreliable tests change the pass rate was no longer tenable with the results right in front of our eyes; it made a mockery of our testing.

没有可视化，投资回报也不明朗。 现在我们可以看到我们发现了多少bug，我们发现通过更好的报告，可以快速识别并缩小原因和更快修复它们。 这也有助于提高测试的可靠性，因为经常看到不可靠的测试会改变通过率，因为结果就在我们眼前; 它嘲弄了我们的测试。

Lastly, feature code work is obviously rewarding; you ship a product that people use, and that is a highly visible reward. Writing automated tests, however, did not have a visible reward before. No one really knew about them or how important they were. Solid reporting changed that so it did become rewarding, because now everyone could see the tests.

最后，功能代码工作显然是有益的; 您发布人们使用的产品，这是一个非常明显的奖励。 然而，编写自动化测试之前没有明显的奖励。 没有人真正了解他们或他们有多重要。 可靠的报告使其发生了改变，所以它确实变得有意义，因为现在每个人都可以看到测试。

Here’s another tip: Try making the test author name visible next to the test too, along with whoever fixed the test after it broke. Then people can see who was writing the tests, who was fixing them, and every time bugs were discovered and prevented us from shipping in a deteriorated state. This credits the test author and whoever fixed the test.

这是另一个提示：尝试在测试旁边显示测试作者姓名，以及在测试完成后修复测试的任何人。 然后人们可以看到谁在编写测试，谁正在修复它们，每次发现bug并阻止我们在恶化的状态下发布。 这归功于测试作者和修复这个测试的人员。

Transform your team’s automated test workflow today with effective reporting. You will find you ship higher quality releases faster.

通过有效的报告，改变您团队的自动化测试工作流程。 您会发现您可以更快地发布更高质量的版本。