unity 路径生成模型
Synthetic data alleviates the challenge of acquiring labeled data needed to train machine learning models. In this post, the second in our blog series on synthetic data, we will introduce tools from Unity to generate and analyze synthetic datasets with an illustrative example of object detection.
合成数据减轻了获取训练机器学习模型所需的标记数据的挑战。 在这篇关于合成数据的博客系列的第二篇文章中，我们将介绍Unity的工具，以生成和分析合成数据集，并提供一个示例性的对象检测示例。
In our first blog post, we discussed the challenges of gathering a large volume of labeled images to train machine learning models for computer vision tasks. We also discussed the state of the art research from the likes of Google Cloud AI and OpenAI that demonstrated the efficacy of synthetic data for tasks such as object detection.
在我们的 第一篇博客文章中 ，我们讨论了收集大量标记图像以训练计算机视觉任务的机器学习模型所面临的挑战。 我们还讨论了来自Google Cloud AI和OpenAI之类的最新研究，这些研究证明了合成数据对诸如对象检测之类的任务的功效。
However, there are many intermediate steps between getting started with synthetic data and creating a dataset ready to train an ML model. In this process, developers often encounter similar problems and are compelled to write custom, one-off solutions that often do not yield the quality of data necessary to train a machine learning model. Today, we are introducing two new tools: Unity Perception Package and Dataset Insights, which remove a number of these redundant steps, making it easy to generate high-quality synthetic datasets as well as analyze them.
但是，在开始使用合成数据和创建准备训练ML模型的数据集之间，有许多中间步骤。 在此过程中，开发人员经常会遇到类似的问题，并且不得不编写自定义的一次性解决方案，而这些解决方案通常无法获得训练机器学习模型所需的数据质量。 今天，我们将引入两个新工具：Unity Perception Package和Dataset Insights，它们删除了许多这些多余的步骤，从而可以轻松生成高质量的合成数据集并进行分析。
使用Unity Perception工具加速合成数据创建 (Accelerating synthetic data creation with Unity Perception tools)
Unity感知包 (Unity Perception Package)
RGB, 2D bounding boxes, and metrics captured by Unity Perception package
Unity Perception软件包捕获的RGB，2D边界框和度量
The Unity Perception package enables a new workflow in Unity for generating synthetic datasets and supports both Universal and High Definition Render Pipelines. In this first release, it provides tools for dataset capture and consists of 4 primary features: object labeling, labelers, image capture, and custom metrics. The package provides a simple interface to input object-label associations, that are picked up automatically and fed to the labelers. A labeler uses this object information to generate ground truth data such as 2D bounding boxes or semantic segmentation masks. The produced ground truth is then captured along with associated metrics in JSON files.
Unity Perception 软件包在Unity中启用了一个新的工作流，用于生成合成数据集，并支持 通用 和 高清晰度渲染管线 。 在此第一个版本中，它提供了用于数据集捕获的工具，并包含4个主要功能：对象标记，标签，图像捕获和自定义指标。 该程序包提供了一个简单的界面来输入对象-标签关联，这些关联会自动被拾取并馈入标签。 贴标机使用此对象信息来生成地面真实数据，例如2D边界框或语义分割蒙版。 然后，将生成的基本事实以及关联的指标捕获到JSON文件中。
In future releases, we plan to add more labelers such as instance segmentation to support other common computer vision tasks, tools for scene generation, the ability to configure and manage large sets of parameters for domain randomization, and scalability in the cloud.
在将来的版本中，我们计划添加更多标签，例如支持其他常见计算机视觉任务的实例分割，场景生成工具，为 域随机化 配置和管理大量参数的能力 以及在云中的可伸缩性。
数据集见解 (Dataset Insights)
Exploration and analysis of labeled data is critical for any ML practitioner. When working with synthetic data, the dataset size can become large very quickly due to the ability to generate millions of images with cloud-based simulation runs. With Dataset Insights, a Python package, we have made the process of computing statistics and generating insights from large synthetic datasets simple and efficient. It can consume the metrics, exported per frame locally or on our managed cloud service, and visualize statistics aggregated over the entire dataset.
对标记数据的探索和分析对于任何机器学习从业者都是至关重要的。 当使用合成数据时，由于能够通过基于云的模拟运行生成数百万个图像，因此数据集的大小会很快变得很大。 借助 Python软件包 Dataset Insights ，我们使统计数据的计算过程以及从大型综合数据集生成见解的过程变得简单而高效。 它可以使用指标，在本地或在我们的托管云服务上每帧导出指标，并可视化整个数据集上汇总的统计信息。
In the next section, we will describe how we used the Unity Perception Package and Dataset Insights to create synthetic datasets-for the purpose of training an object detection model that detects and labels a set of grocery products. The tools are designed to be general and extensible to other environments and computer vision tasks in the future, with the long term goal of enabling more ML practitioners to adopt synthetic data and solve diverse problems.
在下一节中，我们将描述如何使用Unity Perception Package和Dataset Insights创建综合数据集，目的是训练对象检测模型来检测和标记一组杂货产品。 这些工具被设计为通用的，并且将来可以扩展到其他环境和计算机视觉任务，其长期目标是使更多的ML实践者可以采用综合数据并解决各种问题。
3D资产创建 (3D asset creation)
Recent research from Google Cloud AI uses 64 grocery products easily available in stores such as cereal boxes, paper towels, etc. to demonstrate the efficacy of object detection models trained purely on synthetic data. Inspired by this research, we chose an equal number of products that were either the same or close approximations of the original products in size, shape, and textural diversity.
Google Cloud AI的 最新 研究 使用了64种杂货产品，这些产品很容易在商店买到，例如谷物盒，纸巾等，以证明纯粹基于合成数据训练的对象检测模型的功效。 受这项研究的启发，我们选择了相同数量的产品，这些产品在尺寸，形状和纹理多样性上与原始产品相同或近似。
We created a library of 3D assets of selected grocery products using Digital Content Creation (DCC) tools, scanned labels, and photogrammetry. Additionally, we created background and occluding assets using real world imagery mapped onto simple primitives such as cubes, spheres, and cylinders. All of the grocery products used custom shaders created in the Unity Editor using Shadergraph, in the Universal Rendering Pipeline.
我们使用数字内容创建(DCC)工具，扫描标签和摄影测量法创建了选定杂货产品的3D资产库。 此外，我们使用现实世界图像创建了背景和遮挡资产，这些图像被映射到诸如立方体，球体和圆柱体的简单图元上。 所有杂货产品都使用 在通用 编辑 管道中 使用 Shadergraph 在Unity Editor中创建的自定义着色器 。
场景创建 (Scene creation)
We defined the behaviors for placement of the 3D assets along with the background assets and other distractions in shape and texture to add complexity. Adding varied backgrounds helps ML models trained on this dataset to deal with a wide variety of backgrounds that can be encountered in the real world.
我们定义了3D资产的放置行为以及背景资产以及其他形状和纹理干扰，以增加复杂性。 添加各种背景有助于在此数据集上训练的ML模型处理现实世界中可能遇到的各种背景。
Scene view in the Unity Editor showcasing the placement of various assets. From left, occluding objects, 3D assets for grocery products, and background assets
Unity Editor中的场景视图展示了各种资产的放置。 从左开始，遮挡对象，杂货产品的3D资产和背景资产
For each render loop, a new random placement of foreground, background, and occluding objects along with a randomization of lighting, object hue, blur and noise is generated. As shown below, the Perception package captures RGB images, object bounding boxes and other randomization parameters for each image in the dataset.
对于每个渲染循环，都会生成前景，背景和遮挡对象的新随机放置，以及照明，对象色调，模糊和噪声的随机化。 如下所示，Perception包捕获数据集中每个图像的RGB图像，对象边界框和其他随机化参数。
A sample annotation output generated by the Unity Perception package
Unity Perception包生成的示例注释输出
A labeled image sampled from the synthetic dataset
从合成数据集中采样的带标签的图像
使用Dataset Insights进行探索性数据分析 (Using Dataset Insights for exploratory data analysis)
For large synthetic datasets, it is impossible to inspect the images manually and detect anomalies such as: bias, missing objects or artifacts, unintended patterns in placement, pose, etc. that may subsequently result in sub-optimal performance of the ML model.
对于大型合成数据集，无法手动检查图像并检测异常，例如：偏差，丢失的对象或伪影，放置中的意外图案，姿势等，这些异常可能随后导致ML模型表现欠佳。
In this case, the dataset of objects generated previously was input to the Dataset Insights Python package for computing statistics and subsequently training the ML model. These insights proved to be very effective in ensuring that our image data was useful for the purpose of training an object detection model. Some examples of summary statistics that were generated for the dataset are shown below.
在这种情况下，先前生成的对象的数据集被输入到Dataset Insights Python包中，用于计算统计信息，然后训练ML模型。 这些见解被证明在确保我们的图像数据对训练对象检测模型有用方面非常有效。 下面显示了为数据集生成的摘要统计信息的一些示例。
Chart from Dataset Insights showing uniform distribution of objects in the dataset. This helps us confirm that no single product is too popular or ignored by the model.
Dataset Insights中的图表显示了数据集中对象的均匀分布。 这有助于我们确认没有哪个产品太受欢迎或被模型忽略。
During our own testing, we had encountered an anomaly wherein some of the objects appeared multiple times in a frame. This became apparent from the chart visualizing the object count across image frames. We were then able to quickly fix the issue and ensure that our objects of interest were uniformly distributed in the dataset, thereby ensuring an equal likelihood of detection.
在我们自己的测试中，我们遇到了一个异常，其中某些对象在一帧中多次出现。 从图表中可视化跨图像帧的对象计数，这一点变得显而易见。 然后，我们能够快速解决问题，并确保我们感兴趣的对象在数据集中均匀分布，从而确保了检测的可能性均等。
Distribution of object counts per example in the dataset.
数据集中每个示例的对象计数分布。
Although we want a uniform distribution of objects across the dataset, we expect the ML model to be able to detect multiple objects, whether the image has few or several objects of interest. The chart above shows that the number of labeled objects in each frame of the generated dataset follows a familiar normal distribution centered around seven objects per frame.
尽管我们希望对象在整个数据集中均匀分布，但我们希望ML模型能够检测到多个对象，无论图像中只有几个或几个感兴趣的对象。 上表显示，在生成的数据集的每一帧中，标记对象的数量遵循熟悉的正态分布，中心分布在每帧七个对象的周围。
Visualization of different light source positions. Each point represents the position of the light source relative to the objects of interest.To represent real world lighting conditions, we also varied the direction and color of the light in the scene. As shown in the plot above, the light source was placed at various locations, ensuring that the images captured across the dataset have diverse lighting conditions and corresponding shadows.
可视化不同光源的位置。 每个点代表光源相对于感兴趣对象的位置。 为了表示现实世界的照明条件，我们还更改了场景中灯光的方向和颜色。 如上图所示，将光源放置在各个位置，以确保跨数据集捕获的图像具有不同的光照条件和相应的阴影。
使用Unity Simulation缩放合成数据集 (Scaling synthetic datasets with Unity Simulation)
In order to run simulations with a large number of possible permutations, we used Unity Simulation, our managed cloud service that can execute Unity projects and generate a complete dataset needed to train a modern computer vision model.
为了运行具有大量可能排列的模拟，我们使用了Unity Simulation，这是我们的托管云服务，可以执行Unity项目并生成训练现代计算机视觉模型所需的完整数据集。
In the last installment of this blog series, we will share more on generating a large-scale dataset with Unity Simulation, training a machine learning model with synthetic data, and evaluating it on real data, the economics of synthetic data, and our key takeaways from this exercise.
在本系列博客的最后一部分中，我们将分享更多有关使用Unity Simulation生成大型数据集，使用综合数据训练机器学习模型，以及对真实数据，综合数据的经济性以及我们的主要要点进行评估的更多信息。从这个练习。
下一步 (Next Steps)
Get started with perception tools for free. Check out our tools and object detection example on Github.
免费开始使用感知工具。 在Github上 查看我们的 工具和对象检测示例。
Looking to scale-up your synthetic datasets? Sign up for our new service: Unity Simulation (Beta).
想要扩大您的综合数据集？ 注册我们的新服务： Unity Simulation(Beta) 。
We would love to hear from you – leave a comment below, or contact us at [email protected] for questions/feedback.
我们很高兴收到您的来信-在下面发表评论，或发送 电子邮件至 [email protected] 与我们联系， 以提出问题/反馈。
* All trademarks are the property of their respective owners
* 所有商标均为其各自所有者的财产
unity 路径生成模型