ITADN · 发现全球开源项目

版本发布 8

PaddlePaddle 3.2.0 Release Notev3.2.0

? · 2025-09-08

- [English Version](https://github.com/PaddlePaddle/Paddle/wiki/PaddlePaddle-3.2.0-Release-Note-EN) # 重要更新飞桨框架3.2版本在大模型训练推理性能、硬件适配、主流大模型及高性能加速库的支持上进一步提升。 - 大模型训练方面，飞桨框架在计算、并行策略、容错能力三方面进行了升级： - 从基础计算性能层面，提出了存算重叠的稀疏掩码注意力计算FlashMask V3，极致优化Attention的计算效率，同时还实现了高效的FP8混合精度效果无损训练技术。 - 在分布式并行策略层面，提出了动态自适应的显存卸载策略，实现存算最优均衡，再结合创新设计的显存友好的流水线并行调度，进一步降低显存开销。 - 增强了框架原生的容错能力，实现了大规模集群训练容错系统，可在不影响训练效率的前提下在线监测静默数据损坏等难以察觉的故障，并实现了高可用的检查点容灾方法，降低中断恢复损失。 - 在硬件适配方面，面向类CUDA芯片，全面升级插件式适配方案。 - 在设备资源的管理调度和高性能集合通讯库方面，针对类CUDA芯片做了管理接口升级和通信能力的增强，特别增强了分布式通信能力，使XCCL对齐NCCL的各结构体和功能。 - 新增了类CUDA算子注册机制。以沐曦适配为例，在复用GPU算子内核的基础上，仅需一行代码即可完成算子内核注册。经过统计计算，算子内核的复用率最高可以达到92%，可大幅降低硬件适配成本。 - 使用体验方面，重点提升了兼容能力，包括开发接口兼容业界用法、safetensors模型格式兼容、和第三方高性能加速库的兼容。 - 新增和修改开发接口兼容业界用法，新增系列API和别名，新增参数别名，新增专有和通用的参数。 - 全面兼容 Safetensors 模型格式。新增 FlexCheckpoint 机制，支持跨分布式策略、跨模型结构间自动实现参数重切分，可显著降低权重转换成本，进而提升大模型端到端的训练与推理研发效率。 - 系统性增强了接口兼容与算子注册能力，实现了高性能加速库一键导入，无需修改代码直接复用于飞桨的模型训练与推理加速过程中。 ## 1. 用户体验 ### 新特性 - 新增API：`paddle.msort`、`paddle.ravel`、`paddle.nn.functional.dropout1d`、`paddle.Tensor.type_as`、`paddle.Tensor.requires_grad`、`paddle.view_as_complex`、`paddle.view_as_real`、`paddle.nn.Parameter`、`paddle.broadcast_shapes`、`paddle.range`、`paddle.as_tensor`、`paddle.scatter_reduce/scatter_reduce_`、`paddle.scatter_add`、`paddle.tensor`、`paddle.softmax`、`paddle.Tensor.softmax`、`paddle.rand_like`、`paddle.is_autocast_enabled`、`paddle.get_autocast_gpu_dtype`、`paddle.Tensor.repeat`、`paddle.permute`。[#74421](https://github.com/PaddlePaddle/Paddle/pull/74421),[#74439](https://github.com/PaddlePaddle/Paddle/pull/74439),[#74444](https://github.com/PaddlePaddle/Paddle/pull/74444),[#74454](https://github.com/PaddlePaddle/Paddle/pull/74454),[#74459](https://github.com/PaddlePaddle/Paddle/pull/74459),[#74491](https://github.com/PaddlePaddle/Paddle/pull/74491)、[#74466](https://github.com/PaddlePaddle/Paddle/pull/74466),[#74438](https://github.com/PaddlePaddle/Paddle/pull/74438),[#74594](https://github.com/PaddlePaddle/Paddle/pull/74594),[#74542](https://github.com/PaddlePaddle/Paddle/pull/74542),[#74694](https://github.com/PaddlePaddle/Paddle/pull/74694),[#74564](https://github.com/PaddlePaddle/Paddle/pull/74564),[#74540](https://github.com/PaddlePaddle/Paddle/pull/74540),[#74586](https://github.com/PaddlePaddle/Paddle/pull/74586),[#74651](https://github.com/PaddlePaddle/Paddle/pull/74651),[#74807](https://github.com/PaddlePaddle/Paddle/pull/74807),[#74632](https://github.com/PaddlePaddle/Paddle/pull/74632),[#74834](https://github.com/PaddlePaddle/Paddle/pull/74834),[#74952](https://github.com/PaddlePaddle/Paddle/pull/74952),[#74772](https://github.com/PaddlePaddle/Paddle/pull/74772),[#74441](https://github.com/PaddlePaddle/Paddle/pull/74441),[#74561](https://github.com/PaddlePaddle/Paddle/pull/74561),[#74525](https://github.com/PaddlePaddle/Paddle/pull/74525) - 新增`paddle.compat.*`一系列API，支持业界的通用用法，便于迁移代码，包括 `paddle.compat.median`、`paddle.compat.nanmedian`、`paddle.compat.softmax`、`paddle.compat.sort`、`paddle.compat.split`、`paddle.compat.min/max`、`paddle.compat.Unfold`。[#74865](https://github.com/PaddlePaddle/Paddle/pull/74865),[#74874](https://github.com/PaddlePaddle/Paddle/pull/74874) - 新增初始化一系列API，支持业界通用的参数初始化方式，包括`paddle.nn.init.kaiming_uniform_`、`paddle.nn.init.xavier_uniform_`、`paddle.nn.init.uniform_`、`paddle.nn.init.kaiming_normal_`、`paddle.nn.init.xavier_normal_`、`paddle.nn.init.normal_`、`paddle.nn.init.calculate_gain`、`paddle.nn.init.constant_`、`paddle.nn.init.dirac_`、`paddle.nn.init.eye_`、`paddle.nn.init.ones_`、`paddle.nn.init.orthogonal_`、`paddle.nn.init.trunc_normal_`、`paddle.nn.init.zeros_`。[#74478](https://github.com/PaddlePaddle/Paddle/pull/74478) - API新增参数别名用法，例如既可以输入`x`，也可以输入`input`，用法更为灵活。包括 `paddle.maximum`、`paddle.minimum`、`paddle.sqrt`、`paddle.topk`、`paddle.polar`、`paddle.stack`、`paddle.cos`、`paddle.floor`、`paddle.log`、`paddle.pow`、`paddle.rsqrt`、`paddle.sign`、`paddle.sin`、`paddle.multiply`、`paddle.where`等。[#74683](https://github.com/PaddlePaddle/Paddle/pull/74683),[#74795](https://github.com/PaddlePaddle/Paddle/pull/74795),[#74887](https://github.com/PaddlePaddle/Paddle/pull/74887),[#74592](https://github.com/PaddlePaddle/Paddle/pull/74592) - `paddle.Tensor`新增支持多种初始化方式，支持灵活的创建Tensor。[#74619](https://github.com/PaddlePaddle/Paddle/pull/74619),[#75022](https://github.com/PaddlePaddle/Paddle/pull/75022),[#75065](https://github.com/PaddlePaddle/Paddle/pull/75065) - API新增一些专有参数，增强原有功能。包括 `paddle.nn.functional.gelu`、`paddle.divide/div/div_`、`paddle.add`、`paddle.Tensor.copy_`、`paddle.norm`、`paddle.linalg.norm`、`paddle.nn.functional.silu`、`paddle.repeat_interleave`。[#74485](https://github.com/PaddlePaddle/Paddle/pull/74485),[#74562](https://github.com/PaddlePaddle/Paddle/pull/74562),[#74420](https://github.com/PaddlePaddle/Paddle/pull/74420),[#74768](https://github.com/PaddlePaddle/Paddle/pull/74768),[#74855](https://github.com/PaddlePaddle/Paddle/pull/74855),[#74903](https://github.com/PaddlePaddle/Paddle/pull/74903),[#74788](https://github.com/PaddlePaddle/Paddle/pull/74788),[#74631](https://github.com/PaddlePaddle/Paddle/pull/74631),[#74947](https://github.com/PaddlePaddle/Paddle/pull/74947) - API新增一些通用参数：`out`、`device`、`dtype`、`requires_grad`、`pin_memory`、`bias`，增强原有功能。包括 `paddle.zeros`、`paddle.zeros_like`、`paddle.ones`、`paddle.ones_like`、`paddle.arange`、`paddle.eye`、`paddle.empty`、`paddle.empty_like`、`paddle.full`、`paddle.full_like`、`paddle.randn`、`paddle.Tensor.new_full`、`paddle.Tensor.new_empty`、`paddle.Tensor.new_ones`、`paddle.Tensor.new_zeros`、`paddle.tril/triu`、`paddle.bmm`、`paddle.nn.Conv1D/Conv2D/Conv3D/Embedding`、`paddle.diff`、`paddle.cumsum`、`paddle.var`、`paddle.multinomial`、`paddle.mean`等。[#74477](https://github.com/PaddlePaddle/Paddle/pull/74477),[#74526](https://github.com/PaddlePaddle/Paddle/pull/74526),[#74711](https://github.com/PaddlePaddle/Paddle/pull/74711),[#74582](https://github.com/PaddlePaddle/Paddle/pull/74582),[#74624](https://github.com/PaddlePaddle/Paddle/pull/74624),[#74849](https://github.com/PaddlePaddle/Paddle/pull/74849),[#74612](https://github.com/PaddlePaddle/Paddle/pull/74612),[#74875](https://github.com/PaddlePaddle/Paddle/pull/74875),[#74641](https://github.com/PaddlePaddle/Paddle/pull/74641),[#74949](https://github.com/PaddlePaddle/Paddle/pull/74949),[#74918](https://github.com/PaddlePaddle/Paddle/pull/74918),[#74914](https://github.com/PaddlePaddle/Paddle/pull/74914),[#74934](https://github.com/PaddlePaddle/Paddle/pull/74934),[#74920](https://github.com/PaddlePaddle/Paddle/pull/74920),[#74955](https://github.com/PaddlePaddle/Paddle/pull/74955),[#74226](https://github.com/PaddlePaddle/Paddle/pull/74226),[#74946](https://github.com/PaddlePaddle/Paddle/pull/74946) - API新增别名，支持更多调用方式。包括 `paddle.Tensor.mul_/mul`、`paddle.autograd.Function`、`paddle.argwhere`、`paddle.cat`、`paddle.clamp`、`paddle.ger`、`paddle.take_along_dim`、`paddle.linalg.matmul`、`paddle.special.logsumexp`、`paddle.concatenate`、`paddle.eq/gt、`paddle.Tensor.take_along_dim`、`paddle.nn.Conv1d/Conv2d/Conv3d`等。[#74493](https://github.com/PaddlePaddle/Paddle/pull/74493),[#74569](https://github.com/PaddlePaddle/Paddle/pull/74569),[#74870](https://github.com/PaddlePaddle/Paddle/pull/74870) ### Bug修复 - 修复 `paddle.nanmedian` 精度问题。[#74263](https://github.com/PaddlePaddle/Paddle/pull/74263) - 修复 `paddle.distributed.fleet.utils.hybrid_parallel_util.fused_allreduce_gradients` 在0-D下的问题。[#74957](https://github.com/PaddlePaddle/Paddle/pull/74957) - 修复 `paddle.matmul` 在分布式下的问题。[#74989](https://github.com/PaddlePaddle/Paddle/pull/74989) ### 功能增强 - 针对返回多个Tensor的情况，通过paddle数据结构来封装，优化体验。包括 `paddle.topk`。[#74931](https://github.com/PaddlePaddle/Paddle/pull/74931) - 创建类API支持size为可变参数的用法。[#74494](https://github.com/PaddlePaddle/Paddle/pull/74494) ### 文档 - 新增或修复文档。[#74453](https://github.com/PaddlePaddle/Paddle/pull/74453),[#74846](https://github.com/PaddlePaddle/Paddle/pull/74846),[#74982](https://github.com/PaddlePaddle/Paddle/pull/74982) ### 其他 - 代码风格相关的优化。[#74654](https://github.com/PaddlePaddle/Paddle/pull/74654),[#74655](https://github.com/PaddlePaddle/Paddle/pull/74655),[#74665](https://github.com/PaddlePaddle/Paddle/pull/74665),[#74660](https://github.com/PaddlePaddle/Paddle/pull/74660),[#74667](https://github.com/PaddlePaddle/Paddle/pull/74667),[#74664](https://github.com/PaddlePaddle/Paddle/pull/74664),[#74662](https://github.com/PaddlePaddle/Paddle/pull/74662),[#74661](https://github.com/PaddlePaddle/Paddle/pull/74661),[#74658](https://github.com/PaddlePaddle/Paddle/pull/74658),[#74657](https://github.com/PaddlePaddle/Paddle/pull/74657),[#74666](https://github.com/PaddlePaddle/Paddle/pull/74666),[#74659](https://github.com/PaddlePaddle/Paddle/pull/74659),[#74663](https://github.com/PaddlePaddle/Paddle/pull/74663),[#74656](https://github.com/PaddlePaddle/Paddle/pull/74656),[#74673](https://github.com/PaddlePaddle/Paddle/pull/74673),[#74672](https://github.com/PaddlePaddle/Paddle/pull/74672),[#74671](https://github.com/PaddlePaddle/Paddle/pull/74671),[#74674](https://github.com/PaddlePaddle/Paddle/pull/74674),[#74675](https://github.com/PaddlePaddle/Paddle/pull/74675),[#74670](https://github.com/PaddlePaddle/Paddle/pull/74670),[#74669](https://github.com/PaddlePaddle/Paddle/pull/74669),[#74677](https://github.com/PaddlePaddle/Paddle/pull/74677),[#74709](https://github.com/PaddlePaddle/Paddle/pull/74709),[#74714](https://github.com/PaddlePaddle/Paddle/pull/74714),[#74712](https://github.com/PaddlePaddle/Paddle/pull/74712),[#74713](https://github.com/PaddlePaddle/Paddle/pull/74713),[#74704](https://github.com/PaddlePaddle/Paddle/pull/74704),[#74746](https://github.com/PaddlePaddle/Paddle/pull/74746),[#74748](https://github.com/PaddlePaddle/Paddle/pull/74748),[#74743](https://github.com/PaddlePaddle/Paddle/pull/74743),[#74742](https://github.com/PaddlePaddle/Paddle/pull/74742),[#74744](https://github.com/PaddlePaddle/Paddle/pull/74744),[#74745](https://github.com/PaddlePaddle/Paddle/pull/74745),[#74747](https://github.com/PaddlePaddle/Paddle/pull/74747),[#74794](https://github.com/PaddlePaddle/Paddle/pull/74794),[#74789](https://github.com/PaddlePaddle/Paddle/pull/74789),[#74793](https://github.com/PaddlePaddle/Paddle/pull/74793),[#74786](https://github.com/PaddlePaddle/Paddle/pull/74786),[#74791](https://github.com/PaddlePaddle/Paddle/pull/74791),[#74787](https://github.com/PaddlePaddle/Paddle/pull/74787),[#74827](https://github.com/PaddlePaddle/Paddle/pull/74827),[#74608](https://github.com/PaddlePaddle/Paddle/pull/74608),[#74288](https://github.com/PaddlePaddle/Paddle/pull/74288),[#74287](https://github.com/PaddlePaddle/Paddle/pull/74287),[#74385](https://github.com/PaddlePaddle/Paddle/pull/74385),[#74395](https://github.com/PaddlePaddle/Paddle/pull/74395),[#74475](https://github.com/PaddlePaddle/Paddle/pull/74475),[#74647](https://github.com/PaddlePaddle/Paddle/pull/74647) - MKLDNN/ONEDNN相关的优化。[#74299](https://github.com/PaddlePaddle/Paddle/pull/74299),[#74244](https://github.com/PaddlePaddle/Paddle/pull/74244),[#74230](https://github.com/PaddlePaddle/Paddle/pull/74230),[#74314](https://github.com/PaddlePaddle/Paddle/pull/74314),[#74327](https://github.com/PaddlePaddle/Paddle/pull/74327),[#74325](https://github.com/PaddlePaddle/Paddle/pull/74325),[#74326](https://github.com/PaddlePaddle/Paddle/pull/74326),[#74315](https://github.com/PaddlePaddle/Paddle/pull/74315),[#74399](https://github.com/PaddlePaddle/Paddle/pull/74399),[#74398](https://github.com/PaddlePaddle/Paddle/pull/74398),[#74393](https://github.com/PaddlePaddle/Paddle/pull/74393),[#74392](https://github.com/PaddlePaddle/Paddle/pull/74392),[#74367](https://github.com/PaddlePaddle/Paddle/pull/74367),[#74391](https://github.com/PaddlePaddle/Paddle/pull/74391),[#74423](https://github.com/PaddlePaddle/Paddle/pull/74423),[#74424](https://github.com/PaddlePaddle/Paddle/pull/74424),[#74436](https://github.com/PaddlePaddle/Paddle/pull/74436),[#74417](https://github.com/PaddlePaddle/Paddle/pull/74417),[#74410](https://github.com/PaddlePaddle/Paddle/pull/74410),[#74473](https://github.com/PaddlePaddle/Paddle/pull/74473),[#74458](https://github.com/PaddlePaddle/Paddle/pull/74458),[#74501](https://github.com/PaddlePaddle/Paddle/pull/74501),[#74487](https://github.com/PaddlePaddle/Paddle/pull/74487),[#74502](https://github.com/PaddlePaddle/Paddle/pull/74502),[#74513](https://github.com/PaddlePaddle/Paddle/pull/74513),[#74518](https://github.com/PaddlePaddle/Paddle/pull/74518),[#74516](https://github.com/PaddlePaddle/Paddle/pull/74516),[#74507](https://github.com/PaddlePaddle/Paddle/pull/74507),[#74504](https://github.com/PaddlePaddle/Paddle/pull/74504),[#74505](https://github.com/PaddlePaddle/Paddle/pull/74505),[#74509](https://github.com/PaddlePaddle/Paddle/pull/74509),[#74535](https://github.com/PaddlePaddle/Paddle/pull/74535),[#74536](https://github.com/PaddlePaddle/Paddle/pull/74536),[#74517](https://github.com/PaddlePaddle/Paddle/pull/74517),[#74503](https://github.com/PaddlePaddle/Paddle/pull/74503),[#74557](https://github.com/PaddlePaddle/Paddle/pull/74557),[#74550](https://github.com/PaddlePaddle/Paddle/pull/74550),[#74575](https://github.com/PaddlePaddle/Paddle/pull/74575),[#74587](https://github.com/PaddlePaddle/Paddle/pull/74587),[#74576](https://github.com/PaddlePaddle/Paddle/pull/74576),[#74588](https://github.com/PaddlePaddle/Paddle/pull/74588),[#74549](https://github.com/PaddlePaddle/Paddle/pull/74549),[#74581](https://github.com/PaddlePaddle/Paddle/pull/74581),[#74583](https://github.com/PaddlePaddle/Paddle/pull/74583),[#74628](https://github.com/PaddlePaddle/Paddle/pull/74628),[#74630](https://github.com/PaddlePaddle/Paddle/pull/74630),[#74635](https://github.com/PaddlePaddle/Paddle/pull/74635),[#74679](https://github.com/PaddlePaddle/Paddle/pull/74679),[#74648](https://github.com/PaddlePaddle/Paddle/pull/74648),[#74127](https://github.com/PaddlePaddle/Paddle/pull/74127),[#74636](https://github.com/PaddlePaddle/Paddle/pull/74636),[#74552](https://github.com/PaddlePaddle/Paddle/pull/74552),[#74551](https://github.com/PaddlePaddle/Paddle/pull/74551),[#74678](https://github.com/PaddlePaddle/Paddle/pull/74678),[#74680](https://github.com/PaddlePaddle/Paddle/pull/74680),[#74730](https://github.com/PaddlePaddle/Paddle/pull/74730),[#74751](https://github.com/PaddlePaddle/Paddle/pull/74751),[#74895](https://github.com/PaddlePaddle/Paddle/pull/74895),[#74821](https://github.com/PaddlePaddle/Paddle/pull/74821),[#74897](https://github.com/PaddlePaddle/Paddle/pull/74897),[#74734](https://github.com/PaddlePaddle/Paddle/pull/74734) - 代码实现相关的优化，变量与文件重命名。[#74309](https://github.com/PaddlePaddle/Paddle/pull/74309),[#74597](https://github.com/PaddlePaddle/Paddle/pull/74597),[#74613](https://github.com/PaddlePaddle/Paddle/pull/74613),[#74376](https://github.com/PaddlePaddle/Paddle/pull/74376),[#74479](https://github.com/PaddlePaddle/Paddle/pull/74479),[#74960](https://github.com/PaddlePaddle/Paddle/pull/74960),[#74968](https://github.com/PaddlePaddle/Paddle/pull/74968),[#74977](https://github.com/PaddlePaddle/Paddle/pull/74977) - 单测相关的优化，单测问题修复。[#74595](https://github.com/PaddlePaddle/Paddle/pull/74595) - 编译相关的优化，CI问题修复。[#74356](https://github.com/PaddlePaddle/Paddle/pull/74356),[#74936](https://github.com/PaddlePaddle/Paddle/pull/74936) - 优化调试与打印信息，优化报错信息。[#74765](https://github.com/PaddlePaddle/Paddle/pull/74765),[#74381](https://github.com/PaddlePaddle/Paddle/pull/74381),[#74384](https://github.com/PaddlePaddle/Paddle/pull/74384),[#74386](https://github.com/PaddlePaddle/Paddle/pull/74386),[#74387](https://github.com/PaddlePaddle/Paddle/pull/74387),[#74383](https://github.com/PaddlePaddle/Paddle/pull/74383),[#74519](https://github.com/PaddlePaddle/Paddle/pull/74519),[#74520](https://github.com/PaddlePaddle/Paddle/pull/74520),[#74468](https://github.com/PaddlePaddle/Paddle/pull/74468) - 自定义算子相关优化。[#74402](https://github.com/PaddlePaddle/Paddle/pull/74402) - 分布式FlexCheckpoint支持。[#74966](https://github.com/PaddlePaddle/Paddle/pull/74966),[#74593](https://github.com/PaddlePaddle/Paddle/pull/74593),[#74785](https://github.com/PaddlePaddle/Paddle/pull/74785),[#74814](https://github.com/PaddlePaddle/Paddle/pull/74814) ## 2. 基础执行架构 ### 新功能 - 动态图支持。[#74484](https://github.com/PaddlePaddle/Paddle/pull/74484) - 支持 safetensors。[#74642](https://github.com/PaddlePaddle/Paddle/pull/74642), [#74609](https://github.com/PaddlePaddle/Paddle/pull/74609), [#75049](https://github.com/PaddlePaddle/Paddle/pull/75049) - 添加offloader优化计算效率。 [#74837](https://github.com/PaddlePaddle/Paddle/pull/74837) - 为 conv_transpose 前向计算添加 API 支持。 [#74431](https://github.com/PaddlePaddle/Paddle/pull/74431) - 添加offloader优化计算效率。 [#74837](https://github.com/PaddlePaddle/Paddle/pull/74837) - 推理部署增加了w4afp8量化推理，支持w4afp8量化权重纯排及all2all通信[#74270](https://github.com/PaddlePaddle/Paddle/pull/74270) ### Bug修复 - 核心框架与基础设施优化。[#74336](https://github.com/PaddlePaddle/Paddle/pull/74336), [#74554](https://github.com/PaddlePaddle/Paddle/pull/74554), [#74634](https://github.com/PaddlePaddle/Paddle/pull/74634) - 计算精度与类型处理。 [#74278](https://github.com/PaddlePaddle/Paddle/pull/74278), [#74222](https://github.com/PaddlePaddle/Paddle/pull/74222), [#74830](https://github.com/PaddlePaddle/Paddle/pull/74830) - 动态维度检查逻辑优化。 [#74633](https://github.com/PaddlePaddle/Paddle/pull/74633), [#74650](https://github.com/PaddlePaddle/Paddle/pull/74650) - 内存与非法访问修复。 [#74347](https://github.com/PaddlePaddle/Paddle/pull/74347), [#73443](https://github.com/PaddlePaddle/Paddle/pull/73443), [#74953](https://github.com/PaddlePaddle/Paddle/pull/74953) - 修复报错/告警信息打印。 [#74474](https://github.com/PaddlePaddle/Paddle/pull/74474), [#74533](https://github.com/PaddlePaddle/Paddle/pull/74533), [#74685](https://github.com/PaddlePaddle/Paddle/pull/74685), [#74721](https://github.com/PaddlePaddle/Paddle/pull/74721), [#74754](https://github.com/PaddlePaddle/Paddle/pull/74754) - 代码质量与文档修正。 [#74378](https://github.com/PaddlePaddle/Paddle/pull/74378), [#74828](https://github.com/PaddlePaddle/Paddle/pull/74828) - 修复 flashmask API 处理逻辑。 [#74928](https://github.com/PaddlePaddle/Paddle/pull/74928) - 修复动转静模式下切分CudaGraph子图未生效的问题。 ([#74749](https://github.com/PaddlePaddle/Paddle/pull/74749)) ### 功能增强 - C++ 扩展开发。 [#74338](https://github.com/PaddlePaddle/Paddle/pull/74338) - FlexCP 功能优化。 [#74752](https://github.com/PaddlePaddle/Paddle/pull/74752), [#74981](https://github.com/PaddlePaddle/Paddle/pull/74981) - 优化内存分配。[#74463](https://github.com/PaddlePaddle/Paddle/pull/74463) ### 废弃 - 清理动转静旧 IR 相关单测。 [#74698](https://github.com/PaddlePaddle/Paddle/pull/74698), [#74715](https://github.com/PaddlePaddle/Paddle/pull/74715), [#74718](https://github.com/PaddlePaddle/Paddle/pull/74718), [#74782](https://github.com/PaddlePaddle/Paddle/pull/74782), [#74962](https://github.com/PaddlePaddle/Paddle/pull/74962) ### 其他 - 更改补丁版本。 [#74940](https://github.com/PaddlePaddle/Paddle/pull/74940) ## 3. 分布式&自动并行 ### 并行策略在3.2版本中，我们对流水线并行功能进行了多项增强，包括实现了字典参数传递的支持，并扩展了Pipeline Layer和SharedLayerDesc对非流水线并行的兼容性；同时修复了多个关键问题，包括大尺寸张量的IPC API异常、流水线并行中的评估批次和非计算损失问题、MoE模型的梯度释放错误、PP场景下NCCL通信重建导致的hang问题，以及双流水线并行的event管理错误；此外还进行了多项性能优化，改进了双流水线并行的计算重叠效率以提升训练性能，并升级了clear_param_storage方法使其支持sharding模式下多color集合的清除和重置操作。 #### 功能新增 - 实现流水线并行（Pipeline Parallel）中字典参数传递的支持。[#74574](https://github.com/PaddlePaddle/Paddle/pull/74574),[#74867](https://github.com/PaddlePaddle/Paddle/pull/74867) - Pipeline Layer 和 SharedLayerDesc 支持非流水线并行（nonpp parallel）。[#74573](https://github.com/PaddlePaddle/Paddle/pull/74573) #### Bug 修复 - 修复大尺寸张量的 IPC API 问题。[#74472](https://github.com/PaddlePaddle/Paddle/pull/74472) - 修复流水线并行中的评估批次（eval batch）及非计算损失（non-compute_loss）问题。[#74170](https://github.com/PaddlePaddle/Paddle/pull/74170) - 修复 MoE 模型上的梯度释放问题。[#74972](https://github.com/PaddlePaddle/Paddle/pull/74972) - 修复在pp的场景下重建NCCL comm存在hang的问题。[#73625](https://github.com/PaddlePaddle/Paddle/pull/73625) - 修复双流水线并行（dual pp）的event管理错误。[#74158](https://github.com/PaddlePaddle/Paddle/pull/74158) #### 优化改进 - 优化双流水线并行的计算重叠（overlap）效率，提升训练性能。[#74527](https://github.com/PaddlePaddle/Paddle/pull/74527) - 升级clear_param_storage方法，支持sharding下多个color集合清除和重置。[#74741](https://github.com/PaddlePaddle/Paddle/pull/74741) ### 自动并行 #### 功能改进 - 支持分布式张量的同一维度被多个mesh维度切分时的默认切分推导规则。[#74396](https://github.com/PaddlePaddle/Paddle/pull/74396) - 改进 `reshape` 算子的切分推导规则，以支持分布式张量的同一维度被多个mesh维度切分的场景。[#74352](https://github.com/PaddlePaddle/Paddle/pull/74352),[#74579](https://github.com/PaddlePaddle/Paddle/pull/74579), [#74565](https://github.com/PaddlePaddle/Paddle/pull/74565) - 支持在不改变分布式张量数据的情况下改变张量的mesh。[#74248](https://github.com/PaddlePaddle/Paddle/pull/74248) #### Bug 修复 - 修复调用 `ProcessMesh` 的 `get_group` 方法时重复创建通信组的bug。[#73099](https://github.com/PaddlePaddle/Paddle/pull/73099) - 修复MoE场景下`get_local_slices` 方法的bug。[#74705](https://github.com/PaddlePaddle/Paddle/pull/74705) - 修复MoE场景下梯度裁剪的bug。[#74916](https://github.com/PaddlePaddle/Paddle/pull/74916) - 修复流水线并行场景下不同stage间无法传递`stop_gradient`参数的bug。[#73459](https://github.com/PaddlePaddle/Paddle/pull/73459) - 修复流水线并行场景下梯度裁剪的精度bug。[#74409](https://github.com/PaddlePaddle/Paddle/pull/74409) - 修复动态图流水线并行场景下产生冗余输出的bug。[#74913](https://github.com/PaddlePaddle/Paddle/pull/74913) - 修复算子`moe_combine`和`moe_gate_dispatch`在MoE场景下跑不通的bug。[#74645](https://github.com/PaddlePaddle/Paddle/pull/74645) #### 其他 - 支持dataloader手动并行和自动并行的精度对齐。[#73941](https://github.com/PaddlePaddle/Paddle/pull/73941) - 优化动态图流水并行调度逻辑。[#74720](https://github.com/PaddlePaddle/Paddle/pull/74720) ### 通信库在3.2版本中，我们修复了DeepEP支持sm90编译的一个报错，同时对DeepEP申请的显存分配添加了预分配功能，并升级了其intranode和internode计算kernel，进一步优化了性能和稳定性。 #### Bug修复 - 修复DeepEP支持sm90 编译的一个报错。[#74762](https://github.com/PaddlePaddle/Paddle/pull/74762) #### 功能改进 - 对DeepEP申请的显存分配添加预分配功能。[#74465](https://github.com/PaddlePaddle/Paddle/pull/74465) - 升级DeepEP的intranode和internode计算kernel。[#74284](https://github.com/PaddlePaddle/Paddle/pull/74284) ## 4. 算子机制 ### 新特性 - API 兼容性支持。 [#74506](https://github.com/PaddlePaddle/Paddle/pull/74506), [#74676](https://github.com/PaddlePaddle/Paddle/pull/74676), [#74558](https://github.com/PaddlePaddle/Paddle/pull/74558), [#74572](https://github.com/PaddlePaddle/Paddle/pull/74572), [#74691](https://github.com/PaddlePaddle/Paddle/pull/74691), [#74703](https://github.com/PaddlePaddle/Paddle/pull/74703), [#74750](https://github.com/PaddlePaddle/Paddle/pull/74750), [#74757](https://github.com/PaddlePaddle/Paddle/pull/74757), [#74802](https://github.com/PaddlePaddle/Paddle/pull/74802), [#74546](https://github.com/PaddlePaddle/Paddle/pull/74546), [#74547](https://github.com/PaddlePaddle/Paddle/pull/74547), [#74802](https://github.com/PaddlePaddle/Paddle/pull/74802), [#74859](https://github.com/PaddlePaddle/Paddle/pull/74859), [#74910](https://github.com/PaddlePaddle/Paddle/pull/74910), [#74873](https://github.com/PaddlePaddle/Paddle/pull/74873), [#74882](https://github.com/PaddlePaddle/Paddle/pull/74882), [#74901](https://github.com/PaddlePaddle/Paddle/pull/74901), [#74899](https://github.com/PaddlePaddle/Paddle/pull/74899), [#74449](https://github.com/PaddlePaddle/Paddle/pull/74449) - 新增 fused_partial_rope 算子。 [#74577](https://github.com/PaddlePaddle/Paddle/pull/74577) ### Bug修复 - 0-size Tensor 相关修复。 [#74295](https://github.com/PaddlePaddle/Paddle/pull/74295), [#74305](https://github.com/PaddlePaddle/Paddle/pull/74305), [#74323](https://github.com/PaddlePaddle/Paddle/pull/74323), [#74354](https://github.com/PaddlePaddle/Paddle/pull/74354) - 大 Tensor 相关修复。 [#74242](https://github.com/PaddlePaddle/Paddle/pull/74242), [#74293](https://github.com/PaddlePaddle/Paddle/pull/74293), [#74289](https://github.com/PaddlePaddle/Paddle/pull/74289), [#74279](https://github.com/PaddlePaddle/Paddle/pull/74279), [#74330](https://github.com/PaddlePaddle/Paddle/pull/74330), [#74329](https://github.com/PaddlePaddle/Paddle/pull/74329), [#74342](https://github.com/PaddlePaddle/Paddle/pull/74342), [#74369](https://github.com/PaddlePaddle/Paddle/pull/74369), [#74370](https://github.com/PaddlePaddle/Paddle/pull/74370), [#74404](https://github.com/PaddlePaddle/Paddle/pull/74404), [#74537](https://github.com/PaddlePaddle/Paddle/pull/74537), [#74451](https://github.com/PaddlePaddle/Paddle/pull/74451), [#74172](https://github.com/PaddlePaddle/Paddle/pull/74172), [#74324](https://github.com/PaddlePaddle/Paddle/pull/74324), [#74964](https://github.com/PaddlePaddle/Paddle/pull/74964), [#74360](https://github.com/PaddlePaddle/Paddle/pull/74360), [#74379](https://github.com/PaddlePaddle/Paddle/pull/74379), [#74377](https://github.com/PaddlePaddle/Paddle/pull/74377), [#74380](https://github.com/PaddlePaddle/Paddle/pull/74380), [#74362](https://github.com/PaddlePaddle/Paddle/pull/74362), [#74197](https://github.com/PaddlePaddle/Paddle/pull/74197) - API 兼容性相关修复。 [#74764](https://github.com/PaddlePaddle/Paddle/pull/74764), [#74869](https://github.com/PaddlePaddle/Paddle/pull/74869), [#74935](https://github.com/PaddlePaddle/Paddle/pull/74935) - 【开源任务】Paddle CPU/GPU Kernel 精度问题推全。 [#74149](https://github.com/PaddlePaddle/Paddle/pull/74149), [#74598](https://github.com/PaddlePaddle/Paddle/pull/74598), [#74719](https://github.com/PaddlePaddle/Paddle/pull/74719), [#74625](https://github.com/PaddlePaddle/Paddle/pull/74625), [#74555](https://github.com/PaddlePaddle/Paddle/pull/74555) - 其他重要修复。 [#74282](https://github.com/PaddlePaddle/Paddle/pull/74282), [#74313](https://github.com/PaddlePaddle/Paddle/pull/74313), [#74303](https://github.com/PaddlePaddle/Paddle/pull/74303), [#74306](https://github.com/PaddlePaddle/Paddle/pull/74306), [#74298](https://github.com/PaddlePaddle/Paddle/pull/74298), [#74044](https://github.com/PaddlePaddle/Paddle/pull/74044), [#74290](https://github.com/PaddlePaddle/Paddle/pull/74290), [#74348](https://github.com/PaddlePaddle/Paddle/pull/74348), [#74364](https://github.com/PaddlePaddle/Paddle/pull/74364), [#74332](https://github.com/PaddlePaddle/Paddle/pull/74332), [#74224](https://github.com/PaddlePaddle/Paddle/pull/74224), [#74382](https://github.com/PaddlePaddle/Paddle/pull/74382), [#74406](https://github.com/PaddlePaddle/Paddle/pull/74406), [#74434](https://github.com/PaddlePaddle/Paddle/pull/74434), [#74448](https://github.com/PaddlePaddle/Paddle/pull/74448), [#74457](https://github.com/PaddlePaddle/Paddle/pull/74457), [#74322](https://github.com/PaddlePaddle/Paddle/pull/74322), [#74530](https://github.com/PaddlePaddle/Paddle/pull/74530), [#74716](https://github.com/PaddlePaddle/Paddle/pull/74716), [#74839](https://github.com/PaddlePaddle/Paddle/pull/74839), [#74842](https://github.com/PaddlePaddle/Paddle/pull/74842), [#74854](https://github.com/PaddlePaddle/Paddle/pull/74854), [#74919](https://github.com/PaddlePaddle/Paddle/pull/74919), [#74767](https://github.com/PaddlePaddle/Paddle/pull/74767), [#75003](https://github.com/PaddlePaddle/Paddle/pull/75003) ### 功能增强 - API 兼容能力提升。 [#74456](https://github.com/PaddlePaddle/Paddle/pull/74456), [#74480](https://github.com/PaddlePaddle/Paddle/pull/74480), [#74523](https://github.com/PaddlePaddle/Paddle/pull/74523), [#74490](https://github.com/PaddlePaddle/Paddle/pull/74490), [#74548](https://github.com/PaddlePaddle/Paddle/pull/74548), [#74596](https://github.com/PaddlePaddle/Paddle/pull/74596), [#74568](https://github.com/PaddlePaddle/Paddle/pull/74568), [#74559](https://github.com/PaddlePaddle/Paddle/pull/74559), [#74629](https://github.com/PaddlePaddle/Paddle/pull/74629), [#74623](https://github.com/PaddlePaddle/Paddle/pull/74623), [#74700](https://github.com/PaddlePaddle/Paddle/pull/74700), [#74643](https://github.com/PaddlePaddle/Paddle/pull/74643), [#74602](https://github.com/PaddlePaddle/Paddle/pull/74602), [#74783](https://github.com/PaddlePaddle/Paddle/pull/74783), [#74781](https://github.com/PaddlePaddle/Paddle/pull/74781), [#74735](https://github.com/PaddlePaddle/Paddle/pull/74735), [#74725](https://github.com/PaddlePaddle/Paddle/pull/74725), [#74815](https://github.com/PaddlePaddle/Paddle/pull/74815), [#74856](https://github.com/PaddlePaddle/Paddle/pull/74856), [#74925](https://github.com/PaddlePaddle/Paddle/pull/74925), [#74545](https://github.com/PaddlePaddle/Paddle/pull/74545), [#74932](https://github.com/PaddlePaddle/Paddle/pull/74932), [#74784](https://github.com/PaddlePaddle/Paddle/pull/74784) - slice/stride 相关优化。 [#74731](https://github.com/PaddlePaddle/Paddle/pull/74731), [#74740](https://github.com/PaddlePaddle/Paddle/pull/74740), [#74769](https://github.com/PaddlePaddle/Paddle/pull/74769), [#74810](https://github.com/PaddlePaddle/Paddle/pull/74810), [#74841](https://github.com/PaddlePaddle/Paddle/pull/74841), [#74954](https://github.com/PaddlePaddle/Paddle/pull/74954), [#74888](https://github.com/PaddlePaddle/Paddle/pull/74888), [#74944](https://github.com/PaddlePaddle/Paddle/pull/74944), [#74312](https://github.com/PaddlePaddle/Paddle/pull/74312), [#74291](https://github.com/PaddlePaddle/Paddle/pull/74291), [#74271](https://github.com/PaddlePaddle/Paddle/pull/74271), [#74320](https://github.com/PaddlePaddle/Paddle/pull/74320), [#74344](https://github.com/PaddlePaddle/Paddle/pull/74344), [#74727](https://github.com/PaddlePaddle/Paddle/pull/74727), [#74637](https://github.com/PaddlePaddle/Paddle/pull/74637) - 算子优化与 CUDA 支持。 [#74693](https://github.com/PaddlePaddle/Paddle/pull/74693), [#74922](https://github.com/PaddlePaddle/Paddle/pull/74922), [#74967](https://github.com/PaddlePaddle/Paddle/pull/74967) - 改进调试信息、兼容性增强。 [#74372](https://github.com/PaddlePaddle/Paddle/pull/74372), [#74622](https://github.com/PaddlePaddle/Paddle/pull/74622) - 算子功能扩展与优化。 [#74790](https://github.com/PaddlePaddle/Paddle/pull/74790), [#74979](https://github.com/PaddlePaddle/Paddle/pull/74979) ### 性能优化 - FP8 计算优化。 [#74471](https://github.com/PaddlePaddle/Paddle/pull/74471), [#74684](https://github.com/PaddlePaddle/Paddle/pull/74684), [#74911](https://github.com/PaddlePaddle/Paddle/pull/74911) - 基础算子性能优化。 [#74442](https://github.com/PaddlePaddle/Paddle/pull/74442), [#74638](https://github.com/PaddlePaddle/Paddle/pull/74638) - 支持 fa3 变长序列反向计算并优化前向 API。 [#73831](https://github.com/PaddlePaddle/Paddle/pull/73831) - 新增 FlashMask V2 功能。 [#74729](https://github.com/PaddlePaddle/Paddle/pull/74729) ### 文档 - 修复英文文档问题以及版权年份问题。 [#74737](https://github.com/PaddlePaddle/Paddle/pull/74737) ### 其他 - 在XPU硬件上默认开启 WITH_XPU_FFT 选项。 [#74699](https://github.com/PaddlePaddle/Paddle/pull/74699) ## 5. 硬件适配 ### 类CUDA硬件接入方案完善 - 类CUDA硬件接入方案支持cuBlas kernel的复用 [#74591](https://github.com/PaddlePaddle/Paddle/pull/74591), - 类CUDA硬件接入方案已知问题修复 [#74397](https://github.com/PaddlePaddle/Paddle/pull/74397), [#74411](https://github.com/PaddlePaddle/Paddle/pull/74411), [#74428](https://github.com/PaddlePaddle/Paddle/pull/74428), [#74877](https://github.com/PaddlePaddle/Paddle/pull/74877), [#74939](https://github.com/PaddlePaddle/Paddle/pull/74939) ### 主仓单测支持多硬件 - 单测支持多硬件 [#74349](https://github.com/PaddlePaddle/Paddle/pull/74349), [#74363](https://github.com/PaddlePaddle/Paddle/pull/74363)，[#74806](https://github.com/PaddlePaddle/Paddle/pull/74806), [#74868](https://github.com/PaddlePaddle/Paddle/pull/74868), [#74820](https://github.com/PaddlePaddle/Paddle/pull/74820), [#74927](https://github.com/PaddlePaddle/Paddle/pull/74927) ### 新增Custom Device API支持 - 新增Custom Device API支持 [#74308](https://github.com/PaddlePaddle/Paddle/pull/74308), [#74371](https://github.com/PaddlePaddle/Paddle/pull/74371), [#74539](https://github.com/PaddlePaddle/Paddle/pull/74539) ## 6. 安装环境 ### Bug 修复 - 修复flashattent编译缓存的bug。[#74388](https://github.com/PaddlePaddle/Paddle/pull/74388) - 修复site.USER_SITE为None的bug。 [#74373](https://github.com/PaddlePaddle/Paddle/pull/74373) - 修复多架构 Linux 系统下gtest的编译bug。 [#74723](https://github.com/PaddlePaddle/Paddle/pull/74723) - 修复在 WITH_GPU=ON 情况下 DEBUG 模式编译多个报错。 [#74401](https://github.com/PaddlePaddle/Paddle/pull/74401) - 修复Windows下CUDA12.6编译bug。 [#74990](https://github.com/PaddlePaddle/Paddle/pull/74990) - 修复api-benchmark基线流水线bug。 [#74770](https://github.com/PaddlePaddle/Paddle/pull/74770) - 修复api-benchmark基线流水线bug。 [#74778](https://github.com/PaddlePaddle/Paddle/pull/74778) - 修复api-benchmark基线流水线bug。 [#74779](https://github.com/PaddlePaddle/Paddle/pull/74779) - 修复api-benchmark基线流水线bug。 [#74780](https://github.com/PaddlePaddle/Paddle/pull/74780) - 修复api-benchmark基线流水线bug。 [#74800](https://github.com/PaddlePaddle/Paddle/pull/74800) - 修复api-benchmark基线流水线bug。 [#74803](https://github.com/PaddlePaddle/Paddle/pull/74803) ### 其他 - 禁用test_custom_contiguous单测。 [#74337](https://github.com/PaddlePaddle/Paddle/pull/74337) - 支持录取slice 流水线基线任务定时触发。 [#74419](https://github.com/PaddlePaddle/Paddle/pull/74419) - 支持slice录基线添加手动指定pr。 [#74445](https://github.com/PaddlePaddle/Paddle/pull/74445) - 检查代码中是否带有中问题。 [#74460](https://github.com/PaddlePaddle/Paddle/pull/74460) - 支持CI PaddleX在XPU上的任务。 [#74426](https://github.com/PaddlePaddle/Paddle/pull/74426) - 支持slice流水线豁免机制。 [#74482](https://github.com/PaddlePaddle/Paddle/pull/74482) - 更新paddle基础镜像。 [#73423](https://github.com/PaddlePaddle/Paddle/pull/73423) - windows 固定ninja版本1.11。 [#74590](https://github.com/PaddlePaddle/Paddle/pull/74590) - 支持添加关闭pr取消CI。 [#74604](https://github.com/PaddlePaddle/Paddle/pull/74604) - 支持快速跳过所有CI。 [#74696](https://github.com/PaddlePaddle/Paddle/pull/74696) - 增加api-benchmark基线流水线。 [#74690](https://github.com/PaddlePaddle/Paddle/pull/74690) - 更新nccl版本。 [#74809](https://github.com/PaddlePaddle/Paddle/pull/74809) - 更新approve流水线RD名单。 [#74838](https://github.com/PaddlePaddle/Paddle/pull/74838) - 更新approve流水线RD名单。 [#74902](https://github.com/PaddlePaddle/Paddle/pull/74902) - 更新safetensor到镜像中。 [#74904](https://github.com/PaddlePaddle/Paddle/pull/74904) - 添加flashatten的编译flag。 [#74959](https://github.com/PaddlePaddle/Paddle/pull/74959) - 临时禁用win-inference流水线。 [#74980](https://github.com/PaddlePaddle/Paddle/pull/74980) - 支持windows编译phi动态库。 [#74950](https://github.com/PaddlePaddle/Paddle/pull/74950) ## 7. 贡献者名单 AIbin, Ayakouji, baiyue, baoqiwen, Chang Lu, Chen Zhiyang, co63oc, cyberslack_lee, cyy536, datutu-L, Deng Haodong, Difer, Eddie-Wang, enzodechine, fangfangssj, feri, fxyfxy777, ggggxm, GoldPancake, gouzil, Gu Shiwei, Haze188 灏喆, hohdiy, hong, HU Shenwei, huangjiyi, HydrogenSulfate, kjagsdq, LCStayingdullCircuit, Leo Guo, lightbrother, liufengwei0103, liuruyan, LiYuRio, LLSGYN, Lucas, Luckycheng222, lzy, Nana, Nyakku Shigure, ooo oo, Qianyue He, risemeup1, Ruibiao Chen, Ryan, Shuhao Liang, sneaxiy, Starrysea996, SUN Dong, Tao Luo, Tian, tianhaodongbd, tianshuo78520a, umiswing, waliwali777, wanghuancoder, Wenhao.Dai, wyw, XiaoguangHu, xiaoguoguo626807, xingmingyyj, Yichen Zhang, Yohanna, yongqiangma, Yuan Xiaolan, YUNSHEN XIE, Yuntao Nie, Yuqiang Ge, Yutian Rao, Zero Rains, Zhan Rongrui, Zhang Ting, zhanghonggeng, Zhaowu Pan, zhengshengning, ZhenxingLi, Zhou Xin, zhupengyang, zhwesky2010, Zichao, zty-king, Zx, zyfncg, zzm, 周周周, 正在学习, 苍天荒

PaddlePaddle 3.1.0 Release Notev3.1.0

? · 2025-06-29

- [English Version](https://github.com/PaddlePaddle/Paddle/wiki/PaddlePaddle-3.1.0-Release-Note-EN) # 重要更新飞桨框架 3.1 版本，针对核心功能自动并行进一步优化打磨，提升易用性和性能表现；同时提供FP8低精度训练支持，提升大模型训练速度提升10-20%；完善硬件扩展机制，降低类 cuda 类硬件适配成本，用户仅需注册kernel；同时对于框架基础能力进行增强，提升框架稳定性。重点更新功能如下： - **自动并行架构：** 自动并行架构进一步打磨，以提高自动并行核心机制易用性和动态图性能。完善了自动并行核心机制，包括新增了多个算子的切分推导规则，支持分布式张量的同一维度被多个mesh维度切分，支持动态图并行策略（PP，CP，SEP，TP-CONV）等。同时，对动态图自动并行系统地做了性能优化，在 Llama2 Qwen Baichuan 等系列模型上性能基本持平手动并行的性能。 - **低精度训练：** 基于blockwise 的fp8 gemm算子，支持低精度训练，训练精度媲美BF16，大模型训练速度提速 10-20%。 - **异构多芯适配：** 提供类 cuda 算子复用机制，仅需注册即可使用对应 kernel。 - **框架稳定性增强：** 系统修复算子在0-Size 和大维度情况计算结果错误。 ## 1. 用户体验升级 API功能增强、Bug修复与改进，旨在提升用户体验和API的易用性。新增了`paddle.randn_like` API，修复了多个API的功能缺陷，并增强了对复数类型和0-Size Tensor的支持。文档和代码也进行了相应的更新和优化，以提升整体的准确性和专业性。 ### 新特性 - 新增`paddle.randn_like` API。[#72492](https://github.com/PaddlePaddle/Paddle/pull/72492) ### Bug 修复 - 修复`tensordot` API 输入输出类型不一致问题。[#72139](https://github.com/PaddlePaddle/Paddle/pull/72139) - 修复`atleast` API输出是Tensor列表时的问题。[#73102](https://github.com/PaddlePaddle/Paddle/pull/73102) - 修复`nonzer` API问题。[#72003](https://github.com/PaddlePaddle/Paddle/pull/72003) - 修复`dualpipev`中的内存泄漏问题。[#72070](https://github.com/PaddlePaddle/Paddle/pull/72070) - 修复`softmax`计算溢出问题。[#71935](https://github.com/PaddlePaddle/Paddle/pull/71935) - 修复`take_along_axis`中在`broadcast=False`时的形状检查问题。[#72436](https://github.com/PaddlePaddle/Paddle/pull/72436) - 修复`maximum`、`minimum`对Nan输入的不正确问题。[#71933](https://github.com/PaddlePaddle/Paddle/pull/71933) - 修复`visit_type` 问题。[#72782](https://github.com/PaddlePaddle/Paddle/pull/72782) - 修复`gather_scatter_functor`中的int32越界问题。[#72905](https://github.com/PaddlePaddle/Paddle/pull/72905) - 修复`Bernoulli`的inplace实现。[#73271](https://github.com/PaddlePaddle/Paddle/pull/73271) - 修复`moe_permute`、`moe_unpermute`问题。[#73365](https://github.com/PaddlePaddle/Paddle/pull/73365) - 修复`ast.parse`对pyi文件语法检查问题。[#71872](https://github.com/PaddlePaddle/Paddle/pull/71872) - 修复复数除法问题。[#73331](https://github.com/PaddlePaddle/Paddle/pull/73331) - 修复与TensorRT集成相关的问题。[#72302](https://github.com/PaddlePaddle/Paddle/pull/72302), [#72278](https://github.com/PaddlePaddle/Paddle/pull/72278) ### 功能增强 - 增强API的功能，提升API易用性，改善用户体验。包括但不限于扩展API支持的数据类型，API参数检查，纠正API参数默认值，完善API返回值等。[#71997](https://github.com/PaddlePaddle/Paddle/pull/71997), [#72911](https://github.com/PaddlePaddle/Paddle/pull/72911), [#72985](https://github.com/PaddlePaddle/Paddle/pull/72985), [#73240](https://github.com/PaddlePaddle/Paddle/pull/73240), [#72927](https://github.com/PaddlePaddle/Paddle/pull/72927), [#73451](https://github.com/PaddlePaddle/Paddle/pull/73451), [#73416](https://github.com/PaddlePaddle/Paddle/pull/73416), [#73420](https://github.com/PaddlePaddle/Paddle/pull/73420), [#73347](https://github.com/PaddlePaddle/Paddle/pull/73347), [#73050](https://github.com/PaddlePaddle/Paddle/pull/73050), [#73246](https://github.com/PaddlePaddle/Paddle/pull/73246), [#73123](https://github.com/PaddlePaddle/Paddle/pull/73123), [#73336](https://github.com/PaddlePaddle/Paddle/pull/73336), [#73062](https://github.com/PaddlePaddle/Paddle/pull/73062), [#72201](https://github.com/PaddlePaddle/Paddle/pull/72201), [#72190](https://github.com/PaddlePaddle/Paddle/pull/72190) - 增强API对复数类型的支持。[#72279](https://github.com/PaddlePaddle/Paddle/pull/72279), [#72308](https://github.com/PaddlePaddle/Paddle/pull/72308), [#72518](https://github.com/PaddlePaddle/Paddle/pull/72518), [#72391](https://github.com/PaddlePaddle/Paddle/pull/72391), [#72239](https://github.com/PaddlePaddle/Paddle/pull/72239), [#72286](https://github.com/PaddlePaddle/Paddle/pull/72286), [#72169](https://github.com/PaddlePaddle/Paddle/pull/72169), [#72577](https://github.com/PaddlePaddle/Paddle/pull/72577), [#72619](https://github.com/PaddlePaddle/Paddle/pull/72619) - 增强API对0-Size Tensor的支持。[#72570](https://github.com/PaddlePaddle/Paddle/pull/72570), [#72692](https://github.com/PaddlePaddle/Paddle/pull/72692), [#72138](https://github.com/PaddlePaddle/Paddle/pull/72138), [#72410](https://github.com/PaddlePaddle/Paddle/pull/72410), [#72565](https://github.com/PaddlePaddle/Paddle/pull/72565), [#72262](https://github.com/PaddlePaddle/Paddle/pull/72262) - 修改对API代码中的拼写错误，以提高整体的准确性和专业性。[#71780](https://github.com/PaddlePaddle/Paddle/pull/71780), [#71786](https://github.com/PaddlePaddle/Paddle/pull/71786), [#72093](https://github.com/PaddlePaddle/Paddle/pull/72093), [#72113](https://github.com/PaddlePaddle/Paddle/pull/72113), [#72241](https://github.com/PaddlePaddle/Paddle/pull/72241), [#72237](https://github.com/PaddlePaddle/Paddle/pull/72237), [#72590](https://github.com/PaddlePaddle/Paddle/pull/72590), [#72591](https://github.com/PaddlePaddle/Paddle/pull/72591), [#72769](https://github.com/PaddlePaddle/Paddle/pull/72769), [#72858](https://github.com/PaddlePaddle/Paddle/pull/72858), [#73045](https://github.com/PaddlePaddle/Paddle/pull/73045), [#72195](https://github.com/PaddlePaddle/Paddle/pull/72195), [#72627](https://github.com/PaddlePaddle/Paddle/pull/72627), [#72657](https://github.com/PaddlePaddle/Paddle/pull/72657), [#73162](https://github.com/PaddlePaddle/Paddle/pull/73162), [#73402](https://github.com/PaddlePaddle/Paddle/pull/73402), [#72208](https://github.com/PaddlePaddle/Paddle/pull/72208), [#72659](https://github.com/PaddlePaddle/Paddle/pull/72659), [#72658](https://github.com/PaddlePaddle/Paddle/pull/72658), [#72660](https://github.com/PaddlePaddle/Paddle/pull/72660), [#72661](https://github.com/PaddlePaddle/Paddle/pull/72661), [#72656](https://github.com/PaddlePaddle/Paddle/pull/72656) - 通信优化减少显存峰值。[#72035](https://github.com/PaddlePaddle/Paddle/pull/72035) ### 文档 - 修正了文档中的错误，提高了文档的可用性和用户体验。[#72549](https://github.com/PaddlePaddle/Paddle/pull/72549), [#73036](https://github.com/PaddlePaddle/Paddle/pull/73036) ### 开发者相关 - 代码风格检查规则更新。[#72896](https://github.com/PaddlePaddle/Paddle/pull/72896), [#73179](https://github.com/PaddlePaddle/Paddle/pull/73179), [#73060](https://github.com/PaddlePaddle/Paddle/pull/73060), [#72553](https://github.com/PaddlePaddle/Paddle/pull/72553), [#72915](https://github.com/PaddlePaddle/Paddle/pull/72915), [#72916](https://github.com/PaddlePaddle/Paddle/pull/72916), [#73338](https://github.com/PaddlePaddle/Paddle/pull/73338), [#72935](https://github.com/PaddlePaddle/Paddle/pull/72935), [#72325](https://github.com/PaddlePaddle/Paddle/pull/72325), [#72935](https://github.com/PaddlePaddle/Paddle/pull/72935) - 代码变量命名更新与代码迁移。[#73048](https://github.com/PaddlePaddle/Paddle/pull/73048), [#73148](https://github.com/PaddlePaddle/Paddle/pull/73148), [#73149](https://github.com/PaddlePaddle/Paddle/pull/73149), [#73264](https://github.com/PaddlePaddle/Paddle/pull/73264), [#73159](https://github.com/PaddlePaddle/Paddle/pull/73159), [#73124](https://github.com/PaddlePaddle/Paddle/pull/73124), [#73160](https://github.com/PaddlePaddle/Paddle/pull/73160), [#73161](https://github.com/PaddlePaddle/Paddle/pull/73161), [#73374](https://github.com/PaddlePaddle/Paddle/pull/73374), [#73395](https://github.com/PaddlePaddle/Paddle/pull/73395), [#73076](https://github.com/PaddlePaddle/Paddle/pull/73076), [#73163](https://github.com/PaddlePaddle/Paddle/pull/73163), [#73255](https://github.com/PaddlePaddle/Paddle/pull/73255) - LodTensor退场。[#71968](https://github.com/PaddlePaddle/Paddle/pull/71968), [#72152](https://github.com/PaddlePaddle/Paddle/pull/72152), [#72145](https://github.com/PaddlePaddle/Paddle/pull/72145) ### 废弃代码清理 - 无用代码清理。[#71795](https://github.com/PaddlePaddle/Paddle/pull/71795), [#71792](https://github.com/PaddlePaddle/Paddle/pull/71792), [#71794](https://github.com/PaddlePaddle/Paddle/pull/71794), [#71793](https://github.com/PaddlePaddle/Paddle/pull/71793), [#72265](https://github.com/PaddlePaddle/Paddle/pull/72265), [#73167](https://github.com/PaddlePaddle/Paddle/pull/73167), [#73115](https://github.com/PaddlePaddle/Paddle/pull/73115), [#73049](https://github.com/PaddlePaddle/Paddle/pull/73049), [#72162](https://github.com/PaddlePaddle/Paddle/pull/72162), [#72321](https://github.com/PaddlePaddle/Paddle/pull/72321), [#72336](https://github.com/PaddlePaddle/Paddle/pull/72336), [#72952](https://github.com/PaddlePaddle/Paddle/pull/72952), [#72828](https://github.com/PaddlePaddle/Paddle/pull/72828) ## 2. 基础执行架构支持FP8矩阵运算，提升模型训练效率，同时对多个模型进行增强，提升稳定性; 提供是C_ops的方式调用反向接口，方便显存优化和功能实验。 ### 新特性 - 支持FP8矩阵乘法加速，提升计算性能与精度适配能力。 [#73092](https://github.com/PaddlePaddle/Paddle/pull/73092) - 0-size Tensor 执行支持。 [#71829](https://github.com/PaddlePaddle/Paddle/pull/71829), [#72263](https://github.com/PaddlePaddle/Paddle/pull/72263), [#72244](https://github.com/PaddlePaddle/Paddle/pull/72244), [#72814](https://github.com/PaddlePaddle/Paddle/pull/72814) - DeepEP支持。 [#73495](https://github.com/PaddlePaddle/Paddle/pull/73495) - 默认开启CINN后端。 [#71838](https://github.com/PaddlePaddle/Paddle/pull/71838) - 支持SOT相关执行。 [#72472](https://github.com/PaddlePaddle/Paddle/pull/72472), [#72559](https://github.com/PaddlePaddle/Paddle/pull/72559), [#72466](https://github.com/PaddlePaddle/Paddle/pull/72466), [#73269](https://github.com/PaddlePaddle/Paddle/pull/73269), [#73329](https://github.com/PaddlePaddle/Paddle/pull/73329), [#73405](https://github.com/PaddlePaddle/Paddle/pull/73405), [#73399](https://github.com/PaddlePaddle/Paddle/pull/73399), [#73424](https://github.com/PaddlePaddle/Paddle/pull/73424), [#73509](https://github.com/PaddlePaddle/Paddle/pull/73509) - 支持动转静。 [#73417](https://github.com/PaddlePaddle/Paddle/pull/73417), [#73081](https://github.com/PaddlePaddle/Paddle/pull/73081) - 新增支持stride机制的kernel。 [#73053](https://github.com/PaddlePaddle/Paddle/pull/73053) ### Bug 修复 - 性能优化与稳定性：优化训练稳定性，增强Python 3.11+支持，提升CINN编译器在动态图模式下的自动启用逻辑，修复动态shape推断与梯度回传问题，优化GPU内核执行效率（如for_range、常量折叠），改进NPU内存拷贝与上下文管理，提升大规模模型训练性能与硬件利用率。 [#71777](https://github.com/PaddlePaddle/Paddle/pull/71777), [#71837](https://github.com/PaddlePaddle/Paddle/pull/71837), [#71834](https://github.com/PaddlePaddle/Paddle/pull/71834), [#71950](https://github.com/PaddlePaddle/Paddle/pull/71950), [#71960](https://github.com/PaddlePaddle/Paddle/pull/71960), [#72103](https://github.com/PaddlePaddle/Paddle/pull/72103), [#70652](https://github.com/PaddlePaddle/Paddle/pull/70652), [#72313](https://github.com/PaddlePaddle/Paddle/pull/72313), [#72405](https://github.com/PaddlePaddle/Paddle/pull/72405), [#72581](https://github.com/PaddlePaddle/Paddle/pull/72581), [#73418](https://github.com/PaddlePaddle/Paddle/pull/73418) - 大Tensor支持扩展：扩展算子对超大尺寸Tensor的支持，包括数学运算（lerp/mean/bmm/trapezoid）、张量操作（arg_min_max/diag/prelu）、填充（pad）、比较（allclose/isclose）及融合算子（softmax_mask_fuse）等，解决混合精度训练中的兼容性问题。 [#71916](https://github.com/PaddlePaddle/Paddle/pull/71916), [#71970](https://github.com/PaddlePaddle/Paddle/pull/71970), [#72516](https://github.com/PaddlePaddle/Paddle/pull/72516), [#72517](https://github.com/PaddlePaddle/Paddle/pull/72517), [#72638](https://github.com/PaddlePaddle/Paddle/pull/72638), [#72652](https://github.com/PaddlePaddle/Paddle/pull/72652), [#73046](https://github.com/PaddlePaddle/Paddle/pull/73046), [#73093](https://github.com/PaddlePaddle/Paddle/pull/73093), [#73136](https://github.com/PaddlePaddle/Paddle/pull/73136), [#72679](https://github.com/PaddlePaddle/Paddle/pull/72679), [#73174](https://github.com/PaddlePaddle/Paddle/pull/73174), [#73198](https://github.com/PaddlePaddle/Paddle/pull/73198), [#73121](https://github.com/PaddlePaddle/Paddle/pull/73121), [#73096](https://github.com/PaddlePaddle/Paddle/pull/73096), [#73261](https://github.com/PaddlePaddle/Paddle/pull/73261), [#73201](https://github.com/PaddlePaddle/Paddle/pull/73201), [#73291](https://github.com/PaddlePaddle/Paddle/pull/73291), [#73373](https://github.com/PaddlePaddle/Paddle/pull/73373), [#73318](https://github.com/PaddlePaddle/Paddle/pull/73318), [#73436](https://github.com/PaddlePaddle/Paddle/pull/73436), [#72705](https://github.com/PaddlePaddle/Paddle/pull/72705), [#72276](https://github.com/PaddlePaddle/Paddle/pull/72276), [#73135](https://github.com/PaddlePaddle/Paddle/pull/73135), [#73304](https://github.com/PaddlePaddle/Paddle/pull/73304), [#73381](https://github.com/PaddlePaddle/Paddle/pull/73381), [#72712](https://github.com/PaddlePaddle/Paddle/pull/72712), [#72717](https://github.com/PaddlePaddle/Paddle/pull/72717), [#72634](https://github.com/PaddlePaddle/Paddle/pull/72634), [#72562](https://github.com/PaddlePaddle/Paddle/pull/72562), [#72628](https://github.com/PaddlePaddle/Paddle/pull/72628), [#72706](https://github.com/PaddlePaddle/Paddle/pull/72706), [#72831](https://github.com/PaddlePaddle/Paddle/pull/72831), [#72888](https://github.com/PaddlePaddle/Paddle/pull/72888), [#72753](https://github.com/PaddlePaddle/Paddle/pull/72753), [#72931](https://github.com/PaddlePaddle/Paddle/pull/72931), [#73021](https://github.com/PaddlePaddle/Paddle/pull/73021), [#73064](https://github.com/PaddlePaddle/Paddle/pull/73064), [#73069](https://github.com/PaddlePaddle/Paddle/pull/73069), [#73153](https://github.com/PaddlePaddle/Paddle/pull/73153), [#73118](https://github.com/PaddlePaddle/Paddle/pull/73118), [#73252](https://github.com/PaddlePaddle/Paddle/pull/73252), [#73253](https://github.com/PaddlePaddle/Paddle/pull/73253), [#73262](https://github.com/PaddlePaddle/Paddle/pull/73262), [#73259](https://github.com/PaddlePaddle/Paddle/pull/73259), [#73288](https://github.com/PaddlePaddle/Paddle/pull/73288), [#73105](https://github.com/PaddlePaddle/Paddle/pull/73105), [#73275](https://github.com/PaddlePaddle/Paddle/pull/73275), [#73284](https://github.com/PaddlePaddle/Paddle/pull/73284), [#73110](https://github.com/PaddlePaddle/Paddle/pull/73110), [#73335](https://github.com/PaddlePaddle/Paddle/pull/73335), [#73342](https://github.com/PaddlePaddle/Paddle/pull/73342), [#73447](https://github.com/PaddlePaddle/Paddle/pull/73447), [#73460](https://github.com/PaddlePaddle/Paddle/pull/73460), [#73194](https://github.com/PaddlePaddle/Paddle/pull/73194) - 0-Size Tensor问题修复：修复0-Size Tensor导致的计算异常，覆盖池化（max_pool1d/lp_pool1d）、排序（matrix_rank）、统计（std/nanmedian）及元素级操作（elementwise compare）等，确保极端输入场景下的数值稳定性与API一致性。 [#71961](https://github.com/PaddlePaddle/Paddle/pull/71961), [#72017](https://github.com/PaddlePaddle/Paddle/pull/72017), [#72785](https://github.com/PaddlePaddle/Paddle/pull/72785), [#73214](https://github.com/PaddlePaddle/Paddle/pull/73214), [#73263](https://github.com/PaddlePaddle/Paddle/pull/73263), [#73267](https://github.com/PaddlePaddle/Paddle/pull/73267), [#73280](https://github.com/PaddlePaddle/Paddle/pull/73280), [#72444](https://github.com/PaddlePaddle/Paddle/pull/72444), [#72437](https://github.com/PaddlePaddle/Paddle/pull/72437), [#72460](https://github.com/PaddlePaddle/Paddle/pull/72460), [#73090](https://github.com/PaddlePaddle/Paddle/pull/73090), [#73516](https://github.com/PaddlePaddle/Paddle/pull/73516), [#72807](https://github.com/PaddlePaddle/Paddle/pull/72807), [#72799](https://github.com/PaddlePaddle/Paddle/pull/72799), [#72800](https://github.com/PaddlePaddle/Paddle/pull/72800), [#72809](https://github.com/PaddlePaddle/Paddle/pull/72809), [#73497](https://github.com/PaddlePaddle/Paddle/pull/73497) - API功能增强与兼容性：新增对Python标准库类型（dataclasses）的支持，扩展API数据类型兼容性（bfloat16参数创建、-1维自动推断），修复NumPy API交互错误，优化BatchNorm内存布局。 [#72059](https://github.com/PaddlePaddle/Paddle/pull/72059), [#72283](https://github.com/PaddlePaddle/Paddle/pull/72283), [#72451](https://github.com/PaddlePaddle/Paddle/pull/72451), [#72512](https://github.com/PaddlePaddle/Paddle/pull/72512), [#72618](https://github.com/PaddlePaddle/Paddle/pull/72618), [#72976](https://github.com/PaddlePaddle/Paddle/pull/72976), [#73084](https://github.com/PaddlePaddle/Paddle/pull/73084), [#73205](https://github.com/PaddlePaddle/Paddle/pull/73205), [#73250](https://github.com/PaddlePaddle/Paddle/pull/73250), [#73111](https://github.com/PaddlePaddle/Paddle/pull/73111), [#73260](https://github.com/PaddlePaddle/Paddle/pull/73260), [#72094](https://github.com/PaddlePaddle/Paddle/pull/72094), [#71844](https://github.com/PaddlePaddle/Paddle/pull/71844), [#71357](https://github.com/PaddlePaddle/Paddle/pull/71357) - 内存管理与错误修复：解决内存越界（set_value/nonzero）、空指针（data nullptr）、CUDA graph分配失败等高危问题，修复梯度裁剪（clip_grad）、张量赋值（assign）、广播（broadcast）等核心操作的内存泄漏与计算错误，优化NPU异步执行与预测器GIL释放逻辑，提升系统健壮性。 [#71895](https://github.com/PaddlePaddle/Paddle/pull/71895), [#72101](https://github.com/PaddlePaddle/Paddle/pull/72101), [#72133](https://github.com/PaddlePaddle/Paddle/pull/72133), [#72149](https://github.com/PaddlePaddle/Paddle/pull/72149), [#72176](https://github.com/PaddlePaddle/Paddle/pull/72176), [#72314](https://github.com/PaddlePaddle/Paddle/pull/72314), [#72256](https://github.com/PaddlePaddle/Paddle/pull/72256), [#72757](https://github.com/PaddlePaddle/Paddle/pull/72757), [#72749](https://github.com/PaddlePaddle/Paddle/pull/72749), [#72792](https://github.com/PaddlePaddle/Paddle/pull/72792), [#72815](https://github.com/PaddlePaddle/Paddle/pull/72815), [#72819](https://github.com/PaddlePaddle/Paddle/pull/72819), [#72958](https://github.com/PaddlePaddle/Paddle/pull/72958), [#73023](https://github.com/PaddlePaddle/Paddle/pull/73023), [#73103](https://github.com/PaddlePaddle/Paddle/pull/73103), [#73014](https://github.com/PaddlePaddle/Paddle/pull/73014), [#73137](https://github.com/PaddlePaddle/Paddle/pull/73137), [#73256](https://github.com/PaddlePaddle/Paddle/pull/73256), [#73211](https://github.com/PaddlePaddle/Paddle/pull/73211), [#73251](https://github.com/PaddlePaddle/Paddle/pull/73251), [#73210](https://github.com/PaddlePaddle/Paddle/pull/73210), [#73415](https://github.com/PaddlePaddle/Paddle/pull/73415), [#73206](https://github.com/PaddlePaddle/Paddle/pull/73206), [#71983](https://github.com/PaddlePaddle/Paddle/pull/71983), [#72485](https://github.com/PaddlePaddle/Paddle/pull/72485), [#72561](https://github.com/PaddlePaddle/Paddle/pull/72561) - 其他重要修复：修复科学计算、save/load等模块缺陷，改进Slice算子内核配置，优化动态shaoe推断的回退策略，完善异常抛出与类型检查逻辑等。 [#71810](https://github.com/PaddlePaddle/Paddle/pull/71810), [#72246](https://github.com/PaddlePaddle/Paddle/pull/72246), [#72378](https://github.com/PaddlePaddle/Paddle/pull/72378), [#72467](https://github.com/PaddlePaddle/Paddle/pull/72467), [#72635](https://github.com/PaddlePaddle/Paddle/pull/72635), [#72751](https://github.com/PaddlePaddle/Paddle/pull/72751), [#72044](https://github.com/PaddlePaddle/Paddle/pull/72044), [#72051](https://github.com/PaddlePaddle/Paddle/pull/72051), [#73231](https://github.com/PaddlePaddle/Paddle/pull/73231), [#73109](https://github.com/PaddlePaddle/Paddle/pull/73109) - SOT 相关问题修复, [#71932](https://github.com/PaddlePaddle/Paddle/pull/71932), [#71971](https://github.com/PaddlePaddle/Paddle/pull/71971), [#72194](https://github.com/PaddlePaddle/Paddle/pull/72194), [#72288](https://github.com/PaddlePaddle/Paddle/pull/72288), [#72306](https://github.com/PaddlePaddle/Paddle/pull/72306), [#72367](https://github.com/PaddlePaddle/Paddle/pull/72367), [#72495](https://github.com/PaddlePaddle/Paddle/pull/72495), [#72522](https://github.com/PaddlePaddle/Paddle/pull/72522), [#72704](https://github.com/PaddlePaddle/Paddle/pull/72704), [#72631](https://github.com/PaddlePaddle/Paddle/pull/72631), [#72737](https://github.com/PaddlePaddle/Paddle/pull/72737), [#73067](https://github.com/PaddlePaddle/Paddle/pull/73067), [#73030](https://github.com/PaddlePaddle/Paddle/pull/73030), [#73059](https://github.com/PaddlePaddle/Paddle/pull/73059), [#73282](https://github.com/PaddlePaddle/Paddle/pull/73282), [#73511](https://github.com/PaddlePaddle/Paddle/pull/73511), [#73526](https://github.com/PaddlePaddle/Paddle/pull/73526), [#73549](https://github.com/PaddlePaddle/Paddle/pull/73549), [#73515](https://github.com/PaddlePaddle/Paddle/pull/73515) ### 功能增强 - Paddle API 0-size 机制建设。 [#72721](https://github.com/PaddlePaddle/Paddle/pull/72721), [#72756](https://github.com/PaddlePaddle/Paddle/pull/72756), [#72790](https://github.com/PaddlePaddle/Paddle/pull/72790), [#72806](https://github.com/PaddlePaddle/Paddle/pull/72806), [#72764](https://github.com/PaddlePaddle/Paddle/pull/72764), [#72786](https://github.com/PaddlePaddle/Paddle/pull/72786), [#72853](https://github.com/PaddlePaddle/Paddle/pull/72853), [#72826](https://github.com/PaddlePaddle/Paddle/pull/72826), [#72851](https://github.com/PaddlePaddle/Paddle/pull/72851), [#72928](https://github.com/PaddlePaddle/Paddle/pull/72928), [#72912](https://github.com/PaddlePaddle/Paddle/pull/72912), [#72922](https://github.com/PaddlePaddle/Paddle/pull/72922), [#72924](https://github.com/PaddlePaddle/Paddle/pull/72924), [#72887](https://github.com/PaddlePaddle/Paddle/pull/72887), [#72921](https://github.com/PaddlePaddle/Paddle/pull/72921), [#72906](https://github.com/PaddlePaddle/Paddle/pull/72906), [#72895](https://github.com/PaddlePaddle/Paddle/pull/72895), [#72821](https://github.com/PaddlePaddle/Paddle/pull/72821), [#72914](https://github.com/PaddlePaddle/Paddle/pull/72914), [#72936](https://github.com/PaddlePaddle/Paddle/pull/72936), [#72943](https://github.com/PaddlePaddle/Paddle/pull/72943), [#72694](https://github.com/PaddlePaddle/Paddle/pull/72694), [#72919](https://github.com/PaddlePaddle/Paddle/pull/72919), [#72940](https://github.com/PaddlePaddle/Paddle/pull/72940), [#72820](https://github.com/PaddlePaddle/Paddle/pull/72820), [#72934](https://github.com/PaddlePaddle/Paddle/pull/72934), [#72975](https://github.com/PaddlePaddle/Paddle/pull/72975), [#72872](https://github.com/PaddlePaddle/Paddle/pull/72872), [#72984](https://github.com/PaddlePaddle/Paddle/pull/72984), [#72988](https://github.com/PaddlePaddle/Paddle/pull/72988), [#72972](https://github.com/PaddlePaddle/Paddle/pull/72972), [#72977](https://github.com/PaddlePaddle/Paddle/pull/72977), [#72937](https://github.com/PaddlePaddle/Paddle/pull/72937), [#73086](https://github.com/PaddlePaddle/Paddle/pull/73086), [#73042](https://github.com/PaddlePaddle/Paddle/pull/73042), [#73017](https://github.com/PaddlePaddle/Paddle/pull/73017), [#73044](https://github.com/PaddlePaddle/Paddle/pull/73044), [#73077](https://github.com/PaddlePaddle/Paddle/pull/73077), [#73108](https://github.com/PaddlePaddle/Paddle/pull/73108), [#73027](https://github.com/PaddlePaddle/Paddle/pull/73027), [#72970](https://github.com/PaddlePaddle/Paddle/pull/72970), [#73008](https://github.com/PaddlePaddle/Paddle/pull/73008), [#72996](https://github.com/PaddlePaddle/Paddle/pull/72996), [#73165](https://github.com/PaddlePaddle/Paddle/pull/73165), [#73166](https://github.com/PaddlePaddle/Paddle/pull/73166), [#73170](https://github.com/PaddlePaddle/Paddle/pull/73170), [#73122](https://github.com/PaddlePaddle/Paddle/pull/73122), [#73204](https://github.com/PaddlePaddle/Paddle/pull/73204), [#73207](https://github.com/PaddlePaddle/Paddle/pull/73207), [#73186](https://github.com/PaddlePaddle/Paddle/pull/73186), [#73197](https://github.com/PaddlePaddle/Paddle/pull/73197), [#73168](https://github.com/PaddlePaddle/Paddle/pull/73168), [#73172](https://github.com/PaddlePaddle/Paddle/pull/73172), [#73125](https://github.com/PaddlePaddle/Paddle/pull/73125), [#73181](https://github.com/PaddlePaddle/Paddle/pull/73181), [#73270](https://github.com/PaddlePaddle/Paddle/pull/73270), [#73028](https://github.com/PaddlePaddle/Paddle/pull/73028), [#73094](https://github.com/PaddlePaddle/Paddle/pull/73094), [#73180](https://github.com/PaddlePaddle/Paddle/pull/73180), [#73276](https://github.com/PaddlePaddle/Paddle/pull/73276), [#73333](https://github.com/PaddlePaddle/Paddle/pull/73333), [#73341](https://github.com/PaddlePaddle/Paddle/pull/73341), [#73299](https://github.com/PaddlePaddle/Paddle/pull/73299), [#73346](https://github.com/PaddlePaddle/Paddle/pull/73346), [#73361](https://github.com/PaddlePaddle/Paddle/pull/73361), [#73375](https://github.com/PaddlePaddle/Paddle/pull/73375), [#73152](https://github.com/PaddlePaddle/Paddle/pull/73152), [#73377](https://github.com/PaddlePaddle/Paddle/pull/73377), [#73355](https://github.com/PaddlePaddle/Paddle/pull/73355), [#73382](https://github.com/PaddlePaddle/Paddle/pull/73382), [#73385](https://github.com/PaddlePaddle/Paddle/pull/73385), [#73386](https://github.com/PaddlePaddle/Paddle/pull/73386), [#73352](https://github.com/PaddlePaddle/Paddle/pull/73352), [#73387](https://github.com/PaddlePaddle/Paddle/pull/73387), [#73401](https://github.com/PaddlePaddle/Paddle/pull/73401), [#73384](https://github.com/PaddlePaddle/Paddle/pull/73384), [#73450](https://github.com/PaddlePaddle/Paddle/pull/73450), [#73437](https://github.com/PaddlePaddle/Paddle/pull/73437), [#73503](https://github.com/PaddlePaddle/Paddle/pull/73503), [#73507](https://github.com/PaddlePaddle/Paddle/pull/73507), [#73477](https://github.com/PaddlePaddle/Paddle/pull/73477), [#73513](https://github.com/PaddlePaddle/Paddle/pull/73513), [#73525](https://github.com/PaddlePaddle/Paddle/pull/73525), [#73528](https://github.com/PaddlePaddle/Paddle/pull/73528), [#73517](https://github.com/PaddlePaddle/Paddle/pull/73517), [#72898](https://github.com/PaddlePaddle/Paddle/pull/72898), [#72880](https://github.com/PaddlePaddle/Paddle/pull/72880), [#72864](https://github.com/PaddlePaddle/Paddle/pull/72864), [#72993](https://github.com/PaddlePaddle/Paddle/pull/72993), [#72954](https://github.com/PaddlePaddle/Paddle/pull/72954), [#72866](https://github.com/PaddlePaddle/Paddle/pull/72866), [#72878](https://github.com/PaddlePaddle/Paddle/pull/72878), [#72889](https://github.com/PaddlePaddle/Paddle/pull/72889), [#72861](https://github.com/PaddlePaddle/Paddle/pull/72861), [#72837](https://github.com/PaddlePaddle/Paddle/pull/72837) - SOT相关提升：增强了功能（如NumPy互操作性和super支持）、改进训练稳定性，修复多个问题以提升代码健壮性, [#71763](https://github.com/PaddlePaddle/Paddle/pull/71763), [#71666](https://github.com/PaddlePaddle/Paddle/pull/71666), [#71858](https://github.com/PaddlePaddle/Paddle/pull/71858), [#71865](https://github.com/PaddlePaddle/Paddle/pull/71865), [#72474](https://github.com/PaddlePaddle/Paddle/pull/72474), [#72154](https://github.com/PaddlePaddle/Paddle/pull/72154), [#72784](https://github.com/PaddlePaddle/Paddle/pull/72784), [#72956](https://github.com/PaddlePaddle/Paddle/pull/72956), [#73038](https://github.com/PaddlePaddle/Paddle/pull/73038), [#73066](https://github.com/PaddlePaddle/Paddle/pull/73066), [#73287](https://github.com/PaddlePaddle/Paddle/pull/73287), [#73278](https://github.com/PaddlePaddle/Paddle/pull/73278), [#73332](https://github.com/PaddlePaddle/Paddle/pull/73332), [#73372](https://github.com/PaddlePaddle/Paddle/pull/73372), [#73412](https://github.com/PaddlePaddle/Paddle/pull/73412), [#73407](https://github.com/PaddlePaddle/Paddle/pull/73407), [#73506](https://github.com/PaddlePaddle/Paddle/pull/73506) - 代码风格重构：通过代码重构及跨平台内核行为统一，提升代码质量与可维护性，并新增了YAML格式预提交检查工具, [#72216](https://github.com/PaddlePaddle/Paddle/pull/72216), [#72360](https://github.com/PaddlePaddle/Paddle/pull/72360), [#72816](https://github.com/PaddlePaddle/Paddle/pull/72816), [#72969](https://github.com/PaddlePaddle/Paddle/pull/72969), [#73106](https://github.com/PaddlePaddle/Paddle/pull/73106), [#72825](https://github.com/PaddlePaddle/Paddle/pull/72825), [#73150](https://github.com/PaddlePaddle/Paddle/pull/73150), [#73151](https://github.com/PaddlePaddle/Paddle/pull/73151), [#73158](https://github.com/PaddlePaddle/Paddle/pull/73158), [#73101](https://github.com/PaddlePaddle/Paddle/pull/73101), [#73326](https://github.com/PaddlePaddle/Paddle/pull/73326), [#72580](https://github.com/PaddlePaddle/Paddle/pull/72580), [#72424](https://github.com/PaddlePaddle/Paddle/pull/72424) - Paddle CPU/GPU Kernel 精度问题推全。 [#72879](https://github.com/PaddlePaddle/Paddle/pull/72879), [#72894](https://github.com/PaddlePaddle/Paddle/pull/72894), [#73012](https://github.com/PaddlePaddle/Paddle/pull/73012), [#72973](https://github.com/PaddlePaddle/Paddle/pull/72973), [#73018](https://github.com/PaddlePaddle/Paddle/pull/73018), [#72965](https://github.com/PaddlePaddle/Paddle/pull/72965), [#73128](https://github.com/PaddlePaddle/Paddle/pull/73128), [#73229](https://github.com/PaddlePaddle/Paddle/pull/73229), [#72992](https://github.com/PaddlePaddle/Paddle/pull/72992), [#73344](https://github.com/PaddlePaddle/Paddle/pull/73344), [#73274](https://github.com/PaddlePaddle/Paddle/pull/73274), [#73295](https://github.com/PaddlePaddle/Paddle/pull/73295), [#73293](https://github.com/PaddlePaddle/Paddle/pull/73293), [#73317](https://github.com/PaddlePaddle/Paddle/pull/73317), [#73320](https://github.com/PaddlePaddle/Paddle/pull/73320), [#73454](https://github.com/PaddlePaddle/Paddle/pull/73454), [#73492](https://github.com/PaddlePaddle/Paddle/pull/73492), [#73535](https://github.com/PaddlePaddle/Paddle/pull/73535) - slice 问题修复：修复了slice相关问题，包括索引逻辑、性能优化等, [#72644](https://github.com/PaddlePaddle/Paddle/pull/72644), [#72676](https://github.com/PaddlePaddle/Paddle/pull/72676), [#72838](https://github.com/PaddlePaddle/Paddle/pull/72838), [#72966](https://github.com/PaddlePaddle/Paddle/pull/72966), [#73095](https://github.com/PaddlePaddle/Paddle/pull/73095), [#72840](https://github.com/PaddlePaddle/Paddle/pull/72840), [#73112](https://github.com/PaddlePaddle/Paddle/pull/73112), [#73367](https://github.com/PaddlePaddle/Paddle/pull/73367), [#73390](https://github.com/PaddlePaddle/Paddle/pull/73390), [#73307](https://github.com/PaddlePaddle/Paddle/pull/73307), [#73465](https://github.com/PaddlePaddle/Paddle/pull/73465), [#73362](https://github.com/PaddlePaddle/Paddle/pull/73362), [#72733](https://github.com/PaddlePaddle/Paddle/pull/72733), [#72886](https://github.com/PaddlePaddle/Paddle/pull/72886) - 性能优化：通过优化索引逻辑、性能提升等手段，提升整体性能表现, [#72707](https://github.com/PaddlePaddle/Paddle/pull/72707), [#73485](https://github.com/PaddlePaddle/Paddle/pull/73485) - 其他重要提升：包括动态shape支持、修复 meshgrid 并增加单元测试、升级 CUB 至 2.1.0 版本、改进 FP8 数值处理、优化 CUDA 图共享池机制、移除 ShadowFeedOp 以简化数据流、增强 PIR 模型保存/加载的版本兼容性、修复 flip 和 reverse 内核问题、改进 paddle.angle 的 NaN 传播逻辑、引入异步 GC 检查机制、优化 Dy2St 的 Scope 无锁接口、清理未使用的第三方依赖（absl），并进一步推进 PHI 与 Fluid 的解耦，提升框架的稳定性、性能和扩展性。 [#72356](https://github.com/PaddlePaddle/Paddle/pull/72356), [#72380](https://github.com/PaddlePaddle/Paddle/pull/72380), [#72633](https://github.com/PaddlePaddle/Paddle/pull/72633), [#72794](https://github.com/PaddlePaddle/Paddle/pull/72794), [#72917](https://github.com/PaddlePaddle/Paddle/pull/72917), [#72920](https://github.com/PaddlePaddle/Paddle/pull/72920), [#72945](https://github.com/PaddlePaddle/Paddle/pull/72945), [#72620](https://github.com/PaddlePaddle/Paddle/pull/72620), [#73011](https://github.com/PaddlePaddle/Paddle/pull/73011), [#73051](https://github.com/PaddlePaddle/Paddle/pull/73051), [#73052](https://github.com/PaddlePaddle/Paddle/pull/73052), [#73075](https://github.com/PaddlePaddle/Paddle/pull/73075), [#73176](https://github.com/PaddlePaddle/Paddle/pull/73176), [#73191](https://github.com/PaddlePaddle/Paddle/pull/73191), [#73337](https://github.com/PaddlePaddle/Paddle/pull/73337), [#73311](https://github.com/PaddlePaddle/Paddle/pull/73311), [#73173](https://github.com/PaddlePaddle/Paddle/pull/73173), [#73239](https://github.com/PaddlePaddle/Paddle/pull/73239), [#73448](https://github.com/PaddlePaddle/Paddle/pull/73448), [#73478](https://github.com/PaddlePaddle/Paddle/pull/73478), [#73522](https://github.com/PaddlePaddle/Paddle/pull/73522), [#73369](https://github.com/PaddlePaddle/Paddle/pull/73369) ### 性能提升 - SOT相关：通过优化Guard条件机制、增强动态shape处理能力及新增no_grad支持等改进，提升了执行效率并扩展了功能特性，同时优化了代码结构与性能表现。 [#70362](https://github.com/PaddlePaddle/Paddle/pull/70362), [#70154](https://github.com/PaddlePaddle/Paddle/pull/70154), [#71748](https://github.com/PaddlePaddle/Paddle/pull/71748), [#72004](https://github.com/PaddlePaddle/Paddle/pull/72004), [#72159](https://github.com/PaddlePaddle/Paddle/pull/72159), [#72174](https://github.com/PaddlePaddle/Paddle/pull/72174), [#71994](https://github.com/PaddlePaddle/Paddle/pull/71994), [#72250](https://github.com/PaddlePaddle/Paddle/pull/72250), [#72285](https://github.com/PaddlePaddle/Paddle/pull/72285), [#72322](https://github.com/PaddlePaddle/Paddle/pull/72322), [#72272](https://github.com/PaddlePaddle/Paddle/pull/72272), [#72417](https://github.com/PaddlePaddle/Paddle/pull/72417), [#72438](https://github.com/PaddlePaddle/Paddle/pull/72438), [#72462](https://github.com/PaddlePaddle/Paddle/pull/72462), [#72463](https://github.com/PaddlePaddle/Paddle/pull/72463), [#72503](https://github.com/PaddlePaddle/Paddle/pull/72503), [#72501](https://github.com/PaddlePaddle/Paddle/pull/72501), [#72521](https://github.com/PaddlePaddle/Paddle/pull/72521), [#72509](https://github.com/PaddlePaddle/Paddle/pull/72509), [#72544](https://github.com/PaddlePaddle/Paddle/pull/72544), [#73469](https://github.com/PaddlePaddle/Paddle/pull/73469), [#73471](https://github.com/PaddlePaddle/Paddle/pull/73471), [#73555](https://github.com/PaddlePaddle/Paddle/pull/73555) ### 废弃 - 代码清理：清理 Python 3.8 支持声明，并完成了相关代码清理、依赖精简及语法现代化更新，以优化代码维护性与兼容性。 [#71815](https://github.com/PaddlePaddle/Paddle/pull/71815), [#72802](https://github.com/PaddlePaddle/Paddle/pull/72802), [#72856](https://github.com/PaddlePaddle/Paddle/pull/72856), [#72854](https://github.com/PaddlePaddle/Paddle/pull/72854), [#72855](https://github.com/PaddlePaddle/Paddle/pull/72855), [#72873](https://github.com/PaddlePaddle/Paddle/pull/72873), [#72870](https://github.com/PaddlePaddle/Paddle/pull/72870), [#72868](https://github.com/PaddlePaddle/Paddle/pull/72868), [#72891](https://github.com/PaddlePaddle/Paddle/pull/72891) ### 开发者相关 - 优化了CINN后端集成与动态 shape 处理逻辑，通过代码结构重构与测试强化提升了框架稳定性，并新增调试日志功能以增强可维护性。 [#71817](https://github.com/PaddlePaddle/Paddle/pull/71817), [#71896](https://github.com/PaddlePaddle/Paddle/pull/71896), [#71984](https://github.com/PaddlePaddle/Paddle/pull/71984), [#72067](https://github.com/PaddlePaddle/Paddle/pull/72067), [#72165](https://github.com/PaddlePaddle/Paddle/pull/72165), [#72207](https://github.com/PaddlePaddle/Paddle/pull/72207), [#72235](https://github.com/PaddlePaddle/Paddle/pull/72235), [#72273](https://github.com/PaddlePaddle/Paddle/pull/72273), [#72326](https://github.com/PaddlePaddle/Paddle/pull/72326), [#72400](https://github.com/PaddlePaddle/Paddle/pull/72400), [#72381](https://github.com/PaddlePaddle/Paddle/pull/72381), [#72560](https://github.com/PaddlePaddle/Paddle/pull/72560), [#72783](https://github.com/PaddlePaddle/Paddle/pull/72783), [#73530](https://github.com/PaddlePaddle/Paddle/pull/73530) ### 其他 - 其他：新增CPU 部分kernel对FP16/BF16数据类型的内核支持，优化测试模块错误处理与容差配置等。 [#71764](https://github.com/PaddlePaddle/Paddle/pull/71764), [#71951](https://github.com/PaddlePaddle/Paddle/pull/71951), [#72944](https://github.com/PaddlePaddle/Paddle/pull/72944) ## 3. 编译器架构优化编译器性能和增加稳定性 ### 性能优化 - 支持训练场景的Layout自动转换优化。（[#71891](https://github.com/PaddlePaddle/Paddle/pull/71891)） - 后端新增了argmin、argmax、arange等算子的Kernel编译优化。([#71956](https://github.com/PaddlePaddle/Paddle/pull/71956), [#72598](https://github.com/PaddlePaddle/Paddle/pull/72598))) - 支持矩阵乘的融合优化。([#72846](https://github.com/PaddlePaddle/Paddle/pull/72846)) - 优化部分算子 Kernel 计算性能。([#72871](https://github.com/PaddlePaddle/Paddle/pull/72871)) ### Bug修复修复各类场景下的一些处理逻辑 Bug。（[#71813](https://github.com/PaddlePaddle/Paddle/pull/71813), [#71886](https://github.com/PaddlePaddle/Paddle/pull/71886), [#71927](https://github.com/PaddlePaddle/Paddle/pull/71927), [#71915](https://github.com/PaddlePaddle/Paddle/pull/71915), [#71946](https://github.com/PaddlePaddle/Paddle/pull/71946), [#71949](https://github.com/PaddlePaddle/Paddle/pull/71949), [#71955](https://github.com/PaddlePaddle/Paddle/pull/71955), [#71942](https://github.com/PaddlePaddle/Paddle/pull/71942), [#71939](https://github.com/PaddlePaddle/Paddle/pull/71939), [#71973](https://github.com/PaddlePaddle/Paddle/pull/71973), [#72001](https://github.com/PaddlePaddle/Paddle/pull/72001), [#72020](https://github.com/PaddlePaddle/Paddle/pull/72020), [#72014](https://github.com/PaddlePaddle/Paddle/pull/72014), [#72021](https://github.com/PaddlePaddle/Paddle/pull/72021), [#72027](https://github.com/PaddlePaddle/Paddle/pull/72027), [#72061](https://github.com/PaddlePaddle/Paddle/pull/72061), [#72025](https://github.com/PaddlePaddle/Paddle/pull/72025), [#72095](https://github.com/PaddlePaddle/Paddle/pull/72095), [#72108](https://github.com/PaddlePaddle/Paddle/pull/72108), [#72132](https://github.com/PaddlePaddle/Paddle/pull/72132), [#71985](https://github.com/PaddlePaddle/Paddle/pull/71985), [#72106](https://github.com/PaddlePaddle/Paddle/pull/72106), [#72140](https://github.com/PaddlePaddle/Paddle/pull/72140), [#72167](https://github.com/PaddlePaddle/Paddle/pull/72167), [#72037](https://github.com/PaddlePaddle/Paddle/pull/72037), [#72178](https://github.com/PaddlePaddle/Paddle/pull/72178), [#72143](https://github.com/PaddlePaddle/Paddle/pull/72143), [#72175](https://github.com/PaddlePaddle/Paddle/pull/72175), [#72191](https://github.com/PaddlePaddle/Paddle/pull/72191), [#72213](https://github.com/PaddlePaddle/Paddle/pull/72213), [#72189](https://github.com/PaddlePaddle/Paddle/pull/72189), [#72214](https://github.com/PaddlePaddle/Paddle/pull/72214), [#72166](https://github.com/PaddlePaddle/Paddle/pull/72166), [#72180](https://github.com/PaddlePaddle/Paddle/pull/72180), [#72284](https://github.com/PaddlePaddle/Paddle/pull/72284), [#72267](https://github.com/PaddlePaddle/Paddle/pull/72267), [#72348](https://github.com/PaddlePaddle/Paddle/pull/72348), [#72332](https://github.com/PaddlePaddle/Paddle/pull/72332), [#72307](https://github.com/PaddlePaddle/Paddle/pull/72307), [#72353](https://github.com/PaddlePaddle/Paddle/pull/72353), [#72204](https://github.com/PaddlePaddle/Paddle/pull/72204), [#72457](https://github.com/PaddlePaddle/Paddle/pull/72457), [#72426](https://github.com/PaddlePaddle/Paddle/pull/72426), [#72536](https://github.com/PaddlePaddle/Paddle/pull/72536), [#72541](https://github.com/PaddlePaddle/Paddle/pull/72541), [#72365](https://github.com/PaddlePaddle/Paddle/pull/72365), [#72621](https://github.com/PaddlePaddle/Paddle/pull/72621), [#72630](https://github.com/PaddlePaddle/Paddle/pull/72630), [#72669](https://github.com/PaddlePaddle/Paddle/pull/72669), [#72682](https://github.com/PaddlePaddle/Paddle/pull/72682), [#72732](https://github.com/PaddlePaddle/Paddle/pull/72732), [#72811](https://github.com/PaddlePaddle/Paddle/pull/72811), [#72941](https://github.com/PaddlePaddle/Paddle/pull/72941), [#72795](https://github.com/PaddlePaddle/Paddle/pull/72795), [#73536](https://github.com/PaddlePaddle/Paddle/pull/73536)） ## 4. 自动并行架构在3.1 版本中,我们对自动并行架构进一步打磨,以提高自动并行易用性和动态图性能。具体地,我们完善了自动并行核心机制,包括新增了多个算子的切分推导规则,支持分布式张量的同一维度被多个mesh维度切分,支持动态图并行策略（PP,CP,SEP,TP-CONV）等。同时,对动态图自动并行系统地做了性能优化,在Llama等系列模型上性能基本持平手动并行的性能。 ### 功能改进 - 支持分布式张量的同一维度被多个mesh维度切分。 [#73233](https://github.com/PaddlePaddle/Paddle/pull/73233) - 支持自动并行通信拓扑描述ProcessMesh转换为手动并行通信组。 [#72052](https://github.com/PaddlePaddle/Paddle/pull/72052) - 支持任意可序列化python object的send/recv。 [#72098](https://github.com/PaddlePaddle/Paddle/pull/72098) - 动态图并行策略补齐 - 支持流水线并行策略1F1B和VPP调度。 [#72155](https://github.com/PaddlePaddle/Paddle/pull/72155),[#72480](https://github.com/PaddlePaddle/Paddle/pull/72480),[#72179](https://github.com/PaddlePaddle/Paddle/pull/72179) - 支持长文并行策略。[#73195](https://github.com/PaddlePaddle/Paddle/pull/73195) - 支持视觉并行策略。[#73063](https://github.com/PaddlePaddle/Paddle/pull/73063),[#73039](https://github.com/PaddlePaddle/Paddle/pull/73039) - 支持自动并行在数据并行维度的通信。[#72540](https://github.com/PaddlePaddle/Paddle/pull/72540) - 新增以下算子的切分推导规则 - `min`, `min_grad` [#72269](https://github.com/PaddlePaddle/Paddle/pull/72269) - `bitwise_or`,`atan2`,`fmax`,`fmin`,`reciprocal` [#72310](https://github.com/PaddlePaddle/Paddle/pull/72310) - `argmin`, `abs`, `cosh` [#72264](https://github.com/PaddlePaddle/Paddle/pull/72264) - `mean_all`, `mean_all_grad` [#72479](https://github.com/PaddlePaddle/Paddle/pull/72479) - `topk`, `topk_grad` [#72499](https://github.com/PaddlePaddle/Paddle/pull/72499) - `argsort` [#72388](https://github.com/PaddlePaddle/Paddle/pull/72388) - `round`, `mish`, `elu`, `selu`, `celu`, `stanh`, `softplus`, `softshrink`, `thresholded_relu`, `logit`, `nonzero` [#72312](https://github.com/PaddlePaddle/Paddle/pull/72312) - `unique ops` [#72824](https://github.com/PaddlePaddle/Paddle/pull/72824) - `put_along_axis` [#72766](https://github.com/PaddlePaddle/Paddle/pull/72766) - `round_grad`, `trunc_grad`, `ceil_grad`, `floor_grad`, `poisson_grad` [#72677](https://github.com/PaddlePaddle/Paddle/pull/72677) - `log_softmax`, `cummax`, `cummin` [#72720](https://github.com/PaddlePaddle/Paddle/pull/72720) - `unary` [#72177](https://github.com/PaddlePaddle/Paddle/pull/72177) - `unary_grad` [#72260](https://github.com/PaddlePaddle/Paddle/pull/72260) - `index_select`, `index_select_grad` [#72727](https://github.com/PaddlePaddle/Paddle/pull/72727) - `roll`, `roll_grad` [#72740](https://github.com/PaddlePaddle/Paddle/pull/72740) - `empty_like` [#73169](https://github.com/PaddlePaddle/Paddle/pull/73169) - `roi_align`, `roi_align_grad` [#72925](https://github.com/PaddlePaddle/Paddle/pull/72925) - `expand_as`, `expand_as_grad` [#73107](https://github.com/PaddlePaddle/Paddle/pull/73107) - `fused_gemm_epilogur` [#73126](https://github.com/PaddlePaddle/Paddle/pull/73126) - `label_smooth`, `label_smooth` [#72845](https://github.com/PaddlePaddle/Paddle/pull/72845) - `group_norm`, `group_norm_grad` [#72946](https://github.com/PaddlePaddle/Paddle/pull/72946) - `instance_norm`, `instance_norm_grad` [#72938](https://github.com/PaddlePaddle/Paddle/pull/72938) - `batch_norm`, `sync_batch_norm` [#72918](https://github.com/PaddlePaddle/Paddle/pull/72918) - `reduce_any` [#73175](https://github.com/PaddlePaddle/Paddle/pull/73175) - `fused_gemm_epilogue_rule` [#73494](https://github.com/PaddlePaddle/Paddle/pull/73494) ### 性能优化 * 支持分组切分并行的tensor_fusion优化策略和overlap优化策略。 [#72551](https://github.com/PaddlePaddle/Paddle/pull/72551), [#72902](https://github.com/PaddlePaddle/Paddle/pull/72902), [#73142](https://github.com/PaddlePaddle/Paddle/pull/73142),[#71785](https://github.com/PaddlePaddle/Paddle/pull/71785) * 优化reshard模块,以降低通信开销。[#71969](https://github.com/PaddlePaddle/Paddle/pull/71969), [#73024](https://github.com/PaddlePaddle/Paddle/pull/73024),[#71868](https://github.com/PaddlePaddle/Paddle/pull/71868) * 优化multiply 的切分推导规则,以降低通信开销。[#73408](https://github.com/PaddlePaddle/Paddle/pull/73408) * 优化分布式切分状态为 Partial 时反向通信,以降低通信开销。 [#73236](https://github.com/PaddlePaddle/Paddle/pull/73236) * 梯度更新时通信融合优化。 [#72120](https://github.com/PaddlePaddle/Paddle/pull/72120 )、[#72745](https://github.com/PaddlePaddle/Paddle/pull/72745) * 优化 gelu 切分推导,以降低通信开销。 [#73279](https://github.com/PaddlePaddle/Paddle/pull/73279) * 优化 fused_rms_norm 在输入有 Partial 状态时的切分推导规则,以减少通信和计算开销。 [#73054](https://github.com/PaddlePaddle/Paddle/pull/73054) ### Bug 修复 - 修复虚拟流水线并行策略在H卡上通信hang的bug。[#71104](https://github.com/PaddlePaddle/Paddle/pull/71104), [#73470](https://github.com/PaddlePaddle/Paddle/pull/73470) - 修复 save/load 的bug。 [#72023](https://github.com/PaddlePaddle/Paddle/pull/72023) - 修复 linear_fused_grad_add 策略在动态图模式下跑不通的bug。 [#72708](https://github.com/PaddlePaddle/Paddle/pull/72708)) - 修复 fused_rms_norm 算子跑不通和精度bug。 [#72663](https://github.com/PaddlePaddle/Paddle/pull/72663 ) - 修复 expand 算子切分推导规则的bug。[#73154](https://github.com/PaddlePaddle/Paddle/pull/73154) ### 其他 - 清理废弃代码,以便于维护代码。 [#71814](https://github.com/PaddlePaddle/Paddle/pull/71814),[#72538](https://github.com/PaddlePaddle/Paddle/pull/72538) - 新增API local_map,将分布式张量传递给为普通张量编写的函数。 ([#71804](https://github.com/PaddlePaddle/Paddle/pull/71804)) - 为算子 fused_linear_param_grad_add 增加检查。([#72483](https://github.com/PaddlePaddle/Paddle/pull/72483)) ## 5. 算子机制 ### 新特性 - 梯度与自动微分优化：初步支持put_along_axis及repeat_interleave操作的双重梯度计算，提升复杂算子在自动微分场景下的数值稳定性，实现masked_fill操作的算子分解。 [#72789](https://github.com/PaddlePaddle/Paddle/pull/72789), [#73056](https://github.com/PaddlePaddle/Paddle/pull/73056), [#73225](https://github.com/PaddlePaddle/Paddle/pull/73225) - 运算符机制扩展：新增对__radd__和__rmul__的自定义支持，增强框架对非对称运算符的重载能力。 [#73119](https://github.com/PaddlePaddle/Paddle/pull/73119) - FP8模块支持及算子开发：新增FP8块量化GEMM支持，引入多个融合算子，为混合专家（MoE）模型提供高效算子级实现，提升训推性能。 [#73228](https://github.com/PaddlePaddle/Paddle/pull/73228), [#73285](https://github.com/PaddlePaddle/Paddle/pull/73285), [#73133](https://github.com/PaddlePaddle/Paddle/pull/73133), [#73364](https://github.com/PaddlePaddle/Paddle/pull/73364), [#73520](https://github.com/PaddlePaddle/Paddle/pull/73520), [#73531](https://github.com/PaddlePaddle/Paddle/pull/73531) ### Bug 修复 - 梯度与自动微分稳定性提升：修复部分反向算子梯度计算错误，增强自动微分场景下的数值稳定性与功能正确性。 [#71716](https://github.com/PaddlePaddle/Paddle/pull/71716), [#72299](https://github.com/PaddlePaddle/Paddle/pull/72299), [#72358](https://github.com/PaddlePaddle/Paddle/pull/72358), [#73037](https://github.com/PaddlePaddle/Paddle/pull/73037), [#73140](https://github.com/PaddlePaddle/Paddle/pull/73140), [#73185](https://github.com/PaddlePaddle/Paddle/pull/73185) - 数值精度与溢出防护：解决数值溢出、精度损失及大 tensor 溢出问题，保障低精度计算与大张量操作的可靠性。 [#72584](https://github.com/PaddlePaddle/Paddle/pull/72584), [#72608](https://github.com/PaddlePaddle/Paddle/pull/72608), [#72681](https://github.com/PaddlePaddle/Paddle/pull/72681), [#72639](https://github.com/PaddlePaddle/Paddle/pull/72639), [#73245](https://github.com/PaddlePaddle/Paddle/pull/73245), [#73359](https://github.com/PaddlePaddle/Paddle/pull/73359), [#72456](https://github.com/PaddlePaddle/Paddle/pull/72456) - 算子逻辑与框架对齐：对齐算子运算逻辑，修复部分算子输入异常等问题，其他重要修复：添加检查，保障框架功能正确性。 [#72282](https://github.com/PaddlePaddle/Paddle/pull/72282), [#71863](https://github.com/PaddlePaddle/Paddle/pull/71863), [#72650](https://github.com/PaddlePaddle/Paddle/pull/72650), [#72843](https://github.com/PaddlePaddle/Paddle/pull/72843), [#73070](https://github.com/PaddlePaddle/Paddle/pull/73070), [#73141](https://github.com/PaddlePaddle/Paddle/pull/73141), [#73203](https://github.com/PaddlePaddle/Paddle/pull/73203), [#73350](https://github.com/PaddlePaddle/Paddle/pull/73350), [#73440](https://github.com/PaddlePaddle/Paddle/pull/73440), [#73539](https://github.com/PaddlePaddle/Paddle/pull/73539), [#73339](https://github.com/PaddlePaddle/Paddle/pull/73339) - CUDA内核与硬件适配优化：支持NVIDIA SM90架构，修复溢出等问题，移除冗余CUDA错误检查，提升GPU计算效率与新硬件适配性。 [#72507](https://github.com/PaddlePaddle/Paddle/pull/72507), [#72849](https://github.com/PaddlePaddle/Paddle/pull/72849), [#72959](https://github.com/PaddlePaddle/Paddle/pull/72959), [#73130](https://github.com/PaddlePaddle/Paddle/pull/73130), [#73489](https://github.com/PaddlePaddle/Paddle/pull/73489) ### 功能增强 - 新增int64_t版本的快速除法取模实现，提升大整数场景下的计算性能与数值稳定性, [#72530](https://github.com/PaddlePaddle/Paddle/pull/72530) - 优化带步长张量拷贝kernel，改进非连续内存布局下的数据拷贝效率。 [#72662](https://github.com/PaddlePaddle/Paddle/pull/72662) -统一动态图与静态图模式下量化API的使用方式，简化量化模型开发流程, [#73100](https://github.com/PaddlePaddle/Paddle/pull/73100) ### 性能提升 - 优化gelu算子分解性能，提升计算效率。 [#72812](https://github.com/PaddlePaddle/Paddle/pull/72812) ### 其他 - fluid 算子规范化与退场, [#71789](https://github.com/PaddlePaddle/Paddle/pull/71789), [#71818](https://github.com/PaddlePaddle/Paddle/pull/71818), [#71808](https://github.com/PaddlePaddle/Paddle/pull/71808), [#71860](https://github.com/PaddlePaddle/Paddle/pull/71860), [#71806](https://github.com/PaddlePaddle/Paddle/pull/71806), [#72011](https://github.com/PaddlePaddle/Paddle/pull/72011), [#72043](https://github.com/PaddlePaddle/Paddle/pull/72043), [#72034](https://github.com/PaddlePaddle/Paddle/pull/72034), [#72047](https://github.com/PaddlePaddle/Paddle/pull/72047), [#72056](https://github.com/PaddlePaddle/Paddle/pull/72056), [#72087](https://github.com/PaddlePaddle/Paddle/pull/72087), [#72086](https://github.com/PaddlePaddle/Paddle/pull/72086), [#72083](https://github.com/PaddlePaddle/Paddle/pull/72083), [#72079](https://github.com/PaddlePaddle/Paddle/pull/72079), [#72078](https://github.com/PaddlePaddle/Paddle/pull/72078), [#72076](https://github.com/PaddlePaddle/Paddle/pull/72076), [#72057](https://github.com/PaddlePaddle/Paddle/pull/72057), [#72077](https://github.com/PaddlePaddle/Paddle/pull/72077), [#72096](https://github.com/PaddlePaddle/Paddle/pull/72096), [#72085](https://github.com/PaddlePaddle/Paddle/pull/72085), [#72092](https://github.com/PaddlePaddle/Paddle/pull/72092), [#72110](https://github.com/PaddlePaddle/Paddle/pull/72110), [#72127](https://github.com/PaddlePaddle/Paddle/pull/72127), [#72111](https://github.com/PaddlePaddle/Paddle/pull/72111), [#72126](https://github.com/PaddlePaddle/Paddle/pull/72126), [#72135](https://github.com/PaddlePaddle/Paddle/pull/72135), [#72112](https://github.com/PaddlePaddle/Paddle/pull/72112), [#72131](https://github.com/PaddlePaddle/Paddle/pull/72131), [#70358](https://github.com/PaddlePaddle/Paddle/pull/70358), [#72125](https://github.com/PaddlePaddle/Paddle/pull/72125), [#72171](https://github.com/PaddlePaddle/Paddle/pull/72171), [#72160](https://github.com/PaddlePaddle/Paddle/pull/72160), [#72188](https://github.com/PaddlePaddle/Paddle/pull/72188), [#72197](https://github.com/PaddlePaddle/Paddle/pull/72197), [#72212](https://github.com/PaddlePaddle/Paddle/pull/72212), [#72211](https://github.com/PaddlePaddle/Paddle/pull/72211), [#72184](https://github.com/PaddlePaddle/Paddle/pull/72184), [#71897](https://github.com/PaddlePaddle/Paddle/pull/71897), [#72219](https://github.com/PaddlePaddle/Paddle/pull/72219), [#72218](https://github.com/PaddlePaddle/Paddle/pull/72218), [#72074](https://github.com/PaddlePaddle/Paddle/pull/72074), [#70330](https://github.com/PaddlePaddle/Paddle/pull/70330), [#70274](https://github.com/PaddlePaddle/Paddle/pull/70274), [#72295](https://github.com/PaddlePaddle/Paddle/pull/72295), [#72220](https://github.com/PaddlePaddle/Paddle/pull/72220), [#72343](https://github.com/PaddlePaddle/Paddle/pull/72343), [#72303](https://github.com/PaddlePaddle/Paddle/pull/72303), [#72296](https://github.com/PaddlePaddle/Paddle/pull/72296), [#72338](https://github.com/PaddlePaddle/Paddle/pull/72338), [#70001](https://github.com/PaddlePaddle/Paddle/pull/70001), [#70348](https://github.com/PaddlePaddle/Paddle/pull/70348), [#70329](https://github.com/PaddlePaddle/Paddle/pull/70329) ## 6. 框架性能优化 ### 新特性支持`sharding_overlap`的`acc_steps`可配置。 [#72395](https://github.com/PaddlePaddle/Paddle/pull/72395) ### Bug 修复 - 修复算子`c_softmax_with_cross_entropy_grad`的`inplace`问题。 [#72366](https://github.com/PaddlePaddle/Paddle/pull/72366) ### 功能增强 - 性能优化与加速：启用深度卷积的cuDNN支持，提升卷积运算效率。更新池化操作策略并优化permute内存操作，减少CUDA内存占用。优化打印速度，加速调试与日志输出流程。 [#71796](https://github.com/PaddlePaddle/Paddle/pull/71796), [#73442](https://github.com/PaddlePaddle/Paddle/pull/73442), [#73563](https://github.com/PaddlePaddle/Paddle/pull/73563) - 功能增强与操作支持：新增masked_fill操作及布尔索引优化，增强张量掩码处理能力。实现index_elementwise操作，支持基于索引的元素级运算。添加池化与reshape执行策略，提升模型操作的灵活性。 [#72788](https://github.com/PaddlePaddle/Paddle/pull/72788), [#72942](https://github.com/PaddlePaddle/Paddle/pull/72942) - 错误修复与稳定性提升：修复fused_rms_norm在SPMD并行模式下的部分状态支持问题。修正slice操作中输出维度计算及IndexGetStride的索引错误，确保计算正确性。 [#72118](https://github.com/PaddlePaddle/Paddle/pull/72118), [#72223](https://github.com/PaddlePaddle/Paddle/pull/72223), [#73184](https://github.com/PaddlePaddle/Paddle/pull/73184), [#73237](https://github.com/PaddlePaddle/Paddle/pull/73237), [#73054](https://github.com/PaddlePaddle/Paddle/pull/73054) ### 性能提升 - Faster Guard适配：减少SOT端到端链路开销。 [#71900](https://github.com/PaddlePaddle/Paddle/pull/71900), [#71979](https://github.com/PaddlePaddle/Paddle/pull/71979), [#72081](https://github.com/PaddlePaddle/Paddle/pull/72081), [#72327](https://github.com/PaddlePaddle/Paddle/pull/72327), [#72564](https://github.com/PaddlePaddle/Paddle/pull/72564), [#72823](https://github.com/PaddlePaddle/Paddle/pull/72823) - 性能优化与加速：优化算子调度策略。升级Flash Attention至v3版本，减少计算开销。修复模型性能瓶颈，提升推理与训练速度。 [#71937](https://github.com/PaddlePaddle/Paddle/pull/71937), [#71828](https://github.com/PaddlePaddle/Paddle/pull/71828), [#71461](https://github.com/PaddlePaddle/Paddle/pull/71461), [#72039](https://github.com/PaddlePaddle/Paddle/pull/72039), [#72228](https://github.com/PaddlePaddle/Paddle/pull/72228), [#72225](https://github.com/PaddlePaddle/Paddle/pull/72225), [#72623](https://github.com/PaddlePaddle/Paddle/pull/72623), [#72666](https://github.com/PaddlePaddle/Paddle/pull/72666), [#73147](https://github.com/PaddlePaddle/Paddle/pull/73147), [#73393](https://github.com/PaddlePaddle/Paddle/pull/73393) - 并行计算：优化自动并行中的网格重分片策略，实现Sharding Stage的通信融合并优化逻辑，提升分布式训练稳定性，降低分布式训练通信开销。 [#71969](https://github.com/PaddlePaddle/Paddle/pull/71969), [#72120](https://github.com/PaddlePaddle/Paddle/pull/72120), [#73279](https://github.com/PaddlePaddle/Paddle/pull/73279), [#73406](https://github.com/PaddlePaddle/Paddle/pull/73406) 功能增强与修复：- 优化算子索引和内核调度逻辑。 [#72625](https://github.com/PaddlePaddle/Paddle/pull/72625), [#72741](https://github.com/PaddlePaddle/Paddle/pull/72741), [#73082](https://github.com/PaddlePaddle/Paddle/pull/73082), [#73501](https://github.com/PaddlePaddle/Paddle/pull/73501) - 模型与操作支持：支持NHWC格式的深度卷积，适配更多硬件内存布局。 [#72121](https://github.com/PaddlePaddle/Paddle/pull/72121) ## 7. 硬件适配优化硬件机制，提供类cuda硬件kernel复用方案。 ### 新特性以customdevice接入方案为基础，增加低成本支持类cuda后端硬件的支持方案。类cuda后端可以以插件式方式接入paddle，低成本复用paddle中多数nv生态中的cuda kernel，且可以与paddle框架中的特性feature升级解耦，大大降低硬件后端接入与迭代成本，提升用户接入意愿，形成paddle与硬件厂商共建生态的良好合作关系。 [#72604](https://github.com/PaddlePaddle/Paddle/pull/72604)[#72668](https://github.com/PaddlePaddle/Paddle/pull/72668))[#72758](https://github.com/PaddlePaddle/Paddle/pull/72758)[#72865](https://github.com/PaddlePaddle/Paddle/pull/72865)[#72910](https://github.com/PaddlePaddle/Paddle/pull/72910)[#73033](https://github.com/PaddlePaddle/Paddle/pull/73033))[#73145](https://github.com/PaddlePaddle/Paddle/pull/73145)[#73281](https://github.com/PaddlePaddle/Paddle/pull/73281)[#73079](https://github.com/PaddlePaddle/Paddle/pull/73079) 补充XPU 基础能力：XPU 环境下增加kernel ，扩展数据类型，补充分支 [#71424](https://github.com/PaddlePaddle/Paddle/pull/71424)[#71809](https://github.com/PaddlePaddle/Paddle/pull/71809)[#71594](https://github.com/PaddlePaddle/Paddle/pull/71594)[#71779](https://github.com/PaddlePaddle/Paddle/pull/71779)[#71756](https://github.com/PaddlePaddle/Paddle/pull/71756)[#71573](https://github.com/PaddlePaddle/Paddle/pull/71573)[#71883](https://github.com/PaddlePaddle/Paddle/pull/71883)[#71954](https://github.com/PaddlePaddle/Paddle/pull/71954)[#71931](https://github.com/PaddlePaddle/Paddle/pull/71931)[#72280](https://github.com/PaddlePaddle/Paddle/pull/72280)[#72361](https://github.com/PaddlePaddle/Paddle/pull/72361)[#72406](https://github.com/PaddlePaddle/Paddle/pull/72406)[#72528](https://github.com/PaddlePaddle/Paddle/pull/72528)[#72752](https://github.com/PaddlePaddle/Paddle/pull/72752)[#72852](https://github.com/PaddlePaddle/Paddle/pull/72852)[#72982](https://github.com/PaddlePaddle/Paddle/pull/72982)[#73357](https://github.com/PaddlePaddle/Paddle/pull/73357)[#73414](https://github.com/PaddlePaddle/Paddle/pull/73414)[#73464](https://github.com/PaddlePaddle/Paddle/pull/73464)[#73234](https://github.com/PaddlePaddle/Paddle/pull/73234)[#71776](https://github.com/PaddlePaddle/Paddle/pull/71776) DCU kernel 扩展数据类型 [#73129](https://github.com/PaddlePaddle/Paddle/pull/73129) ### Bug 修复修复xpu执行问题 [#71852](https://github.com/PaddlePaddle/Paddle/pull/71852)[#71966](https://github.com/PaddlePaddle/Paddle/pull/71966)[#72005](https://github.com/PaddlePaddle/Paddle/pull/72005)[#71908](https://github.com/PaddlePaddle/Paddle/pull/71908)[#72431](https://github.com/PaddlePaddle/Paddle/pull/72431)[#72519](https://github.com/PaddlePaddle/Paddle/pull/72519)[#72734](https://github.com/PaddlePaddle/Paddle/pull/72734)[#72763](https://github.com/PaddlePaddle/Paddle/pull/72763)[#72762](https://github.com/PaddlePaddle/Paddle/pull/72762)[#72890](https://github.com/PaddlePaddle/Paddle/pull/72890)[#72867](https://github.com/PaddlePaddle/Paddle/pull/72867)[#73071](https://github.com/PaddlePaddle/Paddle/pull/73071)[#73004](https://github.com/PaddlePaddle/Paddle/pull/73004)[#72726](https://github.com/PaddlePaddle/Paddle/pull/72726)[#73113](https://github.com/PaddlePaddle/Paddle/pull/73113)[#73127](https://github.com/PaddlePaddle/Paddle/pull/73127)[#73025](https://github.com/PaddlePaddle/Paddle/pull/73025)[#73301](https://github.com/PaddlePaddle/Paddle/pull/73301)[#73292](https://github.com/PaddlePaddle/Paddle/pull/73292)[#73272](https://github.com/PaddlePaddle/Paddle/pull/73272)[#73305](https://github.com/PaddlePaddle/Paddle/pull/73305)[#73356](https://github.com/PaddlePaddle/Paddle/pull/73356)[#73438](https://github.com/PaddlePaddle/Paddle/pull/73438)[#72041](https://github.com/PaddlePaddle/Paddle/pull/72041)[#72275](https://github.com/PaddlePaddle/Paddle/pull/72275)[#72787](https://github.com/PaddlePaddle/Paddle/pull/72787)[#73504](https://github.com/PaddlePaddle/Paddle/pull/73504)[#73290](https://github.com/PaddlePaddle/Paddle/pull/73290) ## 8. 安装环境适配优化了框架的稳定性和跨平台兼容性，修复了不同平台上的编译安装失败问题；升级CUDA等关键依赖，进一步优化CI/CD流程，提升构建速度并增强系统整体稳定性；停止对Python3.8环境下的编译安装维护。 ### Bug 修复 - 修复使用clang17编译第三方库时的编译错误。[#72524](https://github.com/PaddlePaddle/Paddle/pull/72524) - 修复使用CUDA12.9时的编译问题。 [#72808](https://github.com/PaddlePaddle/Paddle/pull/72808), [#72841](https://github.com/PaddlePaddle/Paddle/pull/72841), [#72978](https://github.com/PaddlePaddle/Paddle/pull/72978), [#73360](https://github.com/PaddlePaddle/Paddle/pull/73360) - 修复使用GCC13.3时的编译问题。[#73144](https://github.com/PaddlePaddle/Paddle/pull/73144) - 修复WITH_PIP_CUDA_LIBRARIES=ON时的编译问题。[#72907](https://github.com/PaddlePaddle/Paddle/pull/72907) - 修复WITH_NVSHMEM=ON时的编译问题。[#73368](https://github.com/PaddlePaddle/Paddle/pull/73368) ### 功能增强 - 避免自定义算子编译产生的临时文件的拷贝。[#73196](https://github.com/PaddlePaddle/Paddle/pull/73196) - Warning信息优化。[#72877](https://github.com/PaddlePaddle/Paddle/pull/72877) ### 开发者相关 - 编译安装维护与升级。[#71911](https://github.com/PaddlePaddle/Paddle/pull/71911), [#73005](https://github.com/PaddlePaddle/Paddle/pull/73005) - 镜像维护与更新。[#71065](https://github.com/PaddlePaddle/Paddle/pull/71065), [#71821](https://github.com/PaddlePaddle/Paddle/pull/71821) - Windows平台符号的导入导出更新。[#72497](https://github.com/PaddlePaddle/Paddle/pull/72497), [#72498](https://github.com/PaddlePaddle/Paddle/pull/72498), [#72500](https://github.com/PaddlePaddle/Paddle/pull/72500) - Windows平台支持CUDA12.8。[#72433](https://github.com/PaddlePaddle/Paddle/pull/72433) - CI维护与升级。[#72443](https://github.com/PaddlePaddle/Paddle/pull/72443), [#72836](https://github.com/PaddlePaddle/Paddle/pull/72836), [#72563](https://github.com/PaddlePaddle/Paddle/pull/72563), [#72653](https://github.com/PaddlePaddle/Paddle/pull/72653), [#72477](https://github.com/PaddlePaddle/Paddle/pull/72477), [#72778](https://github.com/PaddlePaddle/Paddle/pull/72778), [#72960](https://github.com/PaddlePaddle/Paddle/pull/72960), [#73289](https://github.com/PaddlePaddle/Paddle/pull/73289), [#73422](https://github.com/PaddlePaddle/Paddle/pull/73422), [#73514](https://github.com/PaddlePaddle/Paddle/pull/73514), [#72748](https://github.com/PaddlePaddle/Paddle/pull/72748), - Github Action CI建设。[#71738](https://github.com/PaddlePaddle/Paddle/pull/71738), [#70602](https://github.com/PaddlePaddle/Paddle/pull/70602), [#71958](https://github.com/PaddlePaddle/Paddle/pull/71958), [#71959](https://github.com/PaddlePaddle/Paddle/pull/71959), [#71992](https://github.com/PaddlePaddle/Paddle/pull/71992), [#72013](https://github.com/PaddlePaddle/Paddle/pull/72013), [#72153](https://github.com/PaddlePaddle/Paddle/pull/72153), [#72031](https://github.com/PaddlePaddle/Paddle/pull/72031), [#72141](https://github.com/PaddlePaddle/Paddle/pull/72141), [#72104](https://github.com/PaddlePaddle/Paddle/pull/72104), [#72182](https://github.com/PaddlePaddle/Paddle/pull/72182), [#72342](https://github.com/PaddlePaddle/Paddle/pull/72342), [#72352](https://github.com/PaddlePaddle/Paddle/pull/72352), [#72249](https://github.com/PaddlePaddle/Paddle/pull/72249), [#72068](https://github.com/PaddlePaddle/Paddle/pull/72068), [#72441](https://github.com/PaddlePaddle/Paddle/pull/72441), [#72392](https://github.com/PaddlePaddle/Paddle/pull/72392), [#72446](https://github.com/PaddlePaddle/Paddle/pull/72446), [#72435](https://github.com/PaddlePaddle/Paddle/pull/72435), [#72515](https://github.com/PaddlePaddle/Paddle/pull/72515), [#72514](https://github.com/PaddlePaddle/Paddle/pull/72514), [#72396](https://github.com/PaddlePaddle/Paddle/pull/72396), [#72547](https://github.com/PaddlePaddle/Paddle/pull/72547), [#72345](https://github.com/PaddlePaddle/Paddle/pull/72345), [#72236](https://github.com/PaddlePaddle/Paddle/pull/72236), [#72586](https://github.com/PaddlePaddle/Paddle/pull/72586), [#72537](https://github.com/PaddlePaddle/Paddle/pull/72537), [#72609](https://github.com/PaddlePaddle/Paddle/pull/72609), [#72632](https://github.com/PaddlePaddle/Paddle/pull/72632), [#72642](https://github.com/PaddlePaddle/Paddle/pull/72642), [#72673](https://github.com/PaddlePaddle/Paddle/pull/72673), [#72647](https://github.com/PaddlePaddle/Paddle/pull/72647), [#72696](https://github.com/PaddlePaddle/Paddle/pull/72696), [#72771](https://github.com/PaddlePaddle/Paddle/pull/72771), [#72711](https://github.com/PaddlePaddle/Paddle/pull/72711), [#72680](https://github.com/PaddlePaddle/Paddle/pull/72680), [#72774](https://github.com/PaddlePaddle/Paddle/pull/72774), [#72813](https://github.com/PaddlePaddle/Paddle/pull/72813), [#72804](https://github.com/PaddlePaddle/Paddle/pull/72804), [#72903](https://github.com/PaddlePaddle/Paddle/pull/72903), [#72900](https://github.com/PaddlePaddle/Paddle/pull/72900), [#72932](https://github.com/PaddlePaddle/Paddle/pull/72932), [#72967](https://github.com/PaddlePaddle/Paddle/pull/72967), [#72991](https://github.com/PaddlePaddle/Paddle/pull/72991), [#72115](https://github.com/PaddlePaddle/Paddle/pull/72115), [#73242](https://github.com/PaddlePaddle/Paddle/pull/73242), [#72801](https://github.com/PaddlePaddle/Paddle/pull/72801), [#73433](https://github.com/PaddlePaddle/Paddle/pull/73433), [#73391](https://github.com/PaddlePaddle/Paddle/pull/73391), [#73456](https://github.com/PaddlePaddle/Paddle/pull/73456), [#73376](https://github.com/PaddlePaddle/Paddle/pull/73376), [#73453](https://github.com/PaddlePaddle/Paddle/pull/73453), [#73481](https://github.com/PaddlePaddle/Paddle/pull/73481), [#73546](https://github.com/PaddlePaddle/Paddle/pull/73546), [#73446](https://github.com/PaddlePaddle/Paddle/pull/73446), [#72744](https://github.com/PaddlePaddle/Paddle/pull/72744) ### 废弃 - 停止支持Python3.8环境下的编译。[#72827](https://github.com/PaddlePaddle/Paddle/pull/72827) ## 9. 贡献者名单 0x3878f, A-nnonymous, AndSonder, ApricityXX, aquagull, author, baoqiwen, BeingGod, blacksheep-Aristotle, BoShen5, bukejiyu, cangtianhuang, carryyu, chang-wenbin, changeyoung98, chen2016013, ckl117, co63oc, cqulilujia, crashbussy, cszdrg, Cutelemon6, cyy536, DanielSun11, danleifeng, datutu-L, deepllz, Dmovic, DrRyanHuang, dynamicheart, Eddie-Wang1120, eggman-1024, emmanuel-ferdman, Enigmatisms, enkilee, fangfangssj, feixi21, FeixLiu, ForFishes, Function-Samuel, ggggxm, GITD245, Glencsa, GoldenStain, gongshaotian, gouzil, gzy19990617, hanlintang, Hongqing-work, houj04, huangjiyi, hxzd5568, HydrogenSulfate, jzhang533, LCStayingdullCircuit, leon062112, lifulll, linkk08, LittleHeroZZZX, liufengwei0103, Liujie0926, liuruyan, lixinqi, LiYuRio, lizexu123, lizhenyun01, lj970926, lshpku, megemini, mikethegoblin, ming1753, mzj104, NKNaN, ooooo-create, pesionzhao, phlrain, pkuzyc, PolaKuma, Qin-sx, RichardWooSJTU, risemeup1, runzhech, RuohengMa, sasaya123, shanjiang7, SigureMo, sneaxiy, swgu98, SylarTiaNII, tianhaodongbd, tianshuo78520a, timminator, tizhou86, umiswing, waliwali777, wanghuancoder, Waynezee, Wennie396, xiaoguoguo626807, XieYunshen, Xing-lil, xkkkkkk23, Xreki, xuxinyi389, Yeenyeong, yongqiangma, YqGe585, yuanlehome, YuanRisheng, yulangz, yuwu46, zeroRains, zhangbo9674, zhanghonggeng, zhangting2020, ZhangX-21, zhangyk0314, zhangyuqin1998, zhink, zhiqiu, zhouquan32, zhoutianzi666, zhupengyang, zrr1999, zty-king, zyfncg

v0.1.1

? · 2025-05-17

bsrc test it

PaddlePaddle 3.0.0 Release Notev3.0.0

? · 2025-03-31

- [完整中文版本](https://github.com/PaddlePaddle/Paddle/wiki/PaddlePaddle-3.0-%E6%AD%A3%E5%BC%8F%E7%89%88%E6%A6%82%E8%BF%B0) # PaddlePaddle 3.0 正式版概述作为中国首个自主研发的产业级深度学习平台，飞桨一直坚持开源路线，支撑产业智能化升级。飞桨框架3.0版本不仅延续了飞桨框架2.0系列动静统一、训推一体的特性，更在自动并行、神经网络编译器、高阶自动微分等方面取得突破，为大模型时代的技术创新与产业应用提供了强大支撑，为开发者打造了一站式、高性能的深度学习开发体验。无论是前沿算法研究还是产业级大模型落地，飞桨框架3.0都将成为开发者的首选利器。重点特性说明如下： - **动静统一自动并行：** 这一功能大幅度降低了产业开发和训练的成本。用户只需在单卡基础上进行少量的张量切分标记，飞桨框架便会自动完成分布式切分信息的推导，并添加通信算子以确保逻辑的正确性。同时，根据模型结构和集群信息，结合显存和调度层的优化，飞桨能自动寻找最高效的分布式并行策略，从而大幅降低混合并行训练的开发成本，使开发者能够更专注于模型和算法的创新。自动并行架构进行了深入的验证和打磨，以更好地支持纯文稠密模型、纯文稀疏模型（MoE）和多模态理解模型等常见大模型场景的预训练+精调流程；完善算子的切分推导规则，并支持将自动并行训练参数转化成手动并行参数进行下游推理，自动并行达到了全面可用的状态，帮助用户降低大模型并行程序的开发成本。同时，为了进一步简化用户的分布式开发流程，推出全新的`paddle.distributed.parallel`接口，基于对分布式张量标记语法的封装，支持用户在模型组网外不侵入地配置数据并行、模型并行、流水并行等常见的并行策略。此外，静态图自动并行架构基于PIR完成了全面的升级，底层的基础组件、核心模块、并行策略和性能优化策略均统一基于扩展的PIR `DistDialect`进行实现，进一步增强了自动并行的动静一致性，并在Llama系列模型上性能达到了持平甚至领先手动并行方式的水平。 - **大模型训推一体：** 自2.0版本起，飞桨便采用了“动静统一、训推一体”的设计理念，3.0版本也将继续秉持这一理念。得益于动静统一的架构和接口设计，飞桨能够完整支持动态图和静态图这两种不同的运行模式，并且具备出色的整图导出能力。飞桨的动转静整图导出成功率高达95%，高于PyTorch的62%。“训推一体”意味着能够在同一套框架下，尽可能复用训练和推理的代码，特别是复用模型组网代码。在完成模型的开发训练后，只需进行少量的开发工作，即可实现快速推理部署。这一特性为产业提供了极致的开发体验。它使训练和推理的能力能够相互复用，为大模型的全流程提供了统一的开发体验和极致的训练效率。通过动转静的工作，训练和推理的工作得以无缝衔接。支持多款主流大模型、DeepSeek-R1满血版实现单机部署，吞吐提升一倍。 - **科学计算高阶微分：** 飞桨框架3.0为科学计算提供了高阶自动微分、编译优化和分布式训练能力的支撑。英伟达Modulus的41个不同方程实验显示，飞桨的微分方程求解速度比PyTorch开启编译器优化后的版本平均快 112%。同时，飞桨还建设了面向通用数理问题求解的赛桨PaddleScience以及专注于生物计算的螺旋桨PaddleHelix工具包。此外，飞桨框架3.0还原生支持复数技术体系，这对于气象预报、汽车飞行器气动分析等场景下的数据特征分析具有重要意义。 - **神经网络编译器：** 这一功能显著降低了性能优化的成本。飞桨的编译器采用与框架一体化的设计，能够支持生成式模型、科学计算模型等多种模型的高效训练与可变形状推理，在计算灵活性与高性能之间提供了良好的平衡点。使用 CINN 编译器后超过 60%的模型有显著性能提升，平均提升达 27.4%。CINN神经网络编译器在完备性、性能表现等方面效果全面提升。此版本中，我们对编译器前端、后端各个环节进行了全面优化：包括新增反向计算图自动Re-Compute机制、前端 Pass 性能优化、符号推导机制升级、算子融合策略优化、后端 Schedule 策略和下标表达式化简能力增强等，同时排查并修复了大量正确性和性能问题，系统化的提升了编译器的通用优化能力。 - **异构多芯适配：** 飞桨的重要特色之一是适配异构多芯并充分释放硬件潜能。在接入机制上，飞桨提供了简洁高效的抽象接口和基础算子体系，降低了适配成本。在运行机制上，它优化了调度编排和存储共享等机制，提升了调度效率。从算子内核角度，飞桨提供了编译器自动融合调优方案，以提升端到端的性能。同时，飞桨还为新硬件厂商建设了代码合入、持续集成、模型回归测试等研发基础设施。这些机制保障了新硬件被纳入飞桨的正常发版体系中，用户无需编译即可直接安装试用。飞桨这种功能完善、低成本接入的机制吸引了硬件厂商共同为飞桨贡献了4001个PR，共包含26584个commits。除了上述核心特性外，**高扩展中间表示 **为了提升飞桨框架的可扩展性，我们研发了高扩展中间表示PIR（Paddle Intermediate Representation）。这一表示系统性地抽象了底层核心概念，提供了灵活且高效的组件。PIR作为基础设施，支撑着动转静、自动微分、自动并行、组合算子、图优化等多项技术，并广泛应用于分布式训练、模型压缩、推理部署等场景。通过PIR提供的DRR（Declarative Rewrite Rule）机制，Pass的开发成本可以降低60%。同时PIR完成在全场景的验证，并默认开启，支持一键动转静，保证了框架卓越的性能表现和良好的拓展性。对框架2.0版已有功能的持续改进，同时新特性在使用体验、性能、二次开发便利度以及硬件适配能力等方面带来了显著提升。此版本在用户体验层面持续丰富并增强了满足更多场景的API功能，针对大模型场景优化完善了分布式并行策略优化和推理功能增强，在编译安装方面做了比较彻底的易用性改进，对依赖包的安装方式和版本进行了全新同步升级，对系统安全进行了全面加固，对产品文档也进行了全面的纠错检查，同时也对一些废弃代码做了大量的清理以保证架构的简洁性。 ## 不兼容升级飞桨API支持隐式类型提升。在加减乘除等最常用的计算中，如果两个输入的数据类型不一样，就需要确定输出的数据类型问题。飞桨历史上的现状是部分支持且实际规则并不清楚，客观上表现为动静不一致、API和运算符重载不一致及不符合交换率，特别是在大模型广泛使用 bf16/fp16 与 fp32 进行混合计算时容易出现非预期问题且难以定位。飞桨从3.0 beta版本开始，明确了[隐式数据类型提升规则](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/guides/advanced/auto_type_promotion_cn.html)，其中详细定义了 Tensor与Tensor 和 Tensor与1个数(Scalar)计算结果的类型，保证了计算符合交换律，运算符重载与二元 API 结果一致，动态图与静态图结果一致。更符合用户理解和业界习惯。https://github.com/PaddlePaddle/Paddle/pull/60638, https://github.com/PaddlePaddle/Paddle/pull/63842, https://github.com/PaddlePaddle/Paddle/pull/60011 ## 废弃功能支持0维Tensor已经稳定了2个版本，本版本取消了在一些情况下将0维Tensor转成只含1个元素的1维Tensor的开关FLAGS_set_to_1d，这个开关是为了兼容一些套件中用1个元素的1维Tensor表示0维Tensor的不正确写法。即当前飞桨完全区分0维Tensor和只含1个元素的1维Tensor的语义，两者不等价。https://github.com/PaddlePaddle/Paddle/pull/61227 ## 贡献者名单 0x3878f, 0x45f, 2742195759, 86kkd, A-nnonymous, ADream-ki, Aganlengzi, Albresky, AndPuQing, AndSonder, Aoraki-Dream, ApricityXX, Asthestarsfalll, Aurelius84, BHmingyang, BeingGod, Betelgeu, BiynXu, CJ77Qi, Caogration, DDDivano, Dale1314, Deleter-D, DesmonDay, Difers, Dmovic, DongBaiYue, DrRyanHuang, DrownFish19, Eddie-Wang1120, EgoistSA, FeixLiu, ForFishes, Fripping, From00, Function-Samuel, GoldenStain, Guanhuachen2003, GuoxiaWang, Hanyonggong, HarperCy, Hongqing-work, HydrogenSulfate, JZ-LIANG, Jeff114514, JiaWenxuan, LLee233, LanCole, Lans1ot, Layssy, Leoforever123, LiYuRio, LielinJiang, LittleHeroZZZX, Liujie0926, Liyulingyue, Luohongzhige, Marcusryz, MarisaSparkL, Micalling, MikhayEeer, MrXnneHang, MufanColin, NKNaN, Neo-WY, NeroLoh, PolaKuma, Qin-sx, QingshuChen, RachelXu7, RichardWooSJTU, RuohengMa, SCUcookie, Sekiro-x, SigureMo, Sunny-bot1, SylarTiaNII, Sylence8, TBD1, TR666, TimeYWL, Tom-Zheng, Turingg, Victor-Bayim, Vvsmile, WAYKEN-TSE, Wanglongzhi2001, Wangzheee, Waynezee, Wennie396, Whsjrczr, Wizard-ZP, Wong4j, XavierZXY, XiaociZhang, XieYunshen, Xing-lil, Xreki, YKTian-x2b, YZW-explorer, YanhuiDua, YuanRisheng, ZHOU05030, ZhangHandi, ZhangX-21, ZibinGuo, a2064968462, anderson101866, aooxin, aquagull, baoqiwen, bapijun, blacksheep-Aristotle, bukejiyu, carryyu, ccsuzzh, chang-wenbin, changeyoung98, chen2016013, ckl117, cmcamdy, co63oc, continue-coding, cqulilujia, crazyxiaoxi, cszdrg, cubehan3, cyber-pioneer, danleifeng, decade-afk, deepllz, dynamicheart, eee4017, eggman-1024, enkilee, epiphanyer, ethan-sem, fangfangssj, feixi21, fightfat, fufu0615, fxfxfxfxfxfxfxfx, fxy1699, gitliuyf, gongel, gongshaotian, gongweibao, gouzil, gsq7474741, guixxiic, gzy19990617, hanyang2508, haoyu2022, heavyrain-lzy, houj04, huangjiyi, huangkr03, hxzd5568, icpcccpc, inaomIIsfarell, iosmers, jeff41404, jerrywgz, jiachengdai, jiahy0825, jinmingyi1998, jinyouzhi, joseflv, jychen21, jzhang533, kangguangli, kanze1, kineast, kircle888, l1cacheDell, leo0519, lifulll, linkk08, little1d, liufengwei0103, liuruyan, lixcli, liym27, liyongchao911, lizexu123, lizhenyun01, lj970926, lshpku, lszxb, ltd0924, luotao1, lwkhahaha, lxd-cumt, mayang002, megemini, mikemikimike, ming1753, monster1015, mori0umi, ndyysheep, nizne9, nobodynobody, ooooo-create, penPenf28, phlrain, pkuzyc, qili93, rich04lin, risemeup1, ronny1996, rsmallblue, runzhech, skywalker2012, smile2game, sneaxiy, successfulbarrier, sunzhongkai588, swgu98, tc20042008, tianhaodongbd, tianshuo78520a, tizhou86, tlxd, uanu2002, umiswing, vivienfanghuagood, waliwali777, walkalone20, wanghuancoder, wangna11BD, will-jl944, winffke, winter-wang, wwwuyan, xiaoguoguo626807, xiaoluomi, xiaoyao0115, xingmingyyj, xkkkkkk23, xu8117, xuxinyi389, xz-alex, yangrongxinuser, yeteye, yinfan98, yongqiangma, yuan20041218, yuanlehome, yuguo-Jack, yumin066, zbt78, zeroRains, zhangbo9674, zhanghonggeng, zhanglirong1999, zhangting2020, zhangyk0314, zhangyuqin1998, zhiminzhang0830, zhink, zhiqiu, zhouquan32, zhoutianzi666, zhwesky2010, zoooo0820, zrr1999, zty-king, zxcd, zyfncg ## 完整版本由于字数超出限制，完整版本请大家访问以下链接： - [完整中文版本](https://github.com/PaddlePaddle/Paddle/wiki/PaddlePaddle-3.0-%E6%AD%A3%E5%BC%8F%E7%89%88%E6%A6%82%E8%BF%B0)

PaddlePaddle 3.0.0-beta0 Release Notev3.0.0-beta0

? · 2024-06-27

- [完整中文版本](https://github.com/PaddlePaddle/Paddle/wiki/PaddlePaddle-3.0.0%E2%80%90beta0-Release-Note-CN) # PaddlePaddle 3.0 Beta 版本概述本版本的核心特性主要包括动静统一自动并行技术和神经网络编译器自动优化等新技术，旨在应对当前深度学习领域的新挑战。飞桨框架3.0 Beta 版本延续了2.x版本动静统一、训推一体的设计理念，其开发接口全面兼容2.x版本。这意味着，使用2.x版本开发的代码，在绝大多数情况下无需修改，即可直接在3.x版本上运行。几个重点特性具体展开说明如下： - 动静统一自动并行：为了降低大模型的编程难度，飞桨还优化了动静统一的半自动并行编程范式，显著简化了编程的复杂度。开发者无需深入研究手动并行编程的复杂概念和API，只需进行少量的张量切分标注，即可完成混合并行模型的构建。框架能够自动推导分布式切分状态并添加通信算子，同时还支持一键动转静分布式训练，从而大幅简化了混合并行训练代码的开发过程。动静统一方面，飞桨通过采用基于字节码的动静转换技术，全面升级了其动转静训练能力，支持自适应的图构建功能。在700多个飞桨产业级模型上进行了验证，实现了一键动转静训练100%的成功率。 - 神经网络编译器自动优化：飞桨神经网络编译器CINN（Compiler Infrastructure for Neural Networks）采用与框架一体化的设计，能够支持生成式模型、科学计算模型等多种模型的高效训练与可变形状推理，为计算灵活性与高性能之间提供了一个良好的平衡点。通过算子的自动融合和代码生成技术，Llama2和Stable Diffusion模型的性能提升了30%。 - 高阶自动微分：为了更好支持科学计算等场景，飞桨框架设计并实现了基于组合算子机制的高阶自动微分技术，结合神经网络编译器自动优化技术，我们测试了超过40多个科学计算场景的微分方程，其求解速度领先业界同类产品70%。 - 高扩展中间表示：为了提升飞桨框架的可扩展性，我们研发了高扩展中间表示PIR（Paddle Intermediate Representation）。这一表示系统性地抽象了底层核心概念，提供了灵活且高效的组件。PIR作为基础设施，支撑着动转静、自动微分、自动并行、组合算子、图优化等多项技术，并广泛应用于分布式训练、模型压缩、推理部署等场景。通过PIR提供的DRR（Declarative Rewrite Rule）机制，Pass的开发成本可以降低60%。我们对超过900个模型配置进行了测试，结果显示，在使用PIR后，推理的整体性能提升了超过10%。 - 多硬件适配：飞桨为大模型硬件适配提供了功能完善且低成本的方案。新硬件仅需适配30余个接口，即可支持大模型的训练、压缩与推理。同时，飞桨提供了基于编译器的硬件接入方式，硬件厂商只需以插件的形式实现编译器的代码生成后端，便能实现与飞桨框架的高效适配。飞桨硬件接入本次新增了对4款硬件昆仑XPU、昇腾NPU、海光DCU和寒武纪MLU的日常发版支持。此版本包含了对框架2.x版本部分已有功能的持续改进，同时本版本的新特性在使用体验、性能、二次开发便利度以及硬件适配能力等方面带来了显著提升。除了上述核心特性外，此版本在用户体验层面持续丰富并增强了满足更多场景的API功能，针对大模型场景优化完善了分布式并行策略优化和推理功能增强，在编译安装方面做了比较彻底的易用性改进，对依赖包的安装方式和版本进行了全新同步升级，对系统安全进行了全面加固，对产品文档也进行了全面的纠错检查，同时也对一些废弃代码做了大量的清理以保证架构的简洁性。飞桨 3.0 Beta 版本在不使用新特性的情况下，表现仍然是成熟稳定的，每个新特性都提供了可灵活进行控制的开关，方便用户快速了解相关产品功能和体验对比。 ## 不兼容升级 - 飞桨API支持隐式类型提升。在加减乘除等最常用的计算中，如果两个输入的数据类型不一样，就需要确定输出的数据类型问题。飞桨历史上的现状是部分支持且实际规则并不清楚，客观上表现为动静不一致、API和运算符重载不一致及不符合交换率，特别是在大模型广泛使用 bf16/fp16 与 fp32 进行混合计算时容易出现非预期问题且难以定位。飞桨从3.0 beta版本开始，明确了[隐式数据类型提升规则](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/guides/advanced/auto_type_promotion_cn.html)，其中详细定义了 Tensor与Tensor 和 Tensor与1个数(Scalar)计算结果的类型，保证了计算符合交换律，运算符重载与二元 API 结果一致，动态图与静态图结果一致。更符合用户理解和业界习惯。[#60638](https://github.com/PaddlePaddle/Paddle/pull/60638), [#63842](https://github.com/PaddlePaddle/Paddle/pull/63842), [#60011](https://github.com/PaddlePaddle/Paddle/pull/60011) ## 废弃功能 - 支持0维Tensor已经稳定了2个版本，本版本取消了在一些情况下将0维Tensor转成只含1个元素的1维Tensor的开关`FLAGS_set_to_1d`，这个开关是为了兼容一些套件中用1个元素的1维Tensor表示0维Tensor的不正确写法。即当前飞桨完全区分0维Tensor和只含1个元素的1维Tensor的语义，两者不等价。[#61227](https://github.com/PaddlePaddle/Paddle/pull/61227) ## 贡献者名单 6clc, Android zhang, Asthestarsfalll, Ataf Fazledin Ahamed, Aurelius84, AyaseNana, Baizhou Zhang, bapijun, BiynXu, Botao Zhou, Bo Zhang, bukejiyu, caozhou, chalsliu, Chang Xu, Charles-hit, chen2016013, Chen Zhiyang, C.J.0_0, cmcamdy, co63oc, coco, cyber-pioneer, cyberslack_lee, danleifeng, diadestiny, Difer, Dmovic, Eddie-Wang, Eddie Zhang, engineer1109, enzodechine, fanhaoxuee, feifei-111, flying-forever, Frank Lin, freeliuzc, fsczz, Galaxy1458, GGBond8488, Ghost Screaming, gongweibao, gouzil, Guoxia Wang, handiz, HankYang, Haohongxiang, haosicheng, hess, hjyp, hong, Hongqing-work, Hongwen Xin, HongyuJia, houj04, huangjiyi, Huihuang Zheng, hxzd5568, hyDONG, HydrogenSulfate, idontkonwher, iLeGend, Jeng Bai-Cheng, Jianbang Yang, Jia Wenxuan, JYChen, jzhang533, JZ-LIANG, Kai Song, kangguangli, kevin, Kunbo Ding, lanxianghit, Leo Chen, Leo Guo, lijialin03, lijin23, linkk08, Liujie0926, Liuyinfeng, liu zhengxi, liuzhenhai93, liym27, LiYuRio, lizexu123, LoneRanger, Longzhi Wang, Lucas, Lu Qi, lzy, lzydev, MayYouBeProsperous, megemini, Meiyim, ming1753, Mingdong Wang, ndren, NeroLoh, NetPunk, Nguyen Cong Vinh, Nyakku Shigure, Omri Alon, onepick, ooo oo, pangengzheng, PommesPeter, Qi Li, QingshuChen, Qi Shao, RedContritio, Reese Wang, RichardWooSJTU, risemeup1, Roc, ronnywang, Ruibiao Chen, Ruibin Cheung, RuohengMa, Ryan, Shaopeng Ling, ShenLiang, Shijie, Shuhao Liang, Siming Dai, skywalker2012, smallpoxscattered, sneaxiy, Sonder, Sunny-bot1, Tao Luo, tc20042008, Terry, Tian, tianhaodongbd, tianshuo78520a, Tianyu Feng, Tian Zheng, Tongkai, Travis-Lee, unseenme, Vigi Zhang, walkalone20, Wang Bojun, wanghuancoder, wangna11BD, Wang Xin, Wangzheee, WangZhen, wanly young, wawltor, wendaxiao, Wen Sun, wentao yu, Wenyu, wenzhe.wang, Winters Montagne, winter-wang, WoWYoYLoL, Wu Chencan, Wu Fei, wuhuachaocoding, Xianduo Li, XiangGao, XiaociZhang, xiaoguoguo626807, xiaoxiaohehe001, Xiao Xiyuan, Xiaoxu Chen, xiaoyao0115, xiaoye, xingmingyyj, Xinyi_LI, Xinyu Yang, xiongkun, xuxinyi389, xysheng-baidu, yangguohao, YibLiu, Yichen Zhang, yinfan98, yinwei, Yiqun Liu, YKTian, Yuang Liu, Yuanle Liu, YuanRisheng, yuguo, yujun, yulangz, YUNSHEN XIE, zbt78, ZelinMa557, Zero Rains, Zeyu Chen, zhangbo9674, Zhang,Lirong, Zhang Ting, zhangyikun02, zhangyuqin1998, Zhan Rongrui, zhaohaixu, zhaoyingli, Zhenghai Zhang, zhengzhonghui, zhink, ZhouMengLei1999, zhouzj, zhupengyang, zhurou603, zhuyipin, zhwesky2010, Zichao, zxcd, zyfncg, zyt1024, 东百月, 傅剑寒, 周周周, 周波涛, 张春乔, 萧 ## 完整版本由于字数超出限制，完整版本请大家访问以下链接： - [完整中文版本](https://github.com/PaddlePaddle/Paddle/wiki/PaddlePaddle-3.0.0%E2%80%90beta0-Release-Note-CN)

PaddlePaddle 2.6.1 Release Notev2.6.1

? · 2024-04-09

# 发版说明此版本在新功能方面，引入了对Fake GroupWise Quant的支持，有助于用户更好地进行模型量化。同时新增了图神经网络训练引擎PGLBox，支持超大规模图模型GPU多机多卡高效训练。此外，增加了对自定义设备的支持，进一步扩展PaddlePaddle的功能范围。在bug方面，解决了一些核心功能、数据加载以及网络通信等方面的问题。修复了多个安全问题，包括一些潜在的安全漏洞，进一步提高框架代码安全性，并更新了安全公告。 ## 新特性 - **支持Fake GroupWise量化（#61900）**：新增对一种量化方法的支持，可以提高模型的性能效率。 - **支持图神经网络GPU训练（#60495，#62111）**：新增图神经网络训练引擎PGLBox，支持超大规模图模型GPU多机多卡高效训练。 - **其他改进**：支持tile op int8模式推理，并添加vlog语句 (#60261)，repeat_interleave支持bfloat16数据类型的Tensor输入 (#61854)，自定义设备支持动态图模式的c_embedding算子 (#60774)，在CINN（自定义中间网络）框架中，将IntrinsicOps添加到`ir_codes_collector`中。 ## Bug修复 - **修复权重量化内核错误（#60184）**：解决了在权重量化内核中当`n`不能被64整除时的问题。 - **修复量化感知测试问题（#61211）**：修复了量化感知训练（QAT）测试中的问题，以确保其正常运行。 - **修复Paddle-TRT集成问题（#61806, #61605, #61966）**：对Paddle-TRT集成进行了多项修复，包括缓存键值(KV)量化和单元测试失败问题。 - **禁用LLM_INT8 UT（#62282）**：禁用了大型语言模型（LLM）INT8精度的单元测试，以避免不必要的运行时。 - **修复test_benchmark单测编译失败问题（#61427）**：修复了test_benchmark单测编译失败的问题#60092。 - **修复工具包的数据加载器（#61867）**：对工具包的数据加载器进行了必要的更正。 - **修复put_along_axis系列问题（#62065）**：对reduce参数新增支持min/max/mean三种可选值，修复了reduce=add/mul下的反向梯度计算Bug，修复了reduce=mul下的GPU前向计算Bug，修复了size过大时的前向计算Bug。 - **修复Windows平台下的编译Bug（#60308）**：修复windows平台下的编译找不到common库的bug。 - **修复OpenSSL-CPU编译错误（#62079）**：修复cpu-openblas编译场景中未正确链接 Python 库导致的编译bug。 ## 安全修复 - **修复安全问题（#61161, #61294, #61032, #61285, #61162, #61389, #61356, #61827）**：修复了涉及绘图功能、解压缩和命令执行的多个安全问题，以确保框架的安全性和完整性。 - **禁用特定测试（#60173, #60662, #60786）**：禁用了某些未通过或不适用于特定配置或版本的测试。 - **修复框架不同组件Bug（#61631, #62117, #60840, #62104, #60217, #60930, #61380）**：在框架的不同组件中修复了各种Bug。 - **修复自定义设备功能（#60616）**：打开自定义设备中生成器偏移量的编译控制。 - **内存和分配检查（#60208, #60545, #61847, #62111, #62057, #62278）**：实施了与内存分配和管理相关的检查和修复，以防止崩溃并提高稳定性。 ## 文档 - **文档风格改进（#61688）**：改进了文档的样式和格式。 - **安全公告更新（#60532, #60649）**：更新了2023年的安全公告，以通知用户潜在的安全问题。

PaddlePaddle 2.6.0 Release Notev2.6.0

? · 2024-01-08

# 1. 重要更新 - **新一代中间表示 PIR**：为了进一步提升飞桨框架的可扩展性，研制了新一代中间表示 PIR（Paddle Intermediate Representation）。实现系统性的抽象飞桨框架底层核心概念，如：Operation、Attribute 和 Type 等，为开发者提供了灵活、高效的基础组件。通过引入 Dialect 机制，可以全面、分层次地满足各模块对中间表示的需求，从而极大地提升了框架的扩展性。PIR 严格遵循 SSA（Static Single Assignment）原则，在实现了顶层结构的统一的同时，还确保了“算子顺序性”与“计算图语义”的和谐共存。此外，PIR 提供了更为简洁、低成本的 Pass 开发流程，内置了一系列丰富且功能完备的 Pass 优化策略，为大型模型的极致性能优化提供了技术支撑。 - **动转静编译优化架构**：为了进一步提升框架的模型开发性能，飞桨动转静训练能力全面升级，支持自适应的图构建能力，在 700 多个飞桨产业级模型上验证，一键动转静训练成功率达到 100%。同时，飞桨框架的神经网络编译器 CINN 整合入飞桨主 Repo，使得编译器与飞桨更加融为一体。CINN 完成了架构的梳理和扩展能力的完善，提升系统稳定性。基于 PIR 完成动转静、组合算子、执行器和编译器的紧密联动，为飞桨框架整体性能的提升提供了更大的空间。 - **增强动态图分布式能力**：大模型对框架的分布式训练性能提出了更高的需求。飞桨在通信库、图分析、分布式策略和任务启停等维度进行了全面优化，增强了飞桨动态图的分布式计算能力，为大型模型高效训练提供了支持。在性能方面，通过减少流水线 GPU 显存占用、采用 TensorFusion 技术、实现通信计算 overlap 以及减少非必要的数据同步拷贝等方式，进一步提升了训练性能。同时，通过环境变量控制 Optimizer 等方式提高了混合并行调试的灵活性。此外，通过相关 Bug 的修复，显著提升了系统的稳定性。 - **动静统一自动并行架构**：为了进一步降低大模型编程和优化难度，飞桨对动静统一的半自动并行（Auto Parallel）编程范式进行了全面的优化，简化了开发者的编程复杂度。开发者无需深入了解手动并行编程范式下的复杂概念和 API 接口，如行切分、列切分等，仅需通过少量的张量切分标注即可完成混合并行模型的构建，框架便能够自动推导出所有张量和算子的分布式切分状态，并添加合适的通信算子。同时支持一键动转静进行分布式训练，使开发者能够高效地实现任意混合并行策略，大幅简化了混合并行训练代码的开发过程。 - **硬件适配方案(CustomDevice)**：大模型场景下新硬件并行训练需求增加，飞桨新增了对分布式高级策略、自定义算子和自定义融合策略的支持。升级了分布式通信库，新增了对 MP、GroupShared、PP、SP 和 MOE 等多项高级分布式策略的支持。同时，支持厂商灵活接入不同颗粒度的 Transformer 算子库并通过融合 Pass 修改计算图进行性能加速。 - **安装和开发体验**：采用模块化编译的方式优化了 CMake 代码的逻辑，提升了飞桨全量编译和增量编译的效率，提升了 RD 开发效率，同时支持了 Python3.12，CUDA12，Hopper 架构编译，并引入 Clang 等工具全面优化了代码格式。此外，将 C++单测从链接静态库的方式转变为链接动态库，减小编译体积。这些改进措施为用户提供更加流畅、高效的安装和开发体验。 # 2. 不兼容升级 - 为了避免误用，去除了 0 维 Tensor 兼容态开关，实现 API 行为和业界主流习惯一致。在上一个版本中，我们已经支持 0 维 Tensor，但是考虑到尽量避免部分模型的报错，添加了兼容态开关。即在一些模型套件使用较多且没有修改完成的场景中还是默认使用只有 1 个元素的 1 维 Tensor 来替代 0 维 Tensor。这个版本去除了兼容态开关，在任何场景中都不会再使用只有 1 个元素的 1 维 Tensor 来替代 0 维 Tensor，应该支持 0 维 Tensor 的 376 个 API 的行为都完成了修正和统一，彻底完成对 0 维 Tensor 的支持。[#57036](https://github.com/PaddlePaddle/Paddle/pull/57036), [#54581](https://github.com/PaddlePaddle/Paddle/pull/54581), [#54500](https://github.com/PaddlePaddle/Paddle/pull/54500) - 为了提升 API 易用性，将 paddle.nn.functional.diag_embed 精简为 paddle.diag_embed，并支持 Tensor.diag_embed 方式使用。 [#58223](https://github.com/PaddlePaddle/Paddle/pull/58223) - 为了解决在静态图下 Tensor 索引写（如 tensor[0] = 10）导致的微分计算错误问题，并符合静态图的规范，本版本引入了 paddle.static.setitem API。在静态图环境中，更推荐使用此 API 来支持 tensor 的索引写操作，而非下标运算符。这一变化并不影响动态图环境，其中仍允许使用下标运算符进行索引写操作。[#53682](https://github.com/PaddlePaddle/Paddle/pull/53682) - 本版本中 paddle.fluid API 全面退出历史舞台。在本次更新中，我们彻底移除了所有 paddle.fluid API，并删除了 fluid 目录。同时，飞桨底层的少量公共组件已被整合至 paddle.base 目录中。使得飞桨用户无需再关注 fluid 相关概念和接口，进一步简化了飞桨 API 体系，提升可读性。[#56576](https://github.com/PaddlePaddle/Paddle/pull/56576), [#54424](https://github.com/PaddlePaddle/Paddle/pull/54424), [#54829](https://github.com/PaddlePaddle/Paddle/pull/54829), [#53992](https://github.com/PaddlePaddle/Paddle/pull/53992), [#54806](https://github.com/PaddlePaddle/Paddle/pull/54806), [#55754](https://github.com/PaddlePaddle/Paddle/pull/55754), [#55986](https://github.com/PaddlePaddle/Paddle/pull/55986), [#55345](https://github.com/PaddlePaddle/Paddle/pull/55345), [#56099](https://github.com/PaddlePaddle/Paddle/pull/56099), [#51717](https://github.com/PaddlePaddle/Paddle/pull/51717), [#54152](https://github.com/PaddlePaddle/Paddle/pull/54152), [#55522](https://github.com/PaddlePaddle/Paddle/pull/55522), [#55757](https://github.com/PaddlePaddle/Paddle/pull/55757), [#58521](https://github.com/PaddlePaddle/Paddle/pull/58521), [#54936](https://github.com/PaddlePaddle/Paddle/pull/54936), [#55007](https://github.com/PaddlePaddle/Paddle/pull/55007), [#55661](https://github.com/PaddlePaddle/Paddle/pull/55661), [#55970](https://github.com/PaddlePaddle/Paddle/pull/55970) # 3. 训练框架（含分布式） ## Python API ### 升级 Tensor 索引机制本版本全面优化了 Tensor 的基础索引、高级索引以及联合索引功能，以更好地符合业界标准与用户习惯。具体包括：在基础索引中增加了对 view 的支持，修正了高级索引中的一些错误行为，并实现了联合索引的读取功能。此外，我们还将索引解析下沉到 C++层面，改进了高级索引算子的性能，并移除了 bool 索引中的冗余计算。通过这些优化措施，Tensor 的基础索引、高级索引和联合索引性能得到了全面提升。[#56893](https://github.com/PaddlePaddle/Paddle/pull/56893), [#58643](https://github.com/PaddlePaddle/Paddle/pull/58643), [#57986](https://github.com/PaddlePaddle/Paddle/pull/57986), [#56272](https://github.com/PaddlePaddle/Paddle/pull/56272), [#58856](https://github.com/PaddlePaddle/Paddle/pull/58856), [#55211](https://github.com/PaddlePaddle/Paddle/pull/55211), [#57023](https://github.com/PaddlePaddle/Paddle/pull/57023), [#56613](https://github.com/PaddlePaddle/Paddle/pull/56613), [#55602](https://github.com/PaddlePaddle/Paddle/pull/55602), [#59281](https://github.com/PaddlePaddle/Paddle/pull/59281), [#57737](https://github.com/PaddlePaddle/Paddle/pull/57737) ### 升级 Inplace 机制在之前的版本中，为了确保反向微分计算的正确性，当某个 API 的反向计算依赖于其前向输入数据时，飞桨会避免使用 Inplace 操作方式，因为这种方法可能会覆盖原始输入数据。虽然这种机制简化了实现过程，但也限制了许多 API 实现 Inplace 功能，从而影响了用户体验。在本版本中，飞桨对 Inplace 机制进行了全面升级。实现自动检测反向计算对前向输入的依赖关系，并在需要时保存这些输入数据，从而支持更多的 Inplace 操作。这一改进不仅提升了内存使用效率，还增强了 API 的功能性。此外，我们新增了 109 个支持 Inplace 操作的 API，包括 paddle.abs*、paddle.sin*/cos*/tan*、比较操作如 paddle.greater*than*/less*than*/equal*、逻辑操作如 paddle.logical_and*/logical*or*/logical*not*，以及 paddle.neg*和 paddle.log*等。在丰富飞桨的功能集同时，提升了用户在数值计算和深度学习任务中的效率与便捷性。[#54683](https://github.com/PaddlePaddle/Paddle/pull/54683), [#55078](https://github.com/PaddlePaddle/Paddle/pull/55078), [#55576](https://github.com/PaddlePaddle/Paddle/pull/55576), [#56888](https://github.com/PaddlePaddle/Paddle/pull/56888), [#55509](https://github.com/PaddlePaddle/Paddle/pull/55509), [#57093](https://github.com/PaddlePaddle/Paddle/pull/57093) ### 其他新增 API - 新增 paddle.nn.functional.scaled_dot_product_attention，显著提升大模型中注意力（attention）机制的计算效率，更好地满足大规模深度学习模型对高性能计算的需求。。[#55242](https://github.com/PaddlePaddle/Paddle/pull/55242) - 新增了一系列科学计算相关 API，包括 paddle.cummax 和 paddle.cummin 用于累积最大值和最小值的计算，paddle.index_fill 和 paddle.masked_fill 用于按索引或掩码填充张量，paddle.linalg.pca_lowrank 用于低秩主成分分析，paddle.hypot 用于计算直角三角形的斜边长，以及 paddle.atleast_1d、paddle.atleast_2d 和 paddle.atleast_3d 用于确保张量至少有一维、二维或三维。同时，我们还提供了 paddle.select_scatter 和 paddle.diagonal_scatter 用于更灵活地选择和散列张量数据，以及 paddle.multigammaln 用于计算多伽马函数的自然对数。此外，本版本新增优化器相关 API，包括：paddle.optimizer.lr.LinearLR 和 paddle.optimizer.lr.CosineAnnealingWarmRestarts 学习率调度策略；引入了 paddle.io.SubsetRandomSampler 以支持从数据子集中进行随机采样。这些新增 API 将进一步提升飞桨在各类应用场景中的灵活性和高效性。。 [#57416](https://github.com/PaddlePaddle/Paddle/pull/57416), [#53546](https://github.com/PaddlePaddle/Paddle/pull/53546), [#53743](https://github.com/PaddlePaddle/Paddle/pull/53743), [#57295](https://github.com/PaddlePaddle/Paddle/pull/57295), [#57726](https://github.com/PaddlePaddle/Paddle/pull/57726), [#58764](https://github.com/PaddlePaddle/Paddle/pull/58764), [#58323](https://github.com/PaddlePaddle/Paddle/pull/58323), [#57720](https://github.com/PaddlePaddle/Paddle/pull/57720), [#58209](https://github.com/PaddlePaddle/Paddle/pull/58209), [#58214](https://github.com/PaddlePaddle/Paddle/pull/58214), [#57792](https://github.com/PaddlePaddle/Paddle/pull/57792), [#51395](https://github.com/PaddlePaddle/Paddle/pull/51395), [#57724](https://github.com/PaddlePaddle/Paddle/pull/57724), [#57355](https://github.com/PaddlePaddle/Paddle/pull/57355), [#57744](https://github.com/PaddlePaddle/Paddle/pull/57744), [#58244](https://github.com/PaddlePaddle/Paddle/pull/58244), [#57599](https://github.com/PaddlePaddle/Paddle/pull/57599), [#59343](https://github.com/PaddlePaddle/Paddle/pull/59343), [#57879](https://github.com/PaddlePaddle/Paddle/pull/57879) ## 新一代中间表示（PIR） PIR（Paddle Intermediate Representation）对底层的核心概念如 Operation、Attribute 和 Type 等进行了系统性的抽象，为开发者构建了一套灵活且强大的基础组件。此外，通过引入 Dialect 这一概念，飞桨框架能够全面且分层次地管理各模块对中间表示（IR）的需求，并支持开发者根据特定需求定制化扩展 Dialect，从而显著提升了框架的扩展性和适应性。在设计上，PIR 严格遵循 SSA（Static Single Assignment）原则，统一了顶层结构，实现了“算子顺序性”与“计算图语义”的兼容表示，为复杂的计算流程提供了清晰且一致的视图。为了进一步优化大模型的性能，PIR 还提供了一套更加简洁、低成本的 Pass 开发流程，包括 DRR（Declarative Rewrite Rule）和模式重写器（Pattern Rewriter）。同时，内置了一系列丰富且功能完备的 Pass 优化策略，这些策略能够针对大模型的特点进行深度优化，从而为大模型的极致性能提供了强有力支撑。通过这些创新设计和优化手段，PIR 为飞桨框架的高效运行和持续扩展奠定了坚实基础。 ### 新功能 - 系统抽象了 IR 底层的核心概念，为开发者提供了灵活的基础组件，如 Operation、Attribute、Value、Type、Trait、Interface 等。[#56354](https://github.com/PaddlePaddle/Paddle/pull/56354),[#57106](https://github.com/PaddlePaddle/Paddle/pull/57106),[#57349](https://github.com/PaddlePaddle/Paddle/pull/57349),[#54844](https://github.com/PaddlePaddle/Paddle/pull/54844),[#54984](https://github.com/PaddlePaddle/Paddle/pull/54984),[#54565](https://github.com/PaddlePaddle/Paddle/pull/54565),[#54562](https://github.com/PaddlePaddle/Paddle/pull/54562),[#57249](https://github.com/PaddlePaddle/Paddle/pull/57249),[#57550](https://github.com/PaddlePaddle/Paddle/pull/57550),[#59278](https://github.com/PaddlePaddle/Paddle/pull/59278),[#54875](https://github.com/PaddlePaddle/Paddle/pull/54875),[#55041](https://github.com/PaddlePaddle/Paddle/pull/55041),[#54987](https://github.com/PaddlePaddle/Paddle/pull/54987),[#55903](https://github.com/PaddlePaddle/Paddle/pull/55903),[#57582](https://github.com/PaddlePaddle/Paddle/pull/57582),[#57580](https://github.com/PaddlePaddle/Paddle/pull/57580),[#58052](https://github.com/PaddlePaddle/Paddle/pull/58052),[#55322](https://github.com/PaddlePaddle/Paddle/pull/55322),[#57418](https://github.com/PaddlePaddle/Paddle/pull/57418),[#57635](https://github.com/PaddlePaddle/Paddle/pull/57635),[#55328](https://github.com/PaddlePaddle/Paddle/pull/55328),[#57463](https://github.com/PaddlePaddle/Paddle/pull/57463),[#59791](https://github.com/PaddlePaddle/Paddle/pull/59791),[#59821](https://github.com/PaddlePaddle/Paddle/pull/59821),[#59115](https://github.com/PaddlePaddle/Paddle/pull/59115),[#57461](https://github.com/PaddlePaddle/Paddle/pull/57461),[#59392](https://github.com/PaddlePaddle/Paddle/pull/59392),[#57373](https://github.com/PaddlePaddle/Paddle/pull/57373),[#59118](https://github.com/PaddlePaddle/Paddle/pull/59118) - 新增引入 Dialect 机制，支持全面、分层次管理框架各个模块对中间表示的需求，且内置了 Builtin Dialect，支持开发者根据需求自定义化扩展 Dialect。 [#56325](https://github.com/PaddlePaddle/Paddle/pull/56325),[#57539](https://github.com/PaddlePaddle/Paddle/pull/57539),[#54682](https://github.com/PaddlePaddle/Paddle/pull/54682),[#55381](https://github.com/PaddlePaddle/Paddle/pull/55381),[#56156](https://github.com/PaddlePaddle/Paddle/pull/56156),[#56431](https://github.com/PaddlePaddle/Paddle/pull/56431),[#56615](https://github.com/PaddlePaddle/Paddle/pull/56615),[#57103](https://github.com/PaddlePaddle/Paddle/pull/57103),[#57209](https://github.com/PaddlePaddle/Paddle/pull/57209) - 规范化了飞桨静态图算子体系，新增 OperatorDialect、KernelDialect，以 Dialect 形式分层管理编译期和执行期的算子表示概念差异性，架构更加清晰。[#56284](https://github.com/PaddlePaddle/Paddle/pull/56284),[#54469](https://github.com/PaddlePaddle/Paddle/pull/54469),[#58660](https://github.com/PaddlePaddle/Paddle/pull/58660),[#58975](https://github.com/PaddlePaddle/Paddle/pull/58975),[#56680](https://github.com/PaddlePaddle/Paddle/pull/56680),[#54790](https://github.com/PaddlePaddle/Paddle/pull/54790),[#54826](https://github.com/PaddlePaddle/Paddle/pull/54826),[#54840](https://github.com/PaddlePaddle/Paddle/pull/54840),[#55699](https://github.com/PaddlePaddle/Paddle/pull/55699),[#55648](https://github.com/PaddlePaddle/Paddle/pull/55648),[#55880](https://github.com/PaddlePaddle/Paddle/pull/55880),[#56101](https://github.com/PaddlePaddle/Paddle/pull/56101),[#56754](https://github.com/PaddlePaddle/Paddle/pull/56754),[#54944](https://github.com/PaddlePaddle/Paddle/pull/54944),[#56836](https://github.com/PaddlePaddle/Paddle/pull/56836),[#57185](https://github.com/PaddlePaddle/Paddle/pull/57185),[#58757](https://github.com/PaddlePaddle/Paddle/pull/58757),[#56243](https://github.com/PaddlePaddle/Paddle/pull/56243),[#56436](https://github.com/PaddlePaddle/Paddle/pull/56436),[#57741](https://github.com/PaddlePaddle/Paddle/pull/57741),[#59124](https://github.com/PaddlePaddle/Paddle/pull/59124),[#57054](https://github.com/PaddlePaddle/Paddle/pull/57054),[#56984](https://github.com/PaddlePaddle/Paddle/pull/56984),[#57403](https://github.com/PaddlePaddle/Paddle/pull/57403),[#57904](https://github.com/PaddlePaddle/Paddle/pull/57904),[#58031](https://github.com/PaddlePaddle/Paddle/pull/58031),[#56924](https://github.com/PaddlePaddle/Paddle/pull/56924),[#59270](https://github.com/PaddlePaddle/Paddle/pull/59270),[#55343](https://github.com/PaddlePaddle/Paddle/pull/55343),[#56557](https://github.com/PaddlePaddle/Paddle/pull/56557),[#55693](https://github.com/PaddlePaddle/Paddle/pull/55693),[#54428](https://github.com/PaddlePaddle/Paddle/pull/54428) - 新增 ShapeDialect，内置了丰富的 Shape 操作算子，用于面向 AI 编译器的动态 Shape 约束关系和表达式的构建。[#56727](https://github.com/PaddlePaddle/Paddle/pull/56727),[#59254](https://github.com/PaddlePaddle/Paddle/pull/59254),[#58368](https://github.com/PaddlePaddle/Paddle/pull/58368),[#57069](https://github.com/PaddlePaddle/Paddle/pull/57069),[#57337](https://github.com/PaddlePaddle/Paddle/pull/57337),[#56351](https://github.com/PaddlePaddle/Paddle/pull/56351),[#57029](https://github.com/PaddlePaddle/Paddle/pull/57029),[#58036](https://github.com/PaddlePaddle/Paddle/pull/58036),[#59032](https://github.com/PaddlePaddle/Paddle/pull/59032),[#57961](https://github.com/PaddlePaddle/Paddle/pull/57961),[#56427](https://github.com/PaddlePaddle/Paddle/pull/56427),[#57459](https://github.com/PaddlePaddle/Paddle/pull/57459) - 统一了框架 Program 顶层结构，支持兼容表示“算子顺序性”和“计算图语义”，解耦对 ir::Graph 的依赖，且严格遵循 SSA （即 Static Single Assignment）原则。[#59369](https://github.com/PaddlePaddle/Paddle/pull/59369),[#54563](https://github.com/PaddlePaddle/Paddle/pull/54563),[#57051](https://github.com/PaddlePaddle/Paddle/pull/57051),[#57306](https://github.com/PaddlePaddle/Paddle/pull/57306),[#57857](https://github.com/PaddlePaddle/Paddle/pull/57857) - 新增了 IrPrinter 和 IrPaser 组件，支持 PIR Program 的序列化和反序列化功能，提供了友好的 PIR 开发调试体验。[#55695](https://github.com/PaddlePaddle/Paddle/pull/55695),[#59449](https://github.com/PaddlePaddle/Paddle/pull/59449),[#54369](https://github.com/PaddlePaddle/Paddle/pull/54369),[#54499](https://github.com/PaddlePaddle/Paddle/pull/54499),[#55518](https://github.com/PaddlePaddle/Paddle/pull/55518),[#55784](https://github.com/PaddlePaddle/Paddle/pull/55784),[#57180](https://github.com/PaddlePaddle/Paddle/pull/57180),[#57471](https://github.com/PaddlePaddle/Paddle/pull/57471),[#54859](https://github.com/PaddlePaddle/Paddle/pull/54859),[#54968](https://github.com/PaddlePaddle/Paddle/pull/54968),[#55209](https://github.com/PaddlePaddle/Paddle/pull/55209),[#57314](https://github.com/PaddlePaddle/Paddle/pull/57314),[#57969](https://github.com/PaddlePaddle/Paddle/pull/57969) - 基于 DRR（即 Declarative Rewrite Rule）和 Pattern Rewriter 构建了全新、简洁、低成本的 Pass 开发体系，并内置了一系列丰富且功能完备的 Pass 优化策略，加速训练和推理执行过程。[#54385](https://github.com/PaddlePaddle/Paddle/pull/54385),[#54738](https://github.com/PaddlePaddle/Paddle/pull/54738),[#55859](https://github.com/PaddlePaddle/Paddle/pull/55859),[#56638](https://github.com/PaddlePaddle/Paddle/pull/56638),[#57090](https://github.com/PaddlePaddle/Paddle/pull/57090),[#58673](https://github.com/PaddlePaddle/Paddle/pull/58673),[#59415](https://github.com/PaddlePaddle/Paddle/pull/59415),[#56729](https://github.com/PaddlePaddle/Paddle/pull/56729),[#58655](https://github.com/PaddlePaddle/Paddle/pull/58655) - 新增 ProgramTranslator 组件，支持由 ProgramDesc 一键转换为飞桨新一代 IR 表示，并提供了易用的 C++和 Python 接口。[#55433](https://github.com/PaddlePaddle/Paddle/pull/55433),[#54470](https://github.com/PaddlePaddle/Paddle/pull/54470),[#58044](https://github.com/PaddlePaddle/Paddle/pull/58044),[#58390](https://github.com/PaddlePaddle/Paddle/pull/58390),[#58100](https://github.com/PaddlePaddle/Paddle/pull/58100),[#55403](https://github.com/PaddlePaddle/Paddle/pull/55403),[#55406](https://github.com/PaddlePaddle/Paddle/pull/55406),[#54719](https://github.com/PaddlePaddle/Paddle/pull/54719),[#56550](https://github.com/PaddlePaddle/Paddle/pull/56550),[#55448](https://github.com/PaddlePaddle/Paddle/pull/55448),[#55453](https://github.com/PaddlePaddle/Paddle/pull/55453),[#56294](https://github.com/PaddlePaddle/Paddle/pull/56294),[#56308](https://github.com/PaddlePaddle/Paddle/pull/56308),[#56842](https://github.com/PaddlePaddle/Paddle/pull/56842),[#58517](https://github.com/PaddlePaddle/Paddle/pull/58517) - 借助自动代码生成技术，一键生成飞桨框架全量静态图算子表示。将静态图组网逻辑下沉至 C++端，统一绑定到\_C_ops 模块，大幅精简 Python 端 API 代码，实现飞桨框架 API 的极致化动静统一，升级了诸多 Python API 以支持新 IR 静态图组网。[#56570](https://github.com/PaddlePaddle/Paddle/pull/56570),[#55745](https://github.com/PaddlePaddle/Paddle/pull/55745),[#56955](https://github.com/PaddlePaddle/Paddle/pull/56955),[#57298](https://github.com/PaddlePaddle/Paddle/pull/57298),[#57946](https://github.com/PaddlePaddle/Paddle/pull/57946),[#57248](https://github.com/PaddlePaddle/Paddle/pull/57248),[#56080](https://github.com/PaddlePaddle/Paddle/pull/56080),[#54396](https://github.com/PaddlePaddle/Paddle/pull/54396),[#54551](https://github.com/PaddlePaddle/Paddle/pull/54551),[#56520](https://github.com/PaddlePaddle/Paddle/pull/56520),[#55002](https://github.com/PaddlePaddle/Paddle/pull/55002),[#57067](https://github.com/PaddlePaddle/Paddle/pull/57067),[#59320](https://github.com/PaddlePaddle/Paddle/pull/59320),[#59348](https://github.com/PaddlePaddle/Paddle/pull/59348),[#57164](https://github.com/PaddlePaddle/Paddle/pull/57164),[#57267](https://github.com/PaddlePaddle/Paddle/pull/57267),[#59064](https://github.com/PaddlePaddle/Paddle/pull/59064),[#54340](https://github.com/PaddlePaddle/Paddle/pull/54340),[#54895](https://github.com/PaddlePaddle/Paddle/pull/54895),[#55004](https://github.com/PaddlePaddle/Paddle/pull/55004),[#56196](https://github.com/PaddlePaddle/Paddle/pull/56196),[#56862](https://github.com/PaddlePaddle/Paddle/pull/56862),[#58991](https://github.com/PaddlePaddle/Paddle/pull/58991),[#55428](https://github.com/PaddlePaddle/Paddle/pull/55428),[#55909](https://github.com/PaddlePaddle/Paddle/pull/55909),[#56241](https://github.com/PaddlePaddle/Paddle/pull/56241),[#56526](https://github.com/PaddlePaddle/Paddle/pull/56526),[#56571](https://github.com/PaddlePaddle/Paddle/pull/56571),[#56518](https://github.com/PaddlePaddle/Paddle/pull/56518),[#57016](https://github.com/PaddlePaddle/Paddle/pull/57016),[#56653](https://github.com/PaddlePaddle/Paddle/pull/56653),[#56809](https://github.com/PaddlePaddle/Paddle/pull/56809),[#57158](https://github.com/PaddlePaddle/Paddle/pull/57158),[#55422](https://github.com/PaddlePaddle/Paddle/pull/55422),[#55458](https://github.com/PaddlePaddle/Paddle/pull/55458),[#55432](https://github.com/PaddlePaddle/Paddle/pull/55432),[#55467](https://github.com/PaddlePaddle/Paddle/pull/55467),[#55483](https://github.com/PaddlePaddle/Paddle/pull/55483),[#55419](https://github.com/PaddlePaddle/Paddle/pull/55419),[#55517](https://github.com/PaddlePaddle/Paddle/pull/55517),[#55500](https://github.com/PaddlePaddle/Paddle/pull/55500),[#56674](https://github.com/PaddlePaddle/Paddle/pull/56674),[#57693](https://github.com/PaddlePaddle/Paddle/pull/57693),[#55008](https://github.com/PaddlePaddle/Paddle/pull/55008),[#57166](https://github.com/PaddlePaddle/Paddle/pull/57166),[#57157](https://github.com/PaddlePaddle/Paddle/pull/57157),[#57159](https://github.com/PaddlePaddle/Paddle/pull/57159),[#57175](https://github.com/PaddlePaddle/Paddle/pull/57175),[#57325](https://github.com/PaddlePaddle/Paddle/pull/57325),[#57330](https://github.com/PaddlePaddle/Paddle/pull/57330),[#57415](https://github.com/PaddlePaddle/Paddle/pull/57415),[#57122](https://github.com/PaddlePaddle/Paddle/pull/57122),[#57393](https://github.com/PaddlePaddle/Paddle/pull/57393),[#57344](https://github.com/PaddlePaddle/Paddle/pull/57344),[#57667](https://github.com/PaddlePaddle/Paddle/pull/57667),[#57348](https://github.com/PaddlePaddle/Paddle/pull/57348),[#57700](https://github.com/PaddlePaddle/Paddle/pull/57700),[#58093](https://github.com/PaddlePaddle/Paddle/pull/58093),[#58005](https://github.com/PaddlePaddle/Paddle/pull/58005),[#58081](https://github.com/PaddlePaddle/Paddle/pull/58081),[#58094](https://github.com/PaddlePaddle/Paddle/pull/58094),[#58137](https://github.com/PaddlePaddle/Paddle/pull/58137),[#58287](https://github.com/PaddlePaddle/Paddle/pull/58287),[#58352](https://github.com/PaddlePaddle/Paddle/pull/58352),[#58340](https://github.com/PaddlePaddle/Paddle/pull/58340),[#58363](https://github.com/PaddlePaddle/Paddle/pull/58363),[#58331](https://github.com/PaddlePaddle/Paddle/pull/58331),[#58343](https://github.com/PaddlePaddle/Paddle/pull/58343),[#58317](https://github.com/PaddlePaddle/Paddle/pull/58317),[#58450](https://github.com/PaddlePaddle/Paddle/pull/58450),[#58377](https://github.com/PaddlePaddle/Paddle/pull/58377),[#58466](https://github.com/PaddlePaddle/Paddle/pull/58466),[#58470](https://github.com/PaddlePaddle/Paddle/pull/58470),[#58491](https://github.com/PaddlePaddle/Paddle/pull/58491),[#58546](https://github.com/PaddlePaddle/Paddle/pull/58546),[#58587](https://github.com/PaddlePaddle/Paddle/pull/58587),[#58453](https://github.com/PaddlePaddle/Paddle/pull/58453),[#58634](https://github.com/PaddlePaddle/Paddle/pull/58634),[#58604](https://github.com/PaddlePaddle/Paddle/pull/58604),[#58605](https://github.com/PaddlePaddle/Paddle/pull/58605),[#58593](https://github.com/PaddlePaddle/Paddle/pull/58593),[#58675](https://github.com/PaddlePaddle/Paddle/pull/58675),[#58699](https://github.com/PaddlePaddle/Paddle/pull/58699),[#58384](https://github.com/PaddlePaddle/Paddle/pull/58384),[#58629](https://github.com/PaddlePaddle/Paddle/pull/58629),[#58579](https://github.com/PaddlePaddle/Paddle/pull/58579),[#58695](https://github.com/PaddlePaddle/Paddle/pull/58695),[#58548](https://github.com/PaddlePaddle/Paddle/pull/58548),[#58688](https://github.com/PaddlePaddle/Paddle/pull/58688),[#58792](https://github.com/PaddlePaddle/Paddle/pull/58792),[#58843](https://github.com/PaddlePaddle/Paddle/pull/58843),[#58840](https://github.com/PaddlePaddle/Paddle/pull/58840),[#58718](https://github.com/PaddlePaddle/Paddle/pull/58718),[#58883](https://github.com/PaddlePaddle/Paddle/pull/58883),[#58785](https://github.com/PaddlePaddle/Paddle/pull/58785),[#58608](https://github.com/PaddlePaddle/Paddle/pull/58608),[#58781](https://github.com/PaddlePaddle/Paddle/pull/58781),[#58783](https://github.com/PaddlePaddle/Paddle/pull/58783),[#58429](https://github.com/PaddlePaddle/Paddle/pull/58429),[#58685](https://github.com/PaddlePaddle/Paddle/pull/58685),[#58696](https://github.com/PaddlePaddle/Paddle/pull/58696),[#58690](https://github.com/PaddlePaddle/Paddle/pull/58690),[#58831](https://github.com/PaddlePaddle/Paddle/pull/58831),[#58929](https://github.com/PaddlePaddle/Paddle/pull/58929),[#58740](https://github.com/PaddlePaddle/Paddle/pull/58740),[#58937](https://github.com/PaddlePaddle/Paddle/pull/58937),[#58782](https://github.com/PaddlePaddle/Paddle/pull/58782),[#58833](https://github.com/PaddlePaddle/Paddle/pull/58833),[#58882](https://github.com/PaddlePaddle/Paddle/pull/58882),[#58935](https://github.com/PaddlePaddle/Paddle/pull/58935),[#58931](https://github.com/PaddlePaddle/Paddle/pull/58931),[#59041](https://github.com/PaddlePaddle/Paddle/pull/59041),[#59040](https://github.com/PaddlePaddle/Paddle/pull/59040),[#58877](https://github.com/PaddlePaddle/Paddle/pull/58877),[#58888](https://github.com/PaddlePaddle/Paddle/pull/58888),[#59042](https://github.com/PaddlePaddle/Paddle/pull/59042),[#58780](https://github.com/PaddlePaddle/Paddle/pull/58780),[#58682](https://github.com/PaddlePaddle/Paddle/pull/58682),[#58815](https://github.com/PaddlePaddle/Paddle/pull/58815),[#58676](https://github.com/PaddlePaddle/Paddle/pull/58676),[#58678](https://github.com/PaddlePaddle/Paddle/pull/58678),[#58446](https://github.com/PaddlePaddle/Paddle/pull/58446),[#59077](https://github.com/PaddlePaddle/Paddle/pull/59077),[#59091](https://github.com/PaddlePaddle/Paddle/pull/59091),[#58661](https://github.com/PaddlePaddle/Paddle/pull/58661),[#58832](https://github.com/PaddlePaddle/Paddle/pull/58832),[#58642](https://github.com/PaddlePaddle/Paddle/pull/58642),[#58698](https://github.com/PaddlePaddle/Paddle/pull/58698),[#59313](https://github.com/PaddlePaddle/Paddle/pull/59313),[#59371](https://github.com/PaddlePaddle/Paddle/pull/59371),[#58700](https://github.com/PaddlePaddle/Paddle/pull/58700),[#58953](https://github.com/PaddlePaddle/Paddle/pull/58953),[#58879](https://github.com/PaddlePaddle/Paddle/pull/58879),[#59469](https://github.com/PaddlePaddle/Paddle/pull/59469),[#59573](https://github.com/PaddlePaddle/Paddle/pull/59573),[#59481](https://github.com/PaddlePaddle/Paddle/pull/59481),[#59419](https://github.com/PaddlePaddle/Paddle/pull/59419),[#59509](https://github.com/PaddlePaddle/Paddle/pull/59509),[#58735](https://github.com/PaddlePaddle/Paddle/pull/58735),[#59616](https://github.com/PaddlePaddle/Paddle/pull/59616),[#59582](https://github.com/PaddlePaddle/Paddle/pull/59582),[#59420](https://github.com/PaddlePaddle/Paddle/pull/59420),[#59500](https://github.com/PaddlePaddle/Paddle/pull/59500),[#58911](https://github.com/PaddlePaddle/Paddle/pull/58911),[#59535](https://github.com/PaddlePaddle/Paddle/pull/59535),[#54891](https://github.com/PaddlePaddle/Paddle/pull/54891),[#56794](https://github.com/PaddlePaddle/Paddle/pull/56794),[#57477](https://github.com/PaddlePaddle/Paddle/pull/57477),[#57929](https://github.com/PaddlePaddle/Paddle/pull/57929),[#57765](https://github.com/PaddlePaddle/Paddle/pull/57765),[#58693](https://github.com/PaddlePaddle/Paddle/pull/58693),[#58603](https://github.com/PaddlePaddle/Paddle/pull/58603),[#56291](https://github.com/PaddlePaddle/Paddle/pull/56291),[#57123](https://github.com/PaddlePaddle/Paddle/pull/57123),[#57317](https://github.com/PaddlePaddle/Paddle/pull/57317),[#57341](https://github.com/PaddlePaddle/Paddle/pull/57341),[#57020](https://github.com/PaddlePaddle/Paddle/pull/57020),[#57324](https://github.com/PaddlePaddle/Paddle/pull/57324),[#57761](https://github.com/PaddlePaddle/Paddle/pull/57761),[#57762](https://github.com/PaddlePaddle/Paddle/pull/57762),[#57907](https://github.com/PaddlePaddle/Paddle/pull/57907),[#57909](https://github.com/PaddlePaddle/Paddle/pull/57909),[#58099](https://github.com/PaddlePaddle/Paddle/pull/58099),[#58110](https://github.com/PaddlePaddle/Paddle/pull/58110),[#58114](https://github.com/PaddlePaddle/Paddle/pull/58114),[#58139](https://github.com/PaddlePaddle/Paddle/pull/58139),[#58144](https://github.com/PaddlePaddle/Paddle/pull/58144),[#58165](https://github.com/PaddlePaddle/Paddle/pull/58165),[#58194](https://github.com/PaddlePaddle/Paddle/pull/58194),[#58138](https://github.com/PaddlePaddle/Paddle/pull/58138),[#58113](https://github.com/PaddlePaddle/Paddle/pull/58113),[#58245](https://github.com/PaddlePaddle/Paddle/pull/58245),[#58318](https://github.com/PaddlePaddle/Paddle/pull/58318),[#58105](https://github.com/PaddlePaddle/Paddle/pull/58105),[#58348](https://github.com/PaddlePaddle/Paddle/pull/58348),[#58235](https://github.com/PaddlePaddle/Paddle/pull/58235),[#58354](https://github.com/PaddlePaddle/Paddle/pull/58354),[#58341](https://github.com/PaddlePaddle/Paddle/pull/58341),[#58445](https://github.com/PaddlePaddle/Paddle/pull/58445),[#58418](https://github.com/PaddlePaddle/Paddle/pull/58418),[#58239](https://github.com/PaddlePaddle/Paddle/pull/58239),[#58473](https://github.com/PaddlePaddle/Paddle/pull/58473),[#58239](https://github.com/PaddlePaddle/Paddle/pull/58239),[#58391](https://github.com/PaddlePaddle/Paddle/pull/58391),[#58501](https://github.com/PaddlePaddle/Paddle/pull/58501),[#58519](https://github.com/PaddlePaddle/Paddle/pull/58519),[#58416](https://github.com/PaddlePaddle/Paddle/pull/58416),[#58588](https://github.com/PaddlePaddle/Paddle/pull/58588),[#58531](https://github.com/PaddlePaddle/Paddle/pull/58531),[#58730](https://github.com/PaddlePaddle/Paddle/pull/58730),[#58773](https://github.com/PaddlePaddle/Paddle/pull/58773),[#58862](https://github.com/PaddlePaddle/Paddle/pull/58862),[#58946](https://github.com/PaddlePaddle/Paddle/pull/58946),[#58500](https://github.com/PaddlePaddle/Paddle/pull/58500),[#56585](https://github.com/PaddlePaddle/Paddle/pull/56585),[#57480](https://github.com/PaddlePaddle/Paddle/pull/57480),[#57433](https://github.com/PaddlePaddle/Paddle/pull/57433),[#58498](https://github.com/PaddlePaddle/Paddle/pull/58498) ### 功能优化 - 升级了静态图执行器，扩展了更多 Kernel Instruction 类型，支持加载 PIR 并高效调度执行，在训练、推理环节都有显存和性能收益。[#54570](https://github.com/PaddlePaddle/Paddle/pull/54570),[#58665](https://github.com/PaddlePaddle/Paddle/pull/58665),[#57291](https://github.com/PaddlePaddle/Paddle/pull/57291),[#54452](https://github.com/PaddlePaddle/Paddle/pull/54452),[#57431](https://github.com/PaddlePaddle/Paddle/pull/57431),[#54692](https://github.com/PaddlePaddle/Paddle/pull/54692),[#55112](https://github.com/PaddlePaddle/Paddle/pull/55112),[#55210](https://github.com/PaddlePaddle/Paddle/pull/55210),[#55401](https://github.com/PaddlePaddle/Paddle/pull/55401),[#55772](https://github.com/PaddlePaddle/Paddle/pull/55772),[#55828](https://github.com/PaddlePaddle/Paddle/pull/55828),[#56148](https://github.com/PaddlePaddle/Paddle/pull/56148),[#54763](https://github.com/PaddlePaddle/Paddle/pull/54763),[#56886](https://github.com/PaddlePaddle/Paddle/pull/56886),[#57284](https://github.com/PaddlePaddle/Paddle/pull/57284),[#57268](https://github.com/PaddlePaddle/Paddle/pull/57268),[#57791](https://github.com/PaddlePaddle/Paddle/pull/57791),[#56789](https://github.com/PaddlePaddle/Paddle/pull/56789),[#56704](https://github.com/PaddlePaddle/Paddle/pull/56704),[#57594](https://github.com/PaddlePaddle/Paddle/pull/57594),[#58397](https://github.com/PaddlePaddle/Paddle/pull/58397),[#58337](https://github.com/PaddlePaddle/Paddle/pull/58337),[#58756](https://github.com/PaddlePaddle/Paddle/pull/58756),[#58371](https://github.com/PaddlePaddle/Paddle/pull/58371) - 面向 PIR 重构了自动微分模块，迁移适配了高阶自动微分功能，优化了 Stop Gradient 传递机制，逻辑更加清晰，功能更加鲁棒。[#55660](https://github.com/PaddlePaddle/Paddle/pull/55660),[#57084](https://github.com/PaddlePaddle/Paddle/pull/57084),[#56890](https://github.com/PaddlePaddle/Paddle/pull/56890),[#58942](https://github.com/PaddlePaddle/Paddle/pull/58942),[#59373](https://github.com/PaddlePaddle/Paddle/pull/59373),[#57206](https://github.com/PaddlePaddle/Paddle/pull/57206),[#58145](https://github.com/PaddlePaddle/Paddle/pull/58145),[#55235](https://github.com/PaddlePaddle/Paddle/pull/55235),[#57255](https://github.com/PaddlePaddle/Paddle/pull/57255),[#56925](https://github.com/PaddlePaddle/Paddle/pull/56925),[#55957](https://github.com/PaddlePaddle/Paddle/pull/55957),[#56163](https://github.com/PaddlePaddle/Paddle/pull/56163),[#56316](https://github.com/PaddlePaddle/Paddle/pull/56316),[#57294](https://github.com/PaddlePaddle/Paddle/pull/57294),[#57449](https://github.com/PaddlePaddle/Paddle/pull/57449),[#59520](https://github.com/PaddlePaddle/Paddle/pull/59520),[#59565](https://github.com/PaddlePaddle/Paddle/pull/59565),[#56265](https://github.com/PaddlePaddle/Paddle/pull/56265),[#56512](https://github.com/PaddlePaddle/Paddle/pull/56512),[#56650](https://github.com/PaddlePaddle/Paddle/pull/56650),[#57183](https://github.com/PaddlePaddle/Paddle/pull/57183),[#57956](https://github.com/PaddlePaddle/Paddle/pull/57956),[#59100](https://github.com/PaddlePaddle/Paddle/pull/59100) - 优化了控制流前向,反向算子的设计和表示，引入 ControlFlow Dialect，并支持 ProgramDesc 下控制流算子到 PIR 的转换和执行。[#58729](https://github.com/PaddlePaddle/Paddle/pull/58729),[#57364](https://github.com/PaddlePaddle/Paddle/pull/57364),[#58625](https://github.com/PaddlePaddle/Paddle/pull/58625),[#57475](https://github.com/PaddlePaddle/Paddle/pull/57475),[#57265](https://github.com/PaddlePaddle/Paddle/pull/57265),[#56799](https://github.com/PaddlePaddle/Paddle/pull/56799),[#59033](https://github.com/PaddlePaddle/Paddle/pull/59033),[#57342](https://github.com/PaddlePaddle/Paddle/pull/57342),[#57801](https://github.com/PaddlePaddle/Paddle/pull/57801),[#57958](https://github.com/PaddlePaddle/Paddle/pull/57958),[#57949](https://github.com/PaddlePaddle/Paddle/pull/57949),[#57937](https://github.com/PaddlePaddle/Paddle/pull/57937),[#59231](https://github.com/PaddlePaddle/Paddle/pull/59231),[#59496](https://github.com/PaddlePaddle/Paddle/pull/59496),[#59321](https://github.com/PaddlePaddle/Paddle/pull/59321),[#58088](https://github.com/PaddlePaddle/Paddle/pull/58088),[#58198](https://github.com/PaddlePaddle/Paddle/pull/58198),[#58024](https://github.com/PaddlePaddle/Paddle/pull/58024),[#58089](https://github.com/PaddlePaddle/Paddle/pull/58089),[#58086](https://github.com/PaddlePaddle/Paddle/pull/58086),[#59175](https://github.com/PaddlePaddle/Paddle/pull/59175),[#59423](https://github.com/PaddlePaddle/Paddle/pull/59423),[#59567](https://github.com/PaddlePaddle/Paddle/pull/59567),[#58098](https://github.com/PaddlePaddle/Paddle/pull/58098),[#58163](https://github.com/PaddlePaddle/Paddle/pull/58163),[#58250](https://github.com/PaddlePaddle/Paddle/pull/58250),[#58277](https://github.com/PaddlePaddle/Paddle/pull/58277),[#58355](https://github.com/PaddlePaddle/Paddle/pull/58355),[#59020](https://github.com/PaddlePaddle/Paddle/pull/59020),[#59200](https://github.com/PaddlePaddle/Paddle/pull/59200),[#59585](https://github.com/PaddlePaddle/Paddle/pull/59585),[#58109](https://github.com/PaddlePaddle/Paddle/pull/58109) - 动转静执行流程升级支持 PIR，优化了动转静子图 Pass 机制，支持用户在@to_static 功能下尝鲜使用 PIR 体系下功能。[#57566](https://github.com/PaddlePaddle/Paddle/pull/57566),[#55620](https://github.com/PaddlePaddle/Paddle/pull/55620),[#56791](https://github.com/PaddlePaddle/Paddle/pull/56791),[#57357](https://github.com/PaddlePaddle/Paddle/pull/57357),[#59152](https://github.com/PaddlePaddle/Paddle/pull/59152),[#59312](https://github.com/PaddlePaddle/Paddle/pull/59312),[#58630](https://github.com/PaddlePaddle/Paddle/pull/58630),[#56035](https://github.com/PaddlePaddle/Paddle/pull/56035),[#59447](https://github.com/PaddlePaddle/Paddle/pull/59447),[#57361](https://github.com/PaddlePaddle/Paddle/pull/57361),[#59261](https://github.com/PaddlePaddle/Paddle/pull/59261),[#59774](https://github.com/PaddlePaddle/Paddle/pull/59774) - 升级了组合算子功能，引入 Backend 概念分层管理动、静态图组合算子模块逻辑，将必要组件和算子拆分规则下沉至 C++，大幅降低了维护成本。[#58153](https://github.com/PaddlePaddle/Paddle/pull/58153),[#56391](https://github.com/PaddlePaddle/Paddle/pull/56391),[#56614](https://github.com/PaddlePaddle/Paddle/pull/56614),[#57030](https://github.com/PaddlePaddle/Paddle/pull/57030),[#57554](https://github.com/PaddlePaddle/Paddle/pull/57554),[#58018](https://github.com/PaddlePaddle/Paddle/pull/58018),[#58130](https://github.com/PaddlePaddle/Paddle/pull/58130),[#58581](https://github.com/PaddlePaddle/Paddle/pull/58581),[#58679](https://github.com/PaddlePaddle/Paddle/pull/58679),[#59054](https://github.com/PaddlePaddle/Paddle/pull/59054),[#55480](https://github.com/PaddlePaddle/Paddle/pull/55480),[#58451](https://github.com/PaddlePaddle/Paddle/pull/58451),[#55647](https://github.com/PaddlePaddle/Paddle/pull/55647),[#56342](https://github.com/PaddlePaddle/Paddle/pull/56342),[#56798](https://github.com/PaddlePaddle/Paddle/pull/56798),[#57561](https://github.com/PaddlePaddle/Paddle/pull/57561),[#58023](https://github.com/PaddlePaddle/Paddle/pull/58023),[#57722](https://github.com/PaddlePaddle/Paddle/pull/57722) ### 性能优化 - 新增 DCE、constant_folding_pass 等 PIR Program 算子和结构优化的 Pass。[#54935](https://github.com/PaddlePaddle/Paddle/pull/54935),[#59430](https://github.com/PaddlePaddle/Paddle/pull/59430),[#58753](https://github.com/PaddlePaddle/Paddle/pull/58753),[#58732](https://github.com/PaddlePaddle/Paddle/pull/58732) 2. 新增 fused_attention,fused_dropout_add,fused_gemm_epilogue_pass,fused_linear_param_grad_add_pass,fused_weight_only_linear_pass,fused_softmax_mask_upper_triangle 等优化算子融合类 Pass，提升训练和推理性能。[#57557](https://github.com/PaddlePaddle/Paddle/pull/57557),[#58272](https://github.com/PaddlePaddle/Paddle/pull/58272),[#58188](https://github.com/PaddlePaddle/Paddle/pull/58188),[#58401](https://github.com/PaddlePaddle/Paddle/pull/58401),[#59366](https://github.com/PaddlePaddle/Paddle/pull/59366),[#57655](https://github.com/PaddlePaddle/Paddle/pull/57655),[#57360](https://github.com/PaddlePaddle/Paddle/pull/57360),[#56672](https://github.com/PaddlePaddle/Paddle/pull/56672),[#58537](https://github.com/PaddlePaddle/Paddle/pull/58537),[#56247](https://github.com/PaddlePaddle/Paddle/pull/56247),[#59391](https://github.com/PaddlePaddle/Paddle/pull/59391),[#58897](https://github.com/PaddlePaddle/Paddle/pull/58897),[#54933](https://github.com/PaddlePaddle/Paddle/pull/54933) ## 动转静能力增强动态图到静态图的转换是深度学习框架中的一项关键技术，它允许开发者在灵活性和训练效率之间找到最佳平衡。本版本飞桨对动转静核心功能进行了全面升级，在飞桨产业级模型库的 700 多个模型中，动转静训练的成功率高达 100%。 ### 新功能 - 采用 Python Eval Frame 和虚拟机模拟执行技术，创新性地实现了自适应的 Graph Break 机制。这一机制特别针对控制流场景，通过引入 CallLayer 机制，充分利用飞桨动静统一的优势，支持 AST（抽象语法树）与字节码模拟的混合模式，有效捕获控制流算子，从而大幅度提高了计算图的静态化能力。在缓存优化层面，融合了公共子表达式消除等高级优化技术，显著提升了 Guard 的执行效率。这些优化措施不仅减少了冗余计算，还提高了整体系统的运行速度。为了增强系统的鲁棒性，设计了一个简洁高效的数据中间层结构。这一结构支持 SideEffects 的正确性恢复，确保了系统在复杂环境下的稳定性和可靠性。此外，广泛兼容 Python 3.8 至 3.11 的主流解释器版本，为用户提供了广泛的适用性。[#57824](https://github.com/PaddlePaddle/Paddle/pull/57824),[#55887](https://github.com/PaddlePaddle/Paddle/pull/55887),[#58155](https://github.com/PaddlePaddle/Paddle/pull/58155),[#56107](https://github.com/PaddlePaddle/Paddle/pull/56107),[#57490](https://github.com/PaddlePaddle/Paddle/pull/57490),[#58829](https://github.com/PaddlePaddle/Paddle/pull/58829),[#57240](https://github.com/PaddlePaddle/Paddle/pull/57240),[#57588](https://github.com/PaddlePaddle/Paddle/pull/57588),[#58117](https://github.com/PaddlePaddle/Paddle/pull/58117),[#59823](https://github.com/PaddlePaddle/Paddle/pull/59823),[#56077](https://github.com/PaddlePaddle/Paddle/pull/56077),[#58956](https://github.com/PaddlePaddle/Paddle/pull/58956),[#57653](https://github.com/PaddlePaddle/Paddle/pull/57653),[#59855](https://github.com/PaddlePaddle/Paddle/pull/59855),[#59017](https://github.com/PaddlePaddle/Paddle/pull/59017),[#58424](https://github.com/PaddlePaddle/Paddle/pull/58424),[#58187](https://github.com/PaddlePaddle/Paddle/pull/58187),[#57793](https://github.com/PaddlePaddle/Paddle/pull/57793),[#59698](https://github.com/PaddlePaddle/Paddle/pull/59698),[#59747](https://github.com/PaddlePaddle/Paddle/pull/59747),[#59710](https://github.com/PaddlePaddle/Paddle/pull/59710),[#59297](https://github.com/PaddlePaddle/Paddle/pull/59297),[#58423](https://github.com/PaddlePaddle/Paddle/pull/58423),[#56262](https://github.com/PaddlePaddle/Paddle/pull/56262),[#58103](https://github.com/PaddlePaddle/Paddle/pull/58103),[#58538](https://github.com/PaddlePaddle/Paddle/pull/58538),[#58771](https://github.com/PaddlePaddle/Paddle/pull/58771),[#59191](https://github.com/PaddlePaddle/Paddle/pull/59191),[#57754](https://github.com/PaddlePaddle/Paddle/pull/57754),[#59439](https://github.com/PaddlePaddle/Paddle/pull/59439),[#59816](https://github.com/PaddlePaddle/Paddle/pull/59816),[#59035](https://github.com/PaddlePaddle/Paddle/pull/59035) - 新增对 PyLayer 功能的动转静语法转写解析，使得 PyLayer 在动态图与静态图之间的转换更加顺畅。现在，用户可以在 PyLayer 下无缝地进行动转静的训练，并轻松导出推理模型。[#56108](https://github.com/PaddlePaddle/Paddle/pull/56108),[#56531](https://github.com/PaddlePaddle/Paddle/pull/56531),[#57066](https://github.com/PaddlePaddle/Paddle/pull/57066),[#57633](https://github.com/PaddlePaddle/Paddle/pull/57633) ### Bug Fix - 修复了动转静在 is_test=True 模式部分场景下出现显存异常的问题。[#58350](https://github.com/PaddlePaddle/Paddle/pull/58350) - 修复了被@to_static 装饰的函数在类似 foo(x,x,y) 场景下 jit.save 模型导出的问题。[#55963](https://github.com/PaddlePaddle/Paddle/pull/55963) - 修复了部分 API 行为动静逻辑不统一问题，提升了动转静整图转换成功率和使用体验。[#56092](https://github.com/PaddlePaddle/Paddle/pull/56092) ### 漏洞修复 - 修复了动转静语法转写模块使用 eval()存在的潜在安全漏洞问题。[#60100](https://github.com/PaddlePaddle/Paddle/pull/60100) ## 动态图分布式能力增强为了满足大型模型的需求，本版本重点提升了飞桨动态图的分布式计算能力。在通信库、图分析、分布式策略和任务启停等方面进行了多方面的改进，为大型模型训练提供了全面的支持。在性能方面，我们通过减少流水并行 GPU 显存占用、采用 TensorFusion 技术、实现通信计算 overlap 以及减少非必要的数据同步拷贝等方式，进一步提升了训练性能。同时，通过环境变量控制 Optimizer 等方式提高了混合并行调试的灵活性。此外，通过修复相关 Bug，进一步提升了系统的稳定性。 ### 新功能 - 通信库新增 TraceHang 功能，当集群训练出现 Hang 的问题时，能够快速的定位到出现问题的节点。[#59217](https://github.com/PaddlePaddle/Paddle/pull/59217) - 为了提升训练效率和降低显存，动态图支持 stride 机制。[#55156](https://github.com/PaddlePaddle/Paddle/pull/55156),[#54762](https://github.com/PaddlePaddle/Paddle/pull/54762),[#55850](https://github.com/PaddlePaddle/Paddle/pull/55850),[#59190](https://github.com/PaddlePaddle/Paddle/pull/59190),[#57005](https://github.com/PaddlePaddle/Paddle/pull/57005),[#57005](https://github.com/PaddlePaddle/Paddle/pull/57005),[#57331](https://github.com/PaddlePaddle/Paddle/pull/57331),[#58033](https://github.com/PaddlePaddle/Paddle/pull/58033),[#58033](https://github.com/PaddlePaddle/Paddle/pull/58033),[#58303](https://github.com/PaddlePaddle/Paddle/pull/58303),[#57835](https://github.com/PaddlePaddle/Paddle/pull/57835),[#57189](https://github.com/PaddlePaddle/Paddle/pull/57189) - 为了方便计算图的分析，增强 paddleviz 功能。[#56837](https://github.com/PaddlePaddle/Paddle/pull/56837),[#57626](https://github.com/PaddlePaddle/Paddle/pull/57626) - 分布式 Sharding 策略(Stage1,2,3)新增 main_grad 功能，以支持更高精度的梯度累加，减少低精度累加带来的精度损失。[#57972](https://github.com/PaddlePaddle/Paddle/pull/57972),[#57934](https://github.com/PaddlePaddle/Paddle/pull/57934),[#57473](https://github.com/PaddlePaddle/Paddle/pull/57473),[#57537](https://github.com/PaddlePaddle/Paddle/pull/57537),[#59611](https://github.com/PaddlePaddle/Paddle/pull/59611),[#57960](https://github.com/PaddlePaddle/Paddle/pull/57960) - Sharding Stage1 策略新增开关变量，可以控制是否对 Optimizer 进行 fusion 计算。[#58790](https://github.com/PaddlePaddle/Paddle/pull/58790) - Recompute 功能新增对 Tuple 输入参数的支持，增强了 Recompute 接口的调用能力。[#56793](https://github.com/PaddlePaddle/Paddle/pull/56793) - 增强 Launch 功能，动态图下无需指定 endpoints 也可以进行分布式训练。 [#54636](https://github.com/PaddlePaddle/Paddle/pull/54636) ### 功能优化 - 实现动静统一的新通信库，通信算子全面适配 PHI 算子体系，减少开发和维护成本，更好地支持动态图和自动并行架构升级。[#54417](https://github.com/PaddlePaddle/Paddle/pull/54417),[#57768](https://github.com/PaddlePaddle/Paddle/pull/57768),[#57897](https://github.com/PaddlePaddle/Paddle/pull/57897),[#55537](https://github.com/PaddlePaddle/Paddle/pull/55537),[#56604](https://github.com/PaddlePaddle/Paddle/pull/56604),[#57519](https://github.com/PaddlePaddle/Paddle/pull/57519),[#56088](https://github.com/PaddlePaddle/Paddle/pull/56088),[#57153](https://github.com/PaddlePaddle/Paddle/pull/57153),[#57161](https://github.com/PaddlePaddle/Paddle/pull/57161),[#57252](https://github.com/PaddlePaddle/Paddle/pull/57252),[#57251](https://github.com/PaddlePaddle/Paddle/pull/57251),[#57208](https://github.com/PaddlePaddle/Paddle/pull/57208),[#57305](https://github.com/PaddlePaddle/Paddle/pull/57305),[#57424](https://github.com/PaddlePaddle/Paddle/pull/57424),[#57548](https://github.com/PaddlePaddle/Paddle/pull/57548),[#57560](https://github.com/PaddlePaddle/Paddle/pull/57560),[#57564](https://github.com/PaddlePaddle/Paddle/pull/57564),[#57233](https://github.com/PaddlePaddle/Paddle/pull/57233),[#55726](https://github.com/PaddlePaddle/Paddle/pull/55726),[#58073](https://github.com/PaddlePaddle/Paddle/pull/58073) - TCPStore 改为单例以便更灵活地支持动态图和自动并行功能。[#55956](https://github.com/PaddlePaddle/Paddle/pull/55956) - 改善了 MP/PP/SP 等分布式策略的可维护性和灵活性，包含增加打印 warning、报错信息，对代码文件进行结构清理，梳理 PP 对输入的限制等。[#54448](https://github.com/PaddlePaddle/Paddle/pull/54448),[#59762](https://github.com/PaddlePaddle/Paddle/pull/59762),[#55462](https://github.com/PaddlePaddle/Paddle/pull/55462),[#54788](https://github.com/PaddlePaddle/Paddle/pull/54788),[#54664](https://github.com/PaddlePaddle/Paddle/pull/54664),[#56456](https://github.com/PaddlePaddle/Paddle/pull/56456),[#55540](https://github.com/PaddlePaddle/Paddle/pull/55540) - PP 策略中增加可以在计算流中进行 P2P 通信的支持，通信模式更加灵活。[#54747](https://github.com/PaddlePaddle/Paddle/pull/54747) - Sharding 策略支持对梯度进行 reduce 操作。[#58842](https://github.com/PaddlePaddle/Paddle/pull/58842),[#57967](https://github.com/PaddlePaddle/Paddle/pull/57967),[#55495](https://github.com/PaddlePaddle/Paddle/pull/55495) ### 性能优化 - 实现 PP 策略的最后一层及时释放 output，以节约显存。[#54505](https://github.com/PaddlePaddle/Paddle/pull/54505) - MP 策略 Tensor fusion 支持传入 params group，增强 Tensor fusion 功能；增加 allreduce 异步通信性能，通过计算和通信的 overlap 提升训练性能。[#57690](https://github.com/PaddlePaddle/Paddle/pull/57690),[#55662](https://github.com/PaddlePaddle/Paddle/pull/55662) - Sharding 策略反向计算和梯度通信进行 overlap 以提升训练性能。Sharding stage1 新增 Tensor fusion 和 fuse grad clip，optimizer 等优化提高计算效率。支持 VPP 与 DP/Sharding Stage1 的 overlap，提升通信计算并行度。优化 Sharding Stage1 在 FP16 下的性能，在 check finite 阶段只对本 sharding rank 负责的梯度进行检查，降低计算开销；增加环境变量，控制是否进行 Optimize，以节约显存支持，实现使用更少的资源进行模型训练调试。[#55598](https://github.com/PaddlePaddle/Paddle/pull/55598),[#55427](https://github.com/PaddlePaddle/Paddle/pull/55427),[#56063](https://github.com/PaddlePaddle/Paddle/pull/56063),[#55766](https://github.com/PaddlePaddle/Paddle/pull/55766),[#59848](https://github.com/PaddlePaddle/Paddle/pull/59848) - 混合并行策略将 PP/VPP 下的 Tensor fusion 提到运行前，解决运行时 fuse 对显存额外开销的问题。通过减少非必需的同步 memcpy，以提升模型训练性能。[#54403](https://github.com/PaddlePaddle/Paddle/pull/54403),[#57215](https://github.com/PaddlePaddle/Paddle/pull/57215) ### Bug Fix - 修复了 PP、Launch 功能、MP 策略以及 fuse_rope 等 13 个 bug，增强了分布式策略的稳定性；机制层面，修复 inplace，tensor 引用的错误，提升稳定性。[#55116](https://github.com/PaddlePaddle/Paddle/pull/55116),[#55782](https://github.com/PaddlePaddle/Paddle/pull/55782),[#59609](https://github.com/PaddlePaddle/Paddle/pull/59609),[#57394](https://github.com/PaddlePaddle/Paddle/pull/57394),[#55864](https://github.com/PaddlePaddle/Paddle/pull/55864),[#58482](https://github.com/PaddlePaddle/Paddle/pull/58482),[#54571](https://github.com/PaddlePaddle/Paddle/pull/54571),[#55896](https://github.com/PaddlePaddle/Paddle/pull/55896),[#54648](https://github.com/PaddlePaddle/Paddle/pull/54648),[#58307](https://github.com/PaddlePaddle/Paddle/pull/58307),[#55679](https://github.com/PaddlePaddle/Paddle/pull/55679),[#58133](https://github.com/PaddlePaddle/Paddle/pull/58133),[#58408](https://github.com/PaddlePaddle/Paddle/pull/58408),[#59707](https://github.com/PaddlePaddle/Paddle/pull/59707),[#55342](https://github.com/PaddlePaddle/Paddle/pull/55342),[#54703](https://github.com/PaddlePaddle/Paddle/pull/54703),[#54869](https://github.com/PaddlePaddle/Paddle/pull/54869),[#55568](https://github.com/PaddlePaddle/Paddle/pull/55568),[#55233](https://github.com/PaddlePaddle/Paddle/pull/55233),[#56418](https://github.com/PaddlePaddle/Paddle/pull/56418),[#56428](https://github.com/PaddlePaddle/Paddle/pull/56428),[#56892](https://github.com/PaddlePaddle/Paddle/pull/56892),[#57192](https://github.com/PaddlePaddle/Paddle/pull/57192),[#59161](https://github.com/PaddlePaddle/Paddle/pull/59161),[#59340](https://github.com/PaddlePaddle/Paddle/pull/59340),[#57006](https://github.com/PaddlePaddle/Paddle/pull/57006),[#57353](https://github.com/PaddlePaddle/Paddle/pull/57353),[#57352](https://github.com/PaddlePaddle/Paddle/pull/57352),[#59088](https://github.com/PaddlePaddle/Paddle/pull/59088) - 修复了 PP 策略无法及时释放单层 output 的 bug，以及初始化过程中可能会 Hang 的 bug。 [#54624](https://github.com/PaddlePaddle/Paddle/pull/54624),[#58844](https://github.com/PaddlePaddle/Paddle/pull/58844),[#54673](https://github.com/PaddlePaddle/Paddle/pull/54673),[#58376](https://github.com/PaddlePaddle/Paddle/pull/58376) - 修复了 MP 策略下，当输入数据类型不统一时计算出错的 bug，修复了 MP 策略下参数同步的 bug 和没有正确使用用户输入 config 的 bug。[#58858](https://github.com/PaddlePaddle/Paddle/pull/58858),[#57918](https://github.com/PaddlePaddle/Paddle/pull/57918),[#58037](https://github.com/PaddlePaddle/Paddle/pull/58037) - 统一 dygraph 和 dynamic 模式的判断方法。[#54633](https://github.com/PaddlePaddle/Paddle/pull/54633) - 修复了 fuse_rope 中 sin 和 cos 的 Shape 不对的 bug。[#56132](https://github.com/PaddlePaddle/Paddle/pull/56132) - 修复了 Luanch 功能分布式场景下 endpoints 太长导致不能启动任务的 bug，同时修复了 endpoints 可能乱序的 bug。 [#55011](https://github.com/PaddlePaddle/Paddle/pull/55011),[#55478](https://github.com/PaddlePaddle/Paddle/pull/55478) - 修复了 MEA 功能可能导致 segmentation fault error 的 bug。[#55408](https://github.com/PaddlePaddle/Paddle/pull/55408) ## 自动并行本版本对动静统一自动并行（Auto Parallel）编程范式进行了全面的优化，简化了开发者的编程复杂度。开发者无需深入了解手动并行编程范式下的复杂概念和 API 接口，如行切分、列切分等。仅需通过少量的张量切分标注即可完成混合并行模型的构建。框架能够自动推导出所有张量和算子的分布式切分状态，并添加合适的通信算子。同时支持一键动转静进行分布式训练，使开发者能够高效轻松地实现任意混合并行策略，大幅降低了混合并行训练代码的开发成本。 ### 完善了自动并行核心功能 - 实现 process_mesh、placement、shard_tensor、reshard、dtensor_from_fn、unshard_dtensor、shard_layer、to_static 等自动并行核心接口 [#55494](https://github.com/PaddlePaddle/Paddle/pull/55494),[#59059](https://github.com/PaddlePaddle/Paddle/pull/59059),[#56561](https://github.com/PaddlePaddle/Paddle/pull/56561),[#54425](https://github.com/PaddlePaddle/Paddle/pull/54425),[#59557](https://github.com/PaddlePaddle/Paddle/pull/59557),[#59682](https://github.com/PaddlePaddle/Paddle/pull/59682),[#56565](https://github.com/PaddlePaddle/Paddle/pull/56565),[#59862](https://github.com/PaddlePaddle/Paddle/pull/59862),[#59856](https://github.com/PaddlePaddle/Paddle/pull/59856),[#59342](https://github.com/PaddlePaddle/Paddle/pull/59342),[#59575](https://github.com/PaddlePaddle/Paddle/pull/59575),[#57604](https://github.com/PaddlePaddle/Paddle/pull/57604),[#57293](https://github.com/PaddlePaddle/Paddle/pull/57293),[#57278](https://github.com/PaddlePaddle/Paddle/pull/57278) - 实现基于 Enisum 表达式的切分推导规则，并完成 20+类算子切分推导规则，覆盖 LLaMA、GPT 等主流生成式大语言模型。[#55196](https://github.com/PaddlePaddle/Paddle/pull/55196),[#53863](https://github.com/PaddlePaddle/Paddle/pull/53863),[#56257](https://github.com/PaddlePaddle/Paddle/pull/56257),[#55394](https://github.com/PaddlePaddle/Paddle/pull/55394),[#54810](https://github.com/PaddlePaddle/Paddle/pull/54810),[#55508](https://github.com/PaddlePaddle/Paddle/pull/55508),[#56257](https://github.com/PaddlePaddle/Paddle/pull/56257),[#57813](https://github.com/PaddlePaddle/Paddle/pull/57813),[#58149](https://github.com/PaddlePaddle/Paddle/pull/58149),[#58506](https://github.com/PaddlePaddle/Paddle/pull/58506),[#58563](https://github.com/PaddlePaddle/Paddle/pull/58563),[#58360](https://github.com/PaddlePaddle/Paddle/pull/58360),[#58920](https://github.com/PaddlePaddle/Paddle/pull/58920),[#59050](https://github.com/PaddlePaddle/Paddle/pull/59050),[#58760](https://github.com/PaddlePaddle/Paddle/pull/58760),[#59083](https://github.com/PaddlePaddle/Paddle/pull/59083),[#59236](https://github.com/PaddlePaddle/Paddle/pull/59236),[#59350](https://github.com/PaddlePaddle/Paddle/pull/59350),[#59411](https://github.com/PaddlePaddle/Paddle/pull/59411),[#59260](https://github.com/PaddlePaddle/Paddle/pull/59260),[#54373](https://github.com/PaddlePaddle/Paddle/pull/54373),[#54991](https://github.com/PaddlePaddle/Paddle/pull/54991),[#55397](https://github.com/PaddlePaddle/Paddle/pull/55397),[#55350](https://github.com/PaddlePaddle/Paddle/pull/55350),[#55177](https://github.com/PaddlePaddle/Paddle/pull/55177),[#56443](https://github.com/PaddlePaddle/Paddle/pull/56443),[#58097](https://github.com/PaddlePaddle/Paddle/pull/58097),[#56509](https://github.com/PaddlePaddle/Paddle/pull/56509),[#56502](https://github.com/PaddlePaddle/Paddle/pull/56502),[#56504](https://github.com/PaddlePaddle/Paddle/pull/56504),[#56506](https://github.com/PaddlePaddle/Paddle/pull/56506),[#56507](https://github.com/PaddlePaddle/Paddle/pull/56507),[#56505](https://github.com/PaddlePaddle/Paddle/pull/56505),[#57176](https://github.com/PaddlePaddle/Paddle/pull/57176),[#57374](https://github.com/PaddlePaddle/Paddle/pull/57374),[#57573](https://github.com/PaddlePaddle/Paddle/pull/57573),[#57545](https://github.com/PaddlePaddle/Paddle/pull/57545),[#57875](https://github.com/PaddlePaddle/Paddle/pull/57875),[#57866](https://github.com/PaddlePaddle/Paddle/pull/57866),[#58854](https://github.com/PaddlePaddle/Paddle/pull/58854),[#59109](https://github.com/PaddlePaddle/Paddle/pull/59109),[#59185](https://github.com/PaddlePaddle/Paddle/pull/59185),[#58913](https://github.com/PaddlePaddle/Paddle/pull/58913),[#59547](https://github.com/PaddlePaddle/Paddle/pull/59547),[#58296](https://github.com/PaddlePaddle/Paddle/pull/58296),[#59545](https://github.com/PaddlePaddle/Paddle/pull/59545),[#59039](https://github.com/PaddlePaddle/Paddle/pull/59039),[#59002](https://github.com/PaddlePaddle/Paddle/pull/59002),[#58087](https://github.com/PaddlePaddle/Paddle/pull/58087),[#56367](https://github.com/PaddlePaddle/Paddle/pull/56367),[#57877](https://github.com/PaddlePaddle/Paddle/pull/57877),[#56839](https://github.com/PaddlePaddle/Paddle/pull/56839),[#59003](https://github.com/PaddlePaddle/Paddle/pull/59003),[#57269](https://github.com/PaddlePaddle/Paddle/pull/57269),[#55130](https://github.com/PaddlePaddle/Paddle/pull/55130),[#58474](https://github.com/PaddlePaddle/Paddle/pull/58474),[#57197](https://github.com/PaddlePaddle/Paddle/pull/57197),[#57467](https://github.com/PaddlePaddle/Paddle/pull/57467),[#57259](https://github.com/PaddlePaddle/Paddle/pull/57259),[#57280](https://github.com/PaddlePaddle/Paddle/pull/57280),[#56508](https://github.com/PaddlePaddle/Paddle/pull/56508) - 实现动静统一的分布式 checkpoint 存储和加载，支持任意按切分状态存储和加载时重切分。[#59659](https://github.com/PaddlePaddle/Paddle/pull/59659),[#59843](https://github.com/PaddlePaddle/Paddle/pull/59843),[#60033](https://github.com/PaddlePaddle/Paddle/pull/60033),[#60034](https://github.com/PaddlePaddle/Paddle/pull/60034) ### 增强动态图半自动并行能力 - 基础数据结构补充：C++端新增 DistTensor、Placements 等分布式特有的基础数据结构，并暴露到 Python 端，支持对相关属性和值的调试打印。[#58930](https://github.com/PaddlePaddle/Paddle/pull/58930),[#59068](https://github.com/PaddlePaddle/Paddle/pull/59068),[#55436](https://github.com/PaddlePaddle/Paddle/pull/55436),[#56449](https://github.com/PaddlePaddle/Paddle/pull/56449),[#59683](https://github.com/PaddlePaddle/Paddle/pull/59683),[#55593](https://github.com/PaddlePaddle/Paddle/pull/55593),[#58032](https://github.com/PaddlePaddle/Paddle/pull/58032),[#56368](https://github.com/PaddlePaddle/Paddle/pull/56368),[#59086](https://github.com/PaddlePaddle/Paddle/pull/59086) - 在前、反向算子执行流程中添加 SPMD 推导与 Reshard 的生成逻辑，适配 vector、optional 等多类型输入输出以及 cpu fallback、多 kernel 选择等特殊机制。[#56602](https://github.com/PaddlePaddle/Paddle/pull/56602),[#57321](https://github.com/PaddlePaddle/Paddle/pull/57321),[#57092](https://github.com/PaddlePaddle/Paddle/pull/57092),[#56831](https://github.com/PaddlePaddle/Paddle/pull/56831),[#57119](https://github.com/PaddlePaddle/Paddle/pull/57119),[#58819](https://github.com/PaddlePaddle/Paddle/pull/58819),[#58254](https://github.com/PaddlePaddle/Paddle/pull/58254),[#55698](https://github.com/PaddlePaddle/Paddle/pull/55698),[#59241](https://github.com/PaddlePaddle/Paddle/pull/59241),[#59328](https://github.com/PaddlePaddle/Paddle/pull/59328),[#58644](https://github.com/PaddlePaddle/Paddle/pull/58644),[#56202](https://github.com/PaddlePaddle/Paddle/pull/56202),[#59159](https://github.com/PaddlePaddle/Paddle/pull/59159),[#58573](https://github.com/PaddlePaddle/Paddle/pull/58573),[#59246](https://github.com/PaddlePaddle/Paddle/pull/59246),[#59133](https://github.com/PaddlePaddle/Paddle/pull/59133),[#59186](https://github.com/PaddlePaddle/Paddle/pull/59186),[#57505](https://github.com/PaddlePaddle/Paddle/pull/57505),[#57241](https://github.com/PaddlePaddle/Paddle/pull/57241),[#58928](https://github.com/PaddlePaddle/Paddle/pull/58928) - 对 custom 算子、手写算子等特殊类型的算子，适配自动并行的执行逻辑。支持 DistTensor 和 DenseTensor 作为混合输入时的自动转换。[#57774](https://github.com/PaddlePaddle/Paddle/pull/57774),[#59108](https://github.com/PaddlePaddle/Paddle/pull/59108),[#58436](https://github.com/PaddlePaddle/Paddle/pull/58436),[#59523](https://github.com/PaddlePaddle/Paddle/pull/59523),[#59136](https://github.com/PaddlePaddle/Paddle/pull/59136),[#59352](https://github.com/PaddlePaddle/Paddle/pull/59352),[#59062](https://github.com/PaddlePaddle/Paddle/pull/59062),[#58434](https://github.com/PaddlePaddle/Paddle/pull/58434),[#59148](https://github.com/PaddlePaddle/Paddle/pull/59148),[#58553](https://github.com/PaddlePaddle/Paddle/pull/58553),[#58716](https://github.com/PaddlePaddle/Paddle/pull/58716),[#58369](https://github.com/PaddlePaddle/Paddle/pull/58369),[#59061](https://github.com/PaddlePaddle/Paddle/pull/59061),[#58841](https://github.com/PaddlePaddle/Paddle/pull/58841),[#59139](https://github.com/PaddlePaddle/Paddle/pull/59139),[#59141](https://github.com/PaddlePaddle/Paddle/pull/59141),[#58837](https://github.com/PaddlePaddle/Paddle/pull/58837),[#59137](https://github.com/PaddlePaddle/Paddle/pull/59137),[#59143](https://github.com/PaddlePaddle/Paddle/pull/59143) - 动态图执行体系完善：适配 Autograd 执行过程，支持动态图的反向梯度聚合、AMP、Hook、PyLayer、View、自定义算子等周围机制。[#58437](https://github.com/PaddlePaddle/Paddle/pull/58437),[#58769](https://github.com/PaddlePaddle/Paddle/pull/58769),[#58796](https://github.com/PaddlePaddle/Paddle/pull/58796),[#58339](https://github.com/PaddlePaddle/Paddle/pull/58339),[#58409](https://github.com/PaddlePaddle/Paddle/pull/58409),[#58772](https://github.com/PaddlePaddle/Paddle/pull/58772),[#58380](https://github.com/PaddlePaddle/Paddle/pull/58380),[#58447](https://github.com/PaddlePaddle/Paddle/pull/58447),[#58706](https://github.com/PaddlePaddle/Paddle/pull/58706),[#58656](https://github.com/PaddlePaddle/Paddle/pull/58656),[#58172](https://github.com/PaddlePaddle/Paddle/pull/58172),[#59401](https://github.com/PaddlePaddle/Paddle/pull/59401),[#58727](https://github.com/PaddlePaddle/Paddle/pull/58727),[#58238](https://github.com/PaddlePaddle/Paddle/pull/58238),[#59243](https://github.com/PaddlePaddle/Paddle/pull/59243),[#58469](https://github.com/PaddlePaddle/Paddle/pull/58469),[#58442](https://github.com/PaddlePaddle/Paddle/pull/58442),[#58487](https://github.com/PaddlePaddle/Paddle/pull/58487),[#58476](https://github.com/PaddlePaddle/Paddle/pull/58476),[#59706](https://github.com/PaddlePaddle/Paddle/pull/59706) - 新增对 PP、SP 等分布式策略的支持。[#58126](https://github.com/PaddlePaddle/Paddle/pull/58126),[#59766](https://github.com/PaddlePaddle/Paddle/pull/59766),[#59060](https://github.com/PaddlePaddle/Paddle/pull/59060),[#59841](https://github.com/PaddlePaddle/Paddle/pull/59841),[#58609](https://github.com/PaddlePaddle/Paddle/pull/58609),[#59688](https://github.com/PaddlePaddle/Paddle/pull/59688),[#58449](https://github.com/PaddlePaddle/Paddle/pull/58449)、[#59598](https://github.com/PaddlePaddle/Paddle/pull/59598) - 新增多种 Reshard 策略，支持张量在不同分布式状态间的转换。[#58592](https://github.com/PaddlePaddle/Paddle/pull/58592),[#59138](https://github.com/PaddlePaddle/Paddle/pull/59138),[#59367](https://github.com/PaddlePaddle/Paddle/pull/59367),[#59621](https://github.com/PaddlePaddle/Paddle/pull/59621),[#59758](https://github.com/PaddlePaddle/Paddle/pull/59758),[#59777](https://github.com/PaddlePaddle/Paddle/pull/59777),[#56975](https://github.com/PaddlePaddle/Paddle/pull/56975),[#58550](https://github.com/PaddlePaddle/Paddle/pull/58550),[#58703](https://github.com/PaddlePaddle/Paddle/pull/58703),[#57210](https://github.com/PaddlePaddle/Paddle/pull/57210),[#58734](https://github.com/PaddlePaddle/Paddle/pull/58734),[#56833](https://github.com/PaddlePaddle/Paddle/pull/56833),[#59292](https://github.com/PaddlePaddle/Paddle/pull/59292),[#57432](https://github.com/PaddlePaddle/Paddle/pull/57432),[#57568](https://github.com/PaddlePaddle/Paddle/pull/57568),[#56553](https://github.com/PaddlePaddle/Paddle/pull/56553),[#58284](https://github.com/PaddlePaddle/Paddle/pull/58284),[#56039](https://github.com/PaddlePaddle/Paddle/pull/56039),[#55552](https://github.com/PaddlePaddle/Paddle/pull/55552),[#56149](https://github.com/PaddlePaddle/Paddle/pull/56149) ### 静态图半自动并行能力增强 - 新增 Sequence Parallel 并行策略；流水线并行新增: FThenB、Interleaved 1F1B、Eager 1F1B、VPP 等调度模式，支持流水线调度的可视化，并支持上述策略与原有并行策略的混合并行；升级梯度同步机制，支持数据在任意 broadcast 维度后需要的梯度同步。[#57605](https://github.com/PaddlePaddle/Paddle/pull/57605),[#54727](https://github.com/PaddlePaddle/Paddle/pull/54727),[#54409](https://github.com/PaddlePaddle/Paddle/pull/54409),[#54787](https://github.com/PaddlePaddle/Paddle/pull/54787),[#58313](https://github.com/PaddlePaddle/Paddle/pull/58313),[#59179](https://github.com/PaddlePaddle/Paddle/pull/59179),[#59416](https://github.com/PaddlePaddle/Paddle/pull/59416),[#59719](https://github.com/PaddlePaddle/Paddle/pull/59719),[#59822](https://github.com/PaddlePaddle/Paddle/pull/59822),[#59057](https://github.com/PaddlePaddle/Paddle/pull/59057),[#59522](https://github.com/PaddlePaddle/Paddle/pull/59522),[#57061](https://github.com/PaddlePaddle/Paddle/pull/57061) - 执行体系与 PIR 进一步适配，打通 PIR 的优化 Pass，分布式场景下支持了 fuse_linear fuse 优化，实现性能提升。[#58459](https://github.com/PaddlePaddle/Paddle/pull/58459),[#58528](https://github.com/PaddlePaddle/Paddle/pull/58528),[#55555](https://github.com/PaddlePaddle/Paddle/pull/55555),[#59757](https://github.com/PaddlePaddle/Paddle/pull/59757),[#59102](https://github.com/PaddlePaddle/Paddle/pull/59102),[#57917](https://github.com/PaddlePaddle/Paddle/pull/57917) - 底层架构升级: 执行器升级支持图依赖信息复用和静态化 kernel 选择；整图切分补全机制升级，切换新切分推导规则并支持更多长尾 cases 的正确切分补全；优化了静态图分布式下对控制流的支持，适配更多场景；优化了整图编译速度、日志信息格式等提升用户体验。 [#55389](https://github.com/PaddlePaddle/Paddle/pull/55389),[#55650](https://github.com/PaddlePaddle/Paddle/pull/55650),[#54938](https://github.com/PaddlePaddle/Paddle/pull/54938),[#57447](https://github.com/PaddlePaddle/Paddle/pull/57447),[#57751](https://github.com/PaddlePaddle/Paddle/pull/57751),[#57742](https://github.com/PaddlePaddle/Paddle/pull/57742),[#59524](https://github.com/PaddlePaddle/Paddle/pull/59524),[#59526](https://github.com/PaddlePaddle/Paddle/pull/59526),[#58669](https://github.com/PaddlePaddle/Paddle/pull/58669),[#57616](https://github.com/PaddlePaddle/Paddle/pull/57616),[#56511](https://github.com/PaddlePaddle/Paddle/pull/56511),[#55727](https://github.com/PaddlePaddle/Paddle/pull/55727),[#58906](https://github.com/PaddlePaddle/Paddle/pull/58906),[#56016](https://github.com/PaddlePaddle/Paddle/pull/56016),[#54897](https://github.com/PaddlePaddle/Paddle/pull/54897) - 优化静态图显存管理，新增精细化重计算策略；优化混合精度适配，支持用户手动指定 cast 范围等场景；支持 Cross Entropy 的并行计算；支持 scaled_dot_product_attention、fuse_rope 等融合算子；执行调度优化，支持张量并行、流水线并行中通信计算间更好地 Overlap。[#58421](https://github.com/PaddlePaddle/Paddle/pull/58421),[#58533](https://github.com/PaddlePaddle/Paddle/pull/58533),[#59498](https://github.com/PaddlePaddle/Paddle/pull/59498),[#59498](https://github.com/PaddlePaddle/Paddle/pull/59498),[#59187](https://github.com/PaddlePaddle/Paddle/pull/59187),[#59188](https://github.com/PaddlePaddle/Paddle/pull/59188),[#58172](https://github.com/PaddlePaddle/Paddle/pull/58172),[#58628](https://github.com/PaddlePaddle/Paddle/pull/58628),[#56185](https://github.com/PaddlePaddle/Paddle/pull/56185),[#56696](https://github.com/PaddlePaddle/Paddle/pull/56696),[#59497](https://github.com/PaddlePaddle/Paddle/pull/59497),[#58304](https://github.com/PaddlePaddle/Paddle/pull/58304),[#58977](https://github.com/PaddlePaddle/Paddle/pull/58977) ### AutoTuner 本版本实现基于 Profiling 的并行策略自动搜索和调优工具 AutoTuner，能够在给定模型和硬件资源的条件下，自动将并行策略和优化策略进行组合，并选取有效的组合配置运行实验，从而搜索出大模型训练和推理的最佳配置。此外，AutoTuner 实现了多种剪枝优化策略，包括显存建模等，能够大幅度减少搜索空间和搜索时间。[#54460](https://github.com/PaddlePaddle/Paddle/pull/54460),[#54668](https://github.com/PaddlePaddle/Paddle/pull/54668),[#59794](https://github.com/PaddlePaddle/Paddle/pull/59794),[#59727](https://github.com/PaddlePaddle/Paddle/pull/59727),[#59782](https://github.com/PaddlePaddle/Paddle/pull/59782),[#54834](https://github.com/PaddlePaddle/Paddle/pull/54834),[#58127](https://github.com/PaddlePaddle/Paddle/pull/58127),[#56968](https://github.com/PaddlePaddle/Paddle/pull/56968),[#55466](https://github.com/PaddlePaddle/Paddle/pull/55466),[#56939](https://github.com/PaddlePaddle/Paddle/pull/56939),[#58183](https://github.com/PaddlePaddle/Paddle/pull/58183),[#58314](https://github.com/PaddlePaddle/Paddle/pull/58314),[#55499](https://github.com/PaddlePaddle/Paddle/pull/55499),[#59748](https://github.com/PaddlePaddle/Paddle/pull/59748) ## 算子库 ### 不兼容升级为了提升飞桨框架的可维护性，删除框架中部分废弃的算子(如 diag_v1, isfinite_v1, pad2d_v1 等)，通过飞桨 1.x 版本训练所保存的使用到这些算子的模型将无法在飞桨新版本上进行推理。[#57895](https://github.com/PaddlePaddle/Paddle/pull/57895),[#57892](https://github.com/PaddlePaddle/Paddle/pull/57892),[#57898](https://github.com/PaddlePaddle/Paddle/pull/57898),[#57730](https://github.com/PaddlePaddle/Paddle/pull/57730),[#57732](https://github.com/PaddlePaddle/Paddle/pull/57732),[#57810](https://github.com/PaddlePaddle/Paddle/pull/57810),[#57884](https://github.com/PaddlePaddle/Paddle/pull/57884),[#57794](https://github.com/PaddlePaddle/Paddle/pull/57794),[#57926](https://github.com/PaddlePaddle/Paddle/pull/57926),[#57925](https://github.com/PaddlePaddle/Paddle/pull/57925),[#57807](https://github.com/PaddlePaddle/Paddle/pull/57807),[#57808](https://github.com/PaddlePaddle/Paddle/pull/57808) ### 算子库功能增强 - 飞桨 PHI 算子库复数计算功能进一步增强，累计新增支持复数计算 Kernel 40+。[#55380](https://github.com/PaddlePaddle/Paddle/pull/55380), [#56349](https://github.com/PaddlePaddle/Paddle/pull/56349), [#56412](https://github.com/PaddlePaddle/Paddle/pull/56412), [#56323](https://github.com/PaddlePaddle/Paddle/pull/56323), [#56723](https://github.com/PaddlePaddle/Paddle/pull/56723), [#56457](https://github.com/PaddlePaddle/Paddle/pull/56457), [#56903](https://github.com/PaddlePaddle/Paddle/pull/56903)[#56914](https://github.com/PaddlePaddle/Paddle/pull/56914), [#57116](https://github.com/PaddlePaddle/Paddle/pull/57116), [#56048](https://github.com/PaddlePaddle/Paddle/pull/56048), [#57244](https://github.com/PaddlePaddle/Paddle/pull/57244), [#57639](https://github.com/PaddlePaddle/Paddle/pull/57639), [#57638](https://github.com/PaddlePaddle/Paddle/pull/57638), [#57540](https://github.com/PaddlePaddle/Paddle/pull/57540), [#58545](https://github.com/PaddlePaddle/Paddle/pull/58545), [#58336](https://github.com/PaddlePaddle/Paddle/pull/58336), [#58532](https://github.com/PaddlePaddle/Paddle/pull/58532), [#58839](https://github.com/PaddlePaddle/Paddle/pull/58839), [#59079](https://github.com/PaddlePaddle/Paddle/pull/59079), [#59277](https://github.com/PaddlePaddle/Paddle/pull/59277), [#59122](https://github.com/PaddlePaddle/Paddle/pull/59122), [#57058](https://github.com/PaddlePaddle/Paddle/pull/57058) - 优化和新增部分算子的 XPU Kernel，并增强了 XPU Kernel 对 bfloat16 等数据类型的运算支持。[#54478](https://github.com/PaddlePaddle/Paddle/pull/54478), [#57740](https://github.com/PaddlePaddle/Paddle/pull/57740), [#58346](https://github.com/PaddlePaddle/Paddle/pull/58346), [#58456](https://github.com/PaddlePaddle/Paddle/pull/58456), [#58662](https://github.com/PaddlePaddle/Paddle/pull/58662), [#59066](https://github.com/PaddlePaddle/Paddle/pull/59066), [#59263](https://github.com/PaddlePaddle/Paddle/pull/59263)), [#59375](https://github.com/PaddlePaddle/Paddle/pull/59375), [#59505](https://github.com/PaddlePaddle/Paddle/pull/59505), [#59653](https://github.com/PaddlePaddle/Paddle/pull/59653), [#55001](https://github.com/PaddlePaddle/Paddle/pull/55001), [#57272](https://github.com/PaddlePaddle/Paddle/pull/57272), [#56169](https://github.com/PaddlePaddle/Paddle/pull/56169), [#59454](https://github.com/PaddlePaddle/Paddle/pull/59454), [#59480](https://github.com/PaddlePaddle/Paddle/pull/59480), [#55914](https://github.com/PaddlePaddle/Paddle/pull/55914), [#54758](https://github.com/PaddlePaddle/Paddle/pull/54758), [#54827](https://github.com/PaddlePaddle/Paddle/pull/54827), [#58364](https://github.com/PaddlePaddle/Paddle/pull/58364), [#58419](https://github.com/PaddlePaddle/Paddle/pull/58419), [#58982](https://github.com/PaddlePaddle/Paddle/pull/58982), [#57216](https://github.com/PaddlePaddle/Paddle/pull/57216), [#59166](https://github.com/PaddlePaddle/Paddle/pull/59166), [#55033](https://github.com/PaddlePaddle/Paddle/pull/55033), [#55375](https://github.com/PaddlePaddle/Paddle/pull/55375), [#58805](https://github.com/PaddlePaddle/Paddle/pull/58805), [#59389](https://github.com/PaddlePaddle/Paddle/pull/59389), [#57077](https://github.com/PaddlePaddle/Paddle/pull/57077), [#55166](https://github.com/PaddlePaddle/Paddle/pull/55166), [#56773](https://github.com/PaddlePaddle/Paddle/pull/56773) - 新增了用于优化大模型训练和推理性能的常见算子。[#55758](https://github.com/PaddlePaddle/Paddle/pull/55758), [#54998](https://github.com/PaddlePaddle/Paddle/pull/54998), [#55400](https://github.com/PaddlePaddle/Paddle/pull/55400), [#54630](https://github.com/PaddlePaddle/Paddle/pull/54630), [#55969](https://github.com/PaddlePaddle/Paddle/pull/55969), [#55026](https://github.com/PaddlePaddle/Paddle/pull/55026), [#58986](https://github.com/PaddlePaddle/Paddle/pull/58986) - 完善算子库 Tensor Strided 机制。[#59422](https://github.com/PaddlePaddle/Paddle/pull/59422), [#59325](https://github.com/PaddlePaddle/Paddle/pull/59325), [#56863](https://github.com/PaddlePaddle/Paddle/pull/56863), [#56882](https://github.com/PaddlePaddle/Paddle/pull/56882), [#56947](https://github.com/PaddlePaddle/Paddle/pull/56947) - 对算子 Kernel 中的函数实现以及模板调用接口进行了编译优化，降低算子库编包体积。[#57083](https://github.com/PaddlePaddle/Paddle/pull/57083), [#57299](https://github.com/PaddlePaddle/Paddle/pull/57299), [#57261](https://github.com/PaddlePaddle/Paddle/pull/57261), [#57290](https://github.com/PaddlePaddle/Paddle/pull/57290), [#57118](https://github.com/PaddlePaddle/Paddle/pull/57118), [#57551](https://github.com/PaddlePaddle/Paddle/pull/57551), [#57509](https://github.com/PaddlePaddle/Paddle/pull/57509), [#57558](https://github.com/PaddlePaddle/Paddle/pull/57558), [#57064](https://github.com/PaddlePaddle/Paddle/pull/57064), [#57365](https://github.com/PaddlePaddle/Paddle/pull/57365), [#57327](https://github.com/PaddlePaddle/Paddle/pull/57327), [#57603](https://github.com/PaddlePaddle/Paddle/pull/57603), [#57671](https://github.com/PaddlePaddle/Paddle/pull/57671), [#57672](https://github.com/PaddlePaddle/Paddle/pull/57672), [#57631](https://github.com/PaddlePaddle/Paddle/pull/57631), [#57082](https://github.com/PaddlePaddle/Paddle/pull/57082), [#57721](https://github.com/PaddlePaddle/Paddle/pull/57721), [#57823](https://github.com/PaddlePaddle/Paddle/pull/57823), [#57821](https://github.com/PaddlePaddle/Paddle/pull/57821), [#57815](https://github.com/PaddlePaddle/Paddle/pull/57815), [#57822](https://github.com/PaddlePaddle/Paddle/pull/57822), [#57541](https://github.com/PaddlePaddle/Paddle/pull/57541), [#57817](https://github.com/PaddlePaddle/Paddle/pull/57817), [#57838](https://github.com/PaddlePaddle/Paddle/pull/57838) ### Bug 修复 - 修复了飞桨框架适配 CUDA 12 的一些问题。[#54640](https://github.com/PaddlePaddle/Paddle/pull/54640), [#57820](https://github.com/PaddlePaddle/Paddle/pull/57820), [#58958](https://github.com/PaddlePaddle/Paddle/pull/58958), [#58179](https://github.com/PaddlePaddle/Paddle/pull/58179), [#55594](https://github.com/PaddlePaddle/Paddle/pull/55594) ## CUDA ### 新功能 - 新增调试类 API paddle.amp.debugging.check_check_numerics，计算并返回这个 Tensor 数值中异常值(NaN、Inf)和零元素的数量。[#54301](https://github.com/PaddlePaddle/Paddle/pull/54301) - 新增 fused_rope 融合算子，加速 LLaMA 类大模型训练。[#54351](https://github.com/PaddlePaddle/Paddle/pull/54351) - 更新 CUDNN Frontend API 版本到 v0.9.1，并新增加速 ResNet 网络的 fused_scale_bias_add_relu 融合算子。注意该功能处于实验期，默认不开启。[#58367](https://github.com/PaddlePaddle/Paddle/pull/58367), [#54949](https://github.com/PaddlePaddle/Paddle/pull/54949), [#58504](https://github.com/PaddlePaddle/Paddle/pull/58504) - 基于 Flash-Attention v2，添加 Tensor 类似 Mask 功能支持，反向算子支持确定性计算，便于调试。[#57276](https://github.com/PaddlePaddle/Paddle/pull/57276), [#56363](https://github.com/PaddlePaddle/Paddle/pull/56363) - 修改稀疏 conv3d 后端实现以支持 2d 形状，避免前端 reshape 的开销。[#54707](https://github.com/PaddlePaddle/Paddle/pull/54707) - 新增 matmul_int8 算子。([#55228](https://github.com/PaddlePaddle/Paddle/pull/55228)) ### 功能优化 - 优化 CUDA Graph 对随机数算子的支持。[#58310](https://github.com/PaddlePaddle/Paddle/pull/58310) - 自动混合精度训练默认功能加强，包括： - 优化自动混合精度训练接口的使用体验。[#58152](https://github.com/PaddlePaddle/Paddle/pull/58152),[#55364](https://github.com/PaddlePaddle/Paddle/pull/55364),[#57903](https://github.com/PaddlePaddle/Paddle/pull/57903) - 将 fused_attention、fused_feedforward、fused_gemm_epilogue 等矩阵计算类算子加入框架默认的白名单，并统一动静态图默认黑白名单设置。[#55373](https://github.com/PaddlePaddle/Paddle/pull/55373), [#55713](https://github.com/PaddlePaddle/Paddle/pull/55713) - argsort、dist、erfinv、nanmedian、poisson 算子和 lamb 优化器算子支持 FP16、BF16 低精度计算。[#51662](https://github.com/PaddlePaddle/Paddle/pull/51662), [#55105](https://github.com/PaddlePaddle/Paddle/pull/55105), [#55287](https://github.com/PaddlePaddle/Paddle/pull/55287), [#55824](https://github.com/PaddlePaddle/Paddle/pull/55824), [#56056](https://github.com/PaddlePaddle/Paddle/pull/56056), [#56184](https://github.com/PaddlePaddle/Paddle/pull/56184), [#55641](https://github.com/PaddlePaddle/Paddle/pull/55641) - 修复 elementwise_max 算子低精度实现，改成使用 FP32 类型进行数值计算，减少精度损失。[#54799](https://github.com/PaddlePaddle/Paddle/pull/54799) - 将 Reduce 类算子计算需要的临时结果 Tensor 改成 FP32 类型，避免将中间结果转换成低精度带来的精度损失。[#55709](https://github.com/PaddlePaddle/Paddle/pull/55709)) - flip、roll & roll_grad、index_put & index_put_grad 等算子 GPU 代码实现优化，在性能不下降的前提下移除不必要的 C++模板，优化算子编译耗时并减少编译生成的二进制体积。[#57309](https://github.com/PaddlePaddle/Paddle/pull/57309), [#57525](https://github.com/PaddlePaddle/Paddle/pull/57525) - bernoulli 算子增加对输入概率合法性的检查。[#59174](https://github.com/PaddlePaddle/Paddle/pull/59174) ### 性能优化 - 优化 BroadcastKernel 对大 Tensor 的支持，改成对大 Tensor 切片多次调用 INT32 版本实现的方式，算子性能提升 7.27x。[#57313](https://github.com/PaddlePaddle/Paddle/pull/57313), [#57996](https://github.com/PaddlePaddle/Paddle/pull/57996) - 优化 Tensor 保存接口的性能，通过先将 Tensor 拷贝到 CPU 再转 numpy，避免 Tensor 不连续时自动转换成连续 Tensor 的开销。[#57040](https://github.com/PaddlePaddle/Paddle/pull/57040) ### Bug Fix - 修复 memmory_efficient_attention 算子对 sm_90 的支持。[#58070](https://github.com/PaddlePaddle/Paddle/pull/58070) - 修复 softmax 算子，当 axis=-1 且长度大于 100000 的实现出现的 NaN 问题。[#57851](https://github.com/PaddlePaddle/Paddle/pull/57851) - 修复 set_constant 算子在一些情况下出现 GPU 访存错误问题。[#59905](https://github.com/PaddlePaddle/Paddle/pull/59905) - 修复 layer_norm 算子快速实现版本中出现的 GPU 存储读写竞争问题。[#56435](https://github.com/PaddlePaddle/Paddle/pull/56435) ## 拓展神经网络编译器 CINN 架构能力在本次更新中，飞桨神经网络编译器 CINN 的重点在于架构的梳理和能力的全面扩展。鉴于大模型对动态 Shape 的需求日益增长，初步探索并实现了在动态 shape 下编译器的有效运行和优化策略。在架构层面，引入了 Python DSL，这一举措显著提升了 CINN 的开发便捷性和 Debug 能力，使得开发者能够更高效地编写和调试代码。同时，对 Schedule 的逻辑进行了重构，以 GroupSchedule 为主导，从而在算子 Group 层面实现更加通用且稳定的优化策略。为了增强 CINN 的稳定性，探索并引入了强约束组件，这一组件能够有效减少系统中的不确定性和潜在错误。此外，对 CINN 的历史工具类和软件结构进行了系统性的整理、优化和改进，进一步提升了代码的可读性和可维护性。在与飞桨其他组件的整合方面，进一步加强了 CINN 与 PIR、Paddle 的紧密结合，使得编译器与飞桨整体框架更加协调一致。这一改进不仅提升了编译器的性能，还为开发者提供了更加流畅和统一的开发体验。 ### 兼容性升级 - 更新存储读取接口至兼容 Paddle 2.0。 [#55836](https://github.com/PaddlePaddle/Paddle/pull/55836) - 更新 relu6 Op Mapper 的兼容性。 [#55611](https://github.com/PaddlePaddle/Paddle/pull/55611) ### 改造废弃 - 删除旧的 Schedule 形式。 [#55566](https://github.com/PaddlePaddle/Paddle/pull/55566),[#55391](https://github.com/PaddlePaddle/Paddle/pull/55391) - 删除一些过时测试。 [#56245](https://github.com/PaddlePaddle/Paddle/pull/56245),[#57987](https://github.com/PaddlePaddle/Paddle/pull/57987) - 删除不再适用的 remove_nested_block Visitor 工具。 [#56972](https://github.com/PaddlePaddle/Paddle/pull/56972) - 删除其他无用代码。 [#55413](https://github.com/PaddlePaddle/Paddle/pull/55413) ### 新功能 - 增加飞桨端 CINN paddle.framework.core.is_run_with_cinn()运行接口。 [#54355](https://github.com/PaddlePaddle/Paddle/pull/54355) - 增加 CINN 相关算子逻辑，包括各种组合算子拆解逻辑。 [#56072](https://github.com/PaddlePaddle/Paddle/pull/56072),[#58210](https://github.com/PaddlePaddle/Paddle/pull/58210),[#58502](https://github.com/PaddlePaddle/Paddle/pull/58502), [#58591](https://github.com/PaddlePaddle/Paddle/pull/58591), [#58981](https://github.com/PaddlePaddle/Paddle/pull/58981), [#59135](https://github.com/PaddlePaddle/Paddle/pull/59135), [#59274](https://github.com/PaddlePaddle/Paddle/pull/59274), [#59306](https://github.com/PaddlePaddle/Paddle/pull/59306), [#59202](https://github.com/PaddlePaddle/Paddle/pull/59202), [#59176](https://github.com/PaddlePaddle/Paddle/pull/59176), [#59534](https://github.com/PaddlePaddle/Paddle/pull/59534), [#59713](https://github.com/PaddlePaddle/Paddle/pull/59713), [#59798](https://github.com/PaddlePaddle/Paddle/pull/59798)；支持 bf16、amp 等形式[#54399](https://github.com/PaddlePaddle/Paddle/pull/54399), [#54368](https://github.com/PaddlePaddle/Paddle/pull/54368), [#54608](https://github.com/PaddlePaddle/Paddle/pull/54608)；支持算子零维能力[#54892](https://github.com/PaddlePaddle/Paddle/pull/54892), [#54919](https://github.com/PaddlePaddle/Paddle/pull/54919), [#54907](https://github.com/PaddlePaddle/Paddle/pull/54907), [#54966](https://github.com/PaddlePaddle/Paddle/pull/54966) - CINN 和飞桨 PIR、组合算子交界运行方式，使新增 PIR 和 CINN 运行浑然一体。 [#54732](https://github.com/PaddlePaddle/Paddle/pull/54732), [#56074](https://github.com/PaddlePaddle/Paddle/pull/56074), [#58216](https://github.com/PaddlePaddle/Paddle/pull/58216), [#55680](https://github.com/PaddlePaddle/Paddle/pull/55680), [#56302](https://github.com/PaddlePaddle/Paddle/pull/56302), [#59037](https://github.com/PaddlePaddle/Paddle/pull/59037), [#55186](https://github.com/PaddlePaddle/Paddle/pull/55186), [#58641](https://github.com/PaddlePaddle/Paddle/pull/58641) - 对 CINN 变化起到稳定作用的强约束组件。 [#58719](https://github.com/PaddlePaddle/Paddle/pull/58719), [#59309](https://github.com/PaddlePaddle/Paddle/pull/59309), [#58993](https://github.com/PaddlePaddle/Paddle/pull/58993) - Group Schedule 相关的 CINN 架构流程添加。 [#58399](https://github.com/PaddlePaddle/Paddle/pull/58399), [#56444](https://github.com/PaddlePaddle/Paddle/pull/56444) - CINN 架构功能初步增加 CUTLASS、报错处理、NVRTC Cubin Fmad 选项。 [#58079](https://github.com/PaddlePaddle/Paddle/pull/58079), [#57198](https://github.com/PaddlePaddle/Paddle/pull/57198), [#58794](https://github.com/PaddlePaddle/Paddle/pull/58794) - CINN 增加 Python 界面语言。 [#57731](https://github.com/PaddlePaddle/Paddle/pull/57731), [#57515](https://github.com/PaddlePaddle/Paddle/pull/57515), [#57644](https://github.com/PaddlePaddle/Paddle/pull/57644), [#57981](https://github.com/PaddlePaddle/Paddle/pull/57981), [#58009](https://github.com/PaddlePaddle/Paddle/pull/58009) - CINN 增加动态 Shape 功能，涵盖 ASTGen 可以代替 ISL 产生动态 Shape 符号 [#56360](https://github.com/PaddlePaddle/Paddle/pull/56360), [#57207](https://github.com/PaddlePaddle/Paddle/pull/57207), [#57454](https://github.com/PaddlePaddle/Paddle/pull/57454)；增加分桶条件编译功能 [#59165](https://github.com/PaddlePaddle/Paddle/pull/59165)；增加 Schedule、Device、IR 层面支持动态 shape 的功能 [#58988](https://github.com/PaddlePaddle/Paddle/pull/58988), [#59493](https://github.com/PaddlePaddle/Paddle/pull/59493), [#58717](https://github.com/PaddlePaddle/Paddle/pull/58717), [#58602](https://github.com/PaddlePaddle/Paddle/pull/58602), [#59196](https://github.com/PaddlePaddle/Paddle/pull/59196) - CINN Group Schedule 算子 Group 层面做更通用稳定的 Schedule 优化。 [#56122](https://github.com/PaddlePaddle/Paddle/pull/56122), [#57777](https://github.com/PaddlePaddle/Paddle/pull/57777), [#57569](https://github.com/PaddlePaddle/Paddle/pull/57569) ### 功能优化 - 丰富或改善算子功能，包括修理反向、FP16、Infershape、算子单测等各种算子过程的改善。 [#56320](https://github.com/PaddlePaddle/Paddle/pull/56320), [#56845](https://github.com/PaddlePaddle/Paddle/pull/56845), [#54939](https://github.com/PaddlePaddle/Paddle/pull/54939),[#54378](https://github.com/PaddlePaddle/Paddle/pull/54378),[#55321](https://github.com/PaddlePaddle/Paddle/pull/55321),[#55336](https://github.com/PaddlePaddle/Paddle/pull/55336),[#55337](https://github.com/PaddlePaddle/Paddle/pull/55337),[#55442](https://github.com/PaddlePaddle/Paddle/pull/55442),[#55470](https://github.com/PaddlePaddle/Paddle/pull/55470),[#55489](https://github.com/PaddlePaddle/Paddle/pull/55489),[#55510](https://github.com/PaddlePaddle/Paddle/pull/55510),[#55547](https://github.com/PaddlePaddle/Paddle/pull/55547),[#55505](https://github.com/PaddlePaddle/Paddle/pull/55505),[#55563](https://github.com/PaddlePaddle/Paddle/pull/55563),[#54280](https://github.com/PaddlePaddle/Paddle/pull/54280),[#59650](https://github.com/PaddlePaddle/Paddle/pull/59650),[#54862](https://github.com/PaddlePaddle/Paddle/pull/54862),[#55135](https://github.com/PaddlePaddle/Paddle/pull/55135),[#55292](https://github.com/PaddlePaddle/Paddle/pull/55292),[#55333](https://github.com/PaddlePaddle/Paddle/pull/55333),[#55316](https://github.com/PaddlePaddle/Paddle/pull/55316),[#55379](https://github.com/PaddlePaddle/Paddle/pull/55379),[#55326](https://github.com/PaddlePaddle/Paddle/pull/55326) - CINN、飞桨、PIR、组合算子交界运行方式改善，主要包括各种和 PIR 及其执行器接口和 CINN 互相支持。 [#59170](https://github.com/PaddlePaddle/Paddle/pull/59170),[#58766](https://github.com/PaddlePaddle/Paddle/pull/58766),[#59255](https://github.com/PaddlePaddle/Paddle/pull/59255),[#59203](https://github.com/PaddlePaddle/Paddle/pull/59203),[#59024](https://github.com/PaddlePaddle/Paddle/pull/59024),[#57829](https://github.com/PaddlePaddle/Paddle/pull/57829),[#58135](https://github.com/PaddlePaddle/Paddle/pull/58135),[#58193](https://github.com/PaddlePaddle/Paddle/pull/58193),[#58207](https://github.com/PaddlePaddle/Paddle/pull/58207),[#58606](https://github.com/PaddlePaddle/Paddle/pull/58606),[#59437](https://github.com/PaddlePaddle/Paddle/pull/59437),[#59759](https://github.com/PaddlePaddle/Paddle/pull/59759),[#55075](https://github.com/PaddlePaddle/Paddle/pull/55075),[#56805](https://github.com/PaddlePaddle/Paddle/pull/56805),[#57764](https://github.com/PaddlePaddle/Paddle/pull/57764),[#58620](https://github.com/PaddlePaddle/Paddle/pull/58620),[#59769](https://github.com/PaddlePaddle/Paddle/pull/59769),[#58702](https://github.com/PaddlePaddle/Paddle/pull/58702),[#58749](https://github.com/PaddlePaddle/Paddle/pull/58749),[#59025](https://github.com/PaddlePaddle/Paddle/pull/59025),[#58820](https://github.com/PaddlePaddle/Paddle/pull/58820),[#58908](https://github.com/PaddlePaddle/Paddle/pull/58908),[#58169](https://github.com/PaddlePaddle/Paddle/pull/58169) - 对 CINN 改善稳定作用的强约束组件。 [#55090](https://github.com/PaddlePaddle/Paddle/pull/55090),[#55705](https://github.com/PaddlePaddle/Paddle/pull/55705),[#57587](https://github.com/PaddlePaddle/Paddle/pull/57587),[#59501](https://github.com/PaddlePaddle/Paddle/pull/59501) - CINN IR 和相关工具代码改善。 [#55145](https://github.com/PaddlePaddle/Paddle/pull/55145),[#55955](https://github.com/PaddlePaddle/Paddle/pull/55955),[#56307](https://github.com/PaddlePaddle/Paddle/pull/56307),[#55519](https://github.com/PaddlePaddle/Paddle/pull/55519),[#56958](https://github.com/PaddlePaddle/Paddle/pull/56958),[#57019](https://github.com/PaddlePaddle/Paddle/pull/57019),[#57230](https://github.com/PaddlePaddle/Paddle/pull/57230),[#57531](https://github.com/PaddlePaddle/Paddle/pull/57531),[#57532](https://github.com/PaddlePaddle/Paddle/pull/57532),[#57524](https://github.com/PaddlePaddle/Paddle/pull/57524),[#58770](https://github.com/PaddlePaddle/Paddle/pull/58770),[#59337](https://github.com/PaddlePaddle/Paddle/pull/59337),[#59096](https://github.com/PaddlePaddle/Paddle/pull/59096),[#56274](https://github.com/PaddlePaddle/Paddle/pull/56274),[#56350](https://github.com/PaddlePaddle/Paddle/pull/56350),[#57312](https://github.com/PaddlePaddle/Paddle/pull/57312),[#55171](https://github.com/PaddlePaddle/Paddle/pull/55171) - CINN Group Schedule 算子 Group 层面做更通用稳定的 Schedule 优化。 [#54982](https://github.com/PaddlePaddle/Paddle/pull/54982),[#57963](https://github.com/PaddlePaddle/Paddle/pull/57963),[#58220](https://github.com/PaddlePaddle/Paddle/pull/58220),[#55484](https://github.com/PaddlePaddle/Paddle/pull/55484),[#55935](https://github.com/PaddlePaddle/Paddle/pull/55935),[#55590](https://github.com/PaddlePaddle/Paddle/pull/55590),[#56530](https://github.com/PaddlePaddle/Paddle/pull/56530),[#58344](https://github.com/PaddlePaddle/Paddle/pull/58344),[#59810](https://github.com/PaddlePaddle/Paddle/pull/59810) - CINN 架构功能改善，包括并行编译、低层存储分配方式、打印信息、Group 结构、Pass 结构等。[#56282](https://github.com/PaddlePaddle/Paddle/pull/56282), [#59014](https://github.com/PaddlePaddle/Paddle/pull/59014),[#59209](https://github.com/PaddlePaddle/Paddle/pull/59209),[#52660](https://github.com/PaddlePaddle/Paddle/pull/52660),[#54749](https://github.com/PaddlePaddle/Paddle/pull/54749),[#58694](https://github.com/PaddlePaddle/Paddle/pull/58694),[#58940](https://github.com/PaddlePaddle/Paddle/pull/58940),[#59504](https://github.com/PaddlePaddle/Paddle/pull/59504),[#56123](https://github.com/PaddlePaddle/Paddle/pull/56123) - CINN 改善 codegen、jit instruction、dim args、host kernel 等以支持动态 Shape 功能。 [#58825](https://github.com/PaddlePaddle/Paddle/pull/58825),[#59395](https://github.com/PaddlePaddle/Paddle/pull/59395),[#59398](https://github.com/PaddlePaddle/Paddle/pull/59398),[#59540](https://github.com/PaddlePaddle/Paddle/pull/59540),[#59470](https://github.com/PaddlePaddle/Paddle/pull/59470),[#59640](https://github.com/PaddlePaddle/Paddle/pull/59640) - CINN 报错优化。 [#54983](https://github.com/PaddlePaddle/Paddle/pull/54983),[#55544](https://github.com/PaddlePaddle/Paddle/pull/55544) - CINN 其他代码清理改善、包括 CI、文件路径、C++17、Flags、第三方库、Docker 等 [#55018](https://github.com/PaddlePaddle/Paddle/pull/55018),[#55121](https://github.com/PaddlePaddle/Paddle/pull/55121),[#55009](https://github.com/PaddlePaddle/Paddle/pull/55009),[#55888](https://github.com/PaddlePaddle/Paddle/pull/55888),[#56168](https://github.com/PaddlePaddle/Paddle/pull/56168),[#56192](https://github.com/PaddlePaddle/Paddle/pull/56192),[#56896](https://github.com/PaddlePaddle/Paddle/pull/56896),[#53861](https://github.com/PaddlePaddle/Paddle/pull/53861),[#55208](https://github.com/PaddlePaddle/Paddle/pull/55208) ### 性能优化 - 对 vit attention 进行融合。 [#54139](https://github.com/PaddlePaddle/Paddle/pull/54139) - 优化 block reduce。 [#58196](https://github.com/PaddlePaddle/Paddle/pull/58196) ### bug 修复 - 算子相关 bug 修复。 [#56280](https://github.com/PaddlePaddle/Paddle/pull/56280),[#57767](https://github.com/PaddlePaddle/Paddle/pull/57767),[#58406](https://github.com/PaddlePaddle/Paddle/pull/58406),[#54406](https://github.com/PaddlePaddle/Paddle/pull/54406),[#54494](https://github.com/PaddlePaddle/Paddle/pull/54494),[#54751](https://github.com/PaddlePaddle/Paddle/pull/54751),[#55674](https://github.com/PaddlePaddle/Paddle/pull/55674),[#55684](https://github.com/PaddlePaddle/Paddle/pull/55684),[#55683](https://github.com/PaddlePaddle/Paddle/pull/55683),[#57798](https://github.com/PaddlePaddle/Paddle/pull/57798),[#57816](https://github.com/PaddlePaddle/Paddle/pull/57816),[#57687](https://github.com/PaddlePaddle/Paddle/pull/57687),[#56719](https://github.com/PaddlePaddle/Paddle/pull/56719),[#59756](https://github.com/PaddlePaddle/Paddle/pull/59756),[#59770](https://github.com/PaddlePaddle/Paddle/pull/59770),[#58811](https://github.com/PaddlePaddle/Paddle/pull/58811) - 流程架构相关 bug 修复。 [#54899](https://github.com/PaddlePaddle/Paddle/pull/54899),[#59737](https://github.com/PaddlePaddle/Paddle/pull/59737),[#59356](https://github.com/PaddlePaddle/Paddle/pull/59356),[#56105](https://github.com/PaddlePaddle/Paddle/pull/56105),[#56662](https://github.com/PaddlePaddle/Paddle/pull/56662),[#58146](https://github.com/PaddlePaddle/Paddle/pull/58146),[#58910](https://github.com/PaddlePaddle/Paddle/pull/58910),[#58121](https://github.com/PaddlePaddle/Paddle/pull/58121),[#58943](https://github.com/PaddlePaddle/Paddle/pull/58943),[#58886](https://github.com/PaddlePaddle/Paddle/pull/58886),[#59642](https://github.com/PaddlePaddle/Paddle/pull/59642),[#56164](https://github.com/PaddlePaddle/Paddle/pull/56164),[#56338](https://github.com/PaddlePaddle/Paddle/pull/56338),[#56966](https://github.com/PaddlePaddle/Paddle/pull/56966),[#59112](https://github.com/PaddlePaddle/Paddle/pull/59112),[#55820](https://github.com/PaddlePaddle/Paddle/pull/55820),[#56660](https://github.com/PaddlePaddle/Paddle/pull/56660),[#57307](https://github.com/PaddlePaddle/Paddle/pull/57307),[#57530](https://github.com/PaddlePaddle/Paddle/pull/57530),[#58236](https://github.com/PaddlePaddle/Paddle/pull/58236),[#55190](https://github.com/PaddlePaddle/Paddle/pull/55190),[#55043](https://github.com/PaddlePaddle/Paddle/pull/55043),[#55667](https://github.com/PaddlePaddle/Paddle/pull/55667) - 其他 bug 修复。 [#57239](https://github.com/PaddlePaddle/Paddle/pull/57239),[#55530](https://github.com/PaddlePaddle/Paddle/pull/55530),[#56605](https://github.com/PaddlePaddle/Paddle/pull/56605),[#58243](https://github.com/PaddlePaddle/Paddle/pull/58243),[#58197](https://github.com/PaddlePaddle/Paddle/pull/58197),[#58197](https://github.com/PaddlePaddle/Paddle/pull/58197),[#56086](https://github.com/PaddlePaddle/Paddle/pull/56086),[#56065](https://github.com/PaddlePaddle/Paddle/pull/56065),[#58775](https://github.com/PaddlePaddle/Paddle/pull/58775),[#54750](https://github.com/PaddlePaddle/Paddle/pull/54750),[#58595](https://github.com/PaddlePaddle/Paddle/pull/58595),[#58873](https://github.com/PaddlePaddle/Paddle/pull/58873) ### 文档 - 增加 README 文件。 [#58349](https://github.com/PaddlePaddle/Paddle/pull/58349) # 4. 部署方向（Paddle Inference） ## 通用推理优化本版本升级提升了推理引擎在 GPU 和 CPU 上性能和易用性，降低了用户使用成本和线上推理的应用成本。在 GPU 上支持了高性能的多线程异步执行器，各模型推理性能提升 5%~10%；同时支持新版本 TensorRT 和 BF16 推理能力，TensorRT 推理性能和易用性进一步提升；在 CPU 上，支持最新版本的 OneDNN 高性能推理，在 SwinTransformer、FastRCNN 等系列模型上性能大幅提升。 - matmul 支持 transpose、broadcast 操作。 [#56827](https://github.com/PaddlePaddle/Paddle/pull/56827) - TruncatedNormal and Assign 支持 FP64 数据类型。[#57507](https://github.com/PaddlePaddle/Paddle/pull/57507) - 支持 conv2d 显式量化推理。[#57160](https://github.com/PaddlePaddle/Paddle/pull/57160),[#58015](https://github.com/PaddlePaddle/Paddle/pull/58015) - 新增 conv_fuse_pass，支持 conv + bn 融合，conv2d_fusion 融合重命名为 fused_conv2d_add_act。 [#58724](https://github.com/PaddlePaddle/Paddle/pull/58724),[#55374](https://github.com/PaddlePaddle/Paddle/pull/55374),[#54477](https://github.com/PaddlePaddle/Paddle/pull/54477),[#59431](https://github.com/PaddlePaddle/Paddle/pull/59431) - 混合精度推理支持 OP 白名单。[#56535](https://github.com/PaddlePaddle/Paddle/pull/56535) - 默认开启 OneDNN 优化，支持 SwinTransformer、FastRCNNd 等推理优化。[#58560](https://github.com/PaddlePaddle/Paddle/pull/58560),[#59394](https://github.com/PaddlePaddle/Paddle/pull/59394),[#59421](https://github.com/PaddlePaddle/Paddle/pull/59421),[#58435](https://github.com/PaddlePaddle/Paddle/pull/58435),[#58488](https://github.com/PaddlePaddle/Paddle/pull/58488),[#59259](https://github.com/PaddlePaddle/Paddle/pull/59259),[#56303](https://github.com/PaddlePaddle/Paddle/pull/56303),[#56782](https://github.com/PaddlePaddle/Paddle/pull/56782),[#57598](https://github.com/PaddlePaddle/Paddle/pull/57598),[#58361](https://github.com/PaddlePaddle/Paddle/pull/58361),[#59641](https://github.com/PaddlePaddle/Paddle/pull/59641),[#59527](https://github.com/PaddlePaddle/Paddle/pull/59527),[#59663](https://github.com/PaddlePaddle/Paddle/pull/59663),[#59744](https://github.com/PaddlePaddle/Paddle/pull/59744) - 新增 share_data 支持传入指定数据。[#57933](https://github.com/PaddlePaddle/Paddle/pull/57933) ## 大模型推理优化实现了生成式大模型的细粒度融合推理优化，该优化方案既保证了高性能的推理能力，又具备良好的可拓展性。用户可以根据需要，灵活运用各种细粒度融合算子和飞桨原生算子，自由组合构建生成式大模型的网络结构，从而实现高效且低成本的推理。此外，我们的方案还支持主流的生成式大模型结构，显著降低了这类模型的推理部署成本，为生成式大模型的高效、低成本落地提供了有力支持。 - 支持 FMHA/MMHA 对 CacheKV 划分 block 调度。[#59462](https://github.com/PaddlePaddle/Paddle/pull/59462) - RoPE 编码融合算子支持输入 sin/cos 值。[#55415](https://github.com/PaddlePaddle/Paddle/pull/55415) - 新增细粒度融合算子支持生成式大模型高性能推理优化，新增 quant_linear、weight_quantize、linear_compress 等算子支持大模型量化推理。[#57852](https://github.com/PaddlePaddle/Paddle/pull/57852),[#55128](https://github.com/PaddlePaddle/Paddle/pull/55128),[#59090](https://github.com/PaddlePaddle/Paddle/pull/59090),[#56706](https://github.com/PaddlePaddle/Paddle/pull/56706),[#59951](https://github.com/PaddlePaddle/Paddle/pull/59951),[#55490](https://github.com/PaddlePaddle/Paddle/pull/55490),[#59291](https://github.com/PaddlePaddle/Paddle/pull/59291),[#59441](https://github.com/PaddlePaddle/Paddle/pull/59441),[#59778](https://github.com/PaddlePaddle/Paddle/pull/59778),[#59651](https://github.com/PaddlePaddle/Paddle/pull/59651)[#55301](https://github.com/PaddlePaddle/Paddle/pull/55301),[#58637](https://github.com/PaddlePaddle/Paddle/pull/58637),[#56673](https://github.com/PaddlePaddle/Paddle/pull/56673),[#56401](https://github.com/PaddlePaddle/Paddle/pull/56401) - 支持变长推理系列 API。[#57948](https://github.com/PaddlePaddle/Paddle/pull/57948) - 支持 GQA 推理。[#58472](https://github.com/PaddlePaddle/Paddle/pull/58472),[#58836](https://github.com/PaddlePaddle/Paddle/pull/58836) - 新增 masked multihead attention 支持高性能 MMHA 推理。[#55344](https://github.com/PaddlePaddle/Paddle/pull/55344),[#56411](https://github.com/PaddlePaddle/Paddle/pull/56411),[#58134](https://github.com/PaddlePaddle/Paddle/pull/58134),[#57936](https://github.com/PaddlePaddle/Paddle/pull/57936) - weight_quantize/weight_only_linear 支持 Volta 架构。[#58082](https://github.com/PaddlePaddle/Paddle/pull/58082) - 新增 weight_only_linear_grad 支持大模型 weight only 量化梯度回传。[#57685](https://github.com/PaddlePaddle/Paddle/pull/57685) - 修复大模型动转静问题，优化静态图卡间通信初始化逻辑。[#56390](https://github.com/PaddlePaddle/Paddle/pull/56390),[#57169](https://github.com/PaddlePaddle/Paddle/pull/57169),[#56688](https://github.com/PaddlePaddle/Paddle/pull/56688),[#56592](https://github.com/PaddlePaddle/Paddle/pull/56592),[#58868](https://github.com/PaddlePaddle/Paddle/pull/58868) - 优化 top_p_sampling 随机数生成逻辑。[#59494](https://github.com/PaddlePaddle/Paddle/pull/59494) ## Paddle-TensorRT 推理优化 - elementwise_add 融合支持 NHWC 格式。 [#56795](https://github.com/PaddlePaddle/Paddle/pull/56795) - conv2d 支持 filter 作为输入。[#55246](https://github.com/PaddlePaddle/Paddle/pull/55246)。 - 支持 BF16、FP64 推理。[#59765](https://github.com/PaddlePaddle/Paddle/pull/59765),[#55520](https://github.com/PaddlePaddle/Paddle/pull/55520) - 新增 MarkTrtEngineOutputs API 支持指定 TensorRT Engine 输出。 [#56858](https://github.com/PaddlePaddle/Paddle/pull/56858),[#56188](https://github.com/PaddlePaddle/Paddle/pull/56188),[#57407](https://github.com/PaddlePaddle/Paddle/pull/57407) - 支持自定义 OP 自动生成 TensorRT Plugin。[#58976](https://github.com/PaddlePaddle/Paddle/pull/58976),[#56037](https://github.com/PaddlePaddle/Paddle/pull/56037) - TensorRT 推理支持指定输入 hook，优化 shape 收集流程。[#59466](https://github.com/PaddlePaddle/Paddle/pull/59466),[#54841](https://github.com/PaddlePaddle/Paddle/pull/54841),[#57498](https://github.com/PaddlePaddle/Paddle/pull/57498),[#54861](https://github.com/PaddlePaddle/Paddle/pull/54861),[#54432](https://github.com/PaddlePaddle/Paddle/pull/54432),[#55503](https://github.com/PaddlePaddle/Paddle/pull/55503) - TensorRT 推理支持保存 Tuning 后的推理模型。[#55893](https://github.com/PaddlePaddle/Paddle/pull/55893),[#56952](https://github.com/PaddlePaddle/Paddle/pull/56952),[#57031](https://github.com/PaddlePaddle/Paddle/pull/57031) - 支持变长 Transformer 模型 PromptTuning。[#57034](https://github.com/PaddlePaddle/Paddle/pull/57034) - 新增 bitwise_and、bitwise_or、bitwise_not、cumsum、einsum、lookup_table、assign、flip、size、scatter、solve、unbind、reduce、argsort 算子支持，优化已有算子支持。[#59214](https://github.com/PaddlePaddle/Paddle/pull/59214),[#59293](https://github.com/PaddlePaddle/Paddle/pull/59293),[#54882](https://github.com/PaddlePaddle/Paddle/pull/54882),[#54097](https://github.com/PaddlePaddle/Paddle/pull/54097),[#54860](https://github.com/PaddlePaddle/Paddle/pull/54860),[#55426](https://github.com/PaddlePaddle/Paddle/pull/55426),[#54372](https://github.com/PaddlePaddle/Paddle/pull/54372),[#55688](https://github.com/PaddlePaddle/Paddle/pull/55688),[#56069](https://github.com/PaddlePaddle/Paddle/pull/56069),[#59563](https://github.com/PaddlePaddle/Paddle/pull/59563),[#59317](https://github.com/PaddlePaddle/Paddle/pull/59317),[#59424](https://github.com/PaddlePaddle/Paddle/pull/59424),[#55476](https://github.com/PaddlePaddle/Paddle/pull/55476),[#56043](https://github.com/PaddlePaddle/Paddle/pull/56043),[#58549](https://github.com/PaddlePaddle/Paddle/pull/58549),[#57326](https://github.com/PaddlePaddle/Paddle/pull/57326),[#59409](https://github.com/PaddlePaddle/Paddle/pull/59409)) - TensorRT 默认开启显存共享。[#59495](https://github.com/PaddlePaddle/Paddle/pull/59495),[#58251](https://github.com/PaddlePaddle/Paddle/pull/58251) - PrelnResidualBiasPluginDynamic 支持 4D 输入。[#56304](https://github.com/PaddlePaddle/Paddle/pull/56304) - 新增 SM80 以下架构 Paddle-TRT 推理对 FlashAttention 的支持。[#56492](https://github.com/PaddlePaddle/Paddle/pull/56492) ## 改造废弃 - OneDNN 中删除 fc_elementwise_add 融合。[#55504](https://github.com/PaddlePaddle/Paddle/pull/55504) - 删除 redunant op。 [#54442](https://github.com/PaddlePaddle/Paddle/pull/54442) ## Bug Fix - 修复 Inference so 链接 flags 冲突问题。[#59755](https://github.com/PaddlePaddle/Paddle/pull/59755) - 修复 constant_folding pass 执行报错。[#55556](https://github.com/PaddlePaddle/Paddle/pull/55556) - 修复 softmax 前向速度问题及反向精度问题。[#56036](https://github.com/PaddlePaddle/Paddle/pull/56036),[#57858](https://github.com/PaddlePaddle/Paddle/pull/57858)[#57538](https://github.com/PaddlePaddle/Paddle/pull/57538) - 修复自定义 OP while 报错及导出问题。[#58898](https://github.com/PaddlePaddle/Paddle/pull/58898),[#59318](https://github.com/PaddlePaddle/Paddle/pull/59318) - 修复 Windows 平台 CUDA 12.0 编译问题。[#59852](https://github.com/PaddlePaddle/Paddle/pull/59852) - 修复 TensorRT 版本大于等于 8.6 时推理部分算子报错问题。[#54379](https://github.com/PaddlePaddle/Paddle/pull/54379),[#54679](https://github.com/PaddlePaddle/Paddle/pull/54679),[#54251](https://github.com/PaddlePaddle/Paddle/pull/54251) - 修复、删除推理融合 Pass。[#54846](https://github.com/PaddlePaddle/Paddle/pull/54846),[#54887](https://github.com/PaddlePaddle/Paddle/pull/54887),[#55573](https://github.com/PaddlePaddle/Paddle/pull/55573),[#56434](https://github.com/PaddlePaddle/Paddle/pull/56434),[#56326](https://github.com/PaddlePaddle/Paddle/pull/56326),[#56753](https://github.com/PaddlePaddle/Paddle/pull/56753),[#57491](https://github.com/PaddlePaddle/Paddle/pull/57491),[#56909](https://github.com/PaddlePaddle/Paddle/pull/56909),[#54536](https://github.com/PaddlePaddle/Paddle/pull/54536),[#55073](https://github.com/PaddlePaddle/Paddle/pull/55073),[#55081](https://github.com/PaddlePaddle/Paddle/pull/55081),[#55240](https://github.com/PaddlePaddle/Paddle/pull/55240),[#56439](https://github.com/PaddlePaddle/Paddle/pull/56439),[#59009](https://github.com/PaddlePaddle/Paddle/pull/59009) - 修复多 Stream 推理上下文切换报错问题。[#57629](https://github.com/PaddlePaddle/Paddle/pull/57629),[#58048](https://github.com/PaddlePaddle/Paddle/pull/58048),[#54994](https://github.com/PaddlePaddle/Paddle/pull/54994) # 5. 硬件适配 ## 硬件适配方案 (Custom Device) 在本次更新中，新增了对分布式高级策略、自定义算子和自定义融合策略的支持。通过升级分布式通信库，新增了对 MP、GroupShared、PP、SP 和 MOE 等多项高级分布式策略的支持。同时支持厂商灵活接入不同颗粒度的 Transformer 算子库并通过融合 Pass 修改计算图进行性能加速。 ### 新功能 - CustomDevice 升级对 Paddle 最新分布式通信库 CommContext 的支持，并新增了多种高级分布式策略 GroupShared 和 MOE 等策略。[#56301](https://github.com/PaddlePaddle/Paddle/pull/56301),[#54671](https://github.com/PaddlePaddle/Paddle/pull/54671),[#57957](https://github.com/PaddlePaddle/Paddle/pull/57957),[#56669](https://github.com/PaddlePaddle/Paddle/pull/56669),[#54384](https://github.com/PaddlePaddle/Paddle/pull/54384),[#54572](https://github.com/PaddlePaddle/Paddle/pull/54572),[#54573](https://github.com/PaddlePaddle/Paddle/pull/54573),[#54676](https://github.com/PaddlePaddle/Paddle/pull/54676) - 新增 CustomDevice 对 CustomOP 的支持，并可注册 Paddle PHI 算子库中尚未定义的算子,同时新增 CustomDevice 通过 CAPI 支持 CustomOP。[#57038](https://github.com/PaddlePaddle/Paddle/pull/57038),[#55532](https://github.com/PaddlePaddle/Paddle/pull/55532),[#56755](https://github.com/PaddlePaddle/Paddle/pull/56755),[#55532](https://github.com/PaddlePaddle/Paddle/pull/55532),[#55533](https://github.com/PaddlePaddle/Paddle/pull/55533),[#55659](https://github.com/PaddlePaddle/Paddle/pull/55659) - 新增 CustomDevice 对 CustomPass 功能的功能，支持通过 Python API 修改计算图 IR。[#55511](https://github.com/PaddlePaddle/Paddle/pull/55511),[#55728](https://github.com/PaddlePaddle/Paddle/pull/55728) - 新增 CustomDevice 对 Paddle run_check 健康功能检查的支持。[#56318](https://github.com/PaddlePaddle/Paddle/pull/56318) - 新增 CustomDevice 对 StreamSafeAllocator 的支持。[#55393](https://github.com/PaddlePaddle/Paddle/pull/55393),[#56380](https://github.com/PaddlePaddle/Paddle/pull/56380),[#56536](https://github.com/PaddlePaddle/Paddle/pull/56536),[#58035](https://github.com/PaddlePaddle/Paddle/pull/58035) - 新增 CustomDevice 对 DataTransform 的支持。[#56627](https://github.com/PaddlePaddle/Paddle/pull/56627) ### 功能优化 - 新增 CustomDevice，支持飞桨更多的接口，包括 Variable.set_value,adamw,share_external_data,mp_allreduce_sum,tensor.numpy,get_paddle_place, GeneratorState。[#55272](https://github.com/PaddlePaddle/Paddle/pull/55272), [#56386](https://github.com/PaddlePaddle/Paddle/pull/56386), [#57253](https://github.com/PaddlePaddle/Paddle/pull/57253), [#56927](https://github.com/PaddlePaddle/Paddle/pull/56927),[#56189](https://github.com/PaddlePaddle/Paddle/pull/56189),[#55225](https://github.com/PaddlePaddle/Paddle/pull/55225),[#55247](https://github.com/PaddlePaddle/Paddle/pull/55247) - 修改 CustomDevice 动态库加载方式，从 RTLD_NOW 改为 RTLD_LAZY，方便后续检查 Custom Device 相关软件栈版本的兼容性。 [#57544](https://github.com/PaddlePaddle/Paddle/pull/57544) - 新增 CustomDevice 在混合精度训练下对 FP16 算子的检测功能。[#56053](https://github.com/PaddlePaddle/Paddle/pull/56053),[#56176](https://github.com/PaddlePaddle/Paddle/pull/56176) ### Bug Fix - 修复 CustomDevice 对分布式通信库支持上的一些问题。[#55293](https://github.com/PaddlePaddle/Paddle/pull/55293),[#58038](https://github.com/PaddlePaddle/Paddle/pull/58038),[#59800](https://github.com/PaddlePaddle/Paddle/pull/59800) - 修复 CustomDevice 在部分算子上的问题，包括 c_softmax_with_cross_entropy,data loader,SplitDenseTensor,grad accumulation,atan2 grad。[#56486](https://github.com/PaddlePaddle/Paddle/pull/56486),[#55541](https://github.com/PaddlePaddle/Paddle/pull/55541),[#55615](https://github.com/PaddlePaddle/Paddle/pull/55615),[#56052](https://github.com/PaddlePaddle/Paddle/pull/56052),[#56067](https://github.com/PaddlePaddle/Paddle/pull/56067) - 修复 CustomDevice 中设备管理的一些问题，包括设备异常 ([#56556](https://github.com/PaddlePaddle/Paddle/pull/56556),[#58639](https://github.com/PaddlePaddle/Paddle/pull/58639),[#55173](https://github.com/PaddlePaddle/Paddle/pull/55173)), 异常事件([#56745](https://github.com/PaddlePaddle/Paddle/pull/56745),[#58059](https://github.com/PaddlePaddle/Paddle/pull/58059)), 显存异常([#56977](https://github.com/PaddlePaddle/Paddle/pull/56977),[#59247](https://github.com/PaddlePaddle/Paddle/pull/59247),[#54606](https://github.com/PaddlePaddle/Paddle/pull/54606)), 设备初始化 ([#57099](https://github.com/PaddlePaddle/Paddle/pull/57099),[#57994](https://github.com/PaddlePaddle/Paddle/pull/57994))，设备释放([#54932](https://github.com/PaddlePaddle/Paddle/pull/54932),[#55351](https://github.com/PaddlePaddle/Paddle/pull/55351),[#55783](https://github.com/PaddlePaddle/Paddle/pull/55783))，和设备资源池等。([#55229](https://github.com/PaddlePaddle/Paddle/pull/55229),[#56580](https://github.com/PaddlePaddle/Paddle/pull/56580)) - 修复 CustomDevice 编译相关问题。[#56760](https://github.com/PaddlePaddle/Paddle/pull/56760),[#56766](https://github.com/PaddlePaddle/Paddle/pull/56766) ## 昆仑 XPU ### 新功能 - 新增 XPTI (XPU Profiling Tool Interface) 支持运行时性能数据的采集和分析功能。[#54685](https://github.com/PaddlePaddle/Paddle/pull/54685),[#54690](https://github.com/PaddlePaddle/Paddle/pull/54690),[#54800](https://github.com/PaddlePaddle/Paddle/pull/54800) - 完成对 Paddle 最新分布式通信库 CommContext 的支持。[#59418](https://github.com/PaddlePaddle/Paddle/pull/59418) - 新增 XPU 融合算子包括 fast_where。[#55628](https://github.com/PaddlePaddle/Paddle/pull/55628) - 新增 XPU Pluign 功能支持，方便用户可通过 XTDK 编程方式开发 XPU 自定义算子。[#55101](https://github.com/PaddlePaddle/Paddle/pull/55101),[#59326](https://github.com/PaddlePaddle/Paddle/pull/59326) - 新增 XPU 对 AutoGrowthAllocator 的支持。[#54121](https://github.com/PaddlePaddle/Paddle/pull/54121) - 新增昆仑 3 的算子支持列表。[#57683](https://github.com/PaddlePaddle/Paddle/pull/57683) ### 功能优化 - 对 XPU Inference API 进行升级。[#54342](https://github.com/PaddlePaddle/Paddle/pull/54342) - 优化部分 XPU 算子性能和新增部分 XPU 算子对 bf16 的的支持，包括 unique/index_put,squeeze/unsqueeze kernels,swish/swish_grad,scatter_nd_add_grad/slice,rsqrt/bitwise_or/arange_tensor,where,collective 算子等。[#56582](https://github.com/PaddlePaddle/Paddle/pull/56582),[#58161](https://github.com/PaddlePaddle/Paddle/pull/58161),[#58440](https://github.com/PaddlePaddle/Paddle/pull/58440),[#58580](https://github.com/PaddlePaddle/Paddle/pull/58580),[#58950](https://github.com/PaddlePaddle/Paddle/pull/58950),[#58616](https://github.com/PaddlePaddle/Paddle/pull/58616),[#59273](https://github.com/PaddlePaddle/Paddle/pull/59273) - 优化 XPU 内存管理，避免内存泄漏。[#59334](https://github.com/PaddlePaddle/Paddle/pull/59334),[#54847](https://github.com/PaddlePaddle/Paddle/pull/54847) - 支持 INT8 推理。[#57258](https://github.com/PaddlePaddle/Paddle/pull/57258) - 新增 FP16 系列推理算子支持。[#55642](https://github.com/PaddlePaddle/Paddle/pull/55642),[#54410](https://github.com/PaddlePaddle/Paddle/pull/54410) - 支持 share_external_memory 接口传入输入输出。[#55170](https://github.com/PaddlePaddle/Paddle/pull/55170) - 开源量化模型 XPU 推理支持。[#58568](https://github.com/PaddlePaddle/Paddle/pull/58568) - 新增 context_gm_size 配置代替在 Pass 中分配 global memory。[#54674](https://github.com/PaddlePaddle/Paddle/pull/54674) - 新增 embedding、fast_gather_nd plugin。[#56488](https://github.com/PaddlePaddle/Paddle/pull/56488),[#56103](https://github.com/PaddlePaddle/Paddle/pull/56103) - 支持 fast_layternorm + leaky_relu 融合。[#57113](https://github.com/PaddlePaddle/Paddle/pull/57113) - KL1 和 KL2 精度下 elementwise_min/max/floordiv/where 推理支持。[#58422](https://github.com/PaddlePaddle/Paddle/pull/58422) - 支持 fc 和 conv2d 算子 autotune 配置。[#58801](https://github.com/PaddlePaddle/Paddle/pull/58801) - 支持 conv 和 fc 动态量化。[#59307](https://github.com/PaddlePaddle/Paddle/pull/59307) - fc + act 融合支持 sigmoid, swish and relu6。[#54486](https://github.com/PaddlePaddle/Paddle/pull/54486) - elementwise_sub/elementwise_div 支持 int 数据类型。[#55920](https://github.com/PaddlePaddle/Paddle/pull/55920) ### Bug Fix - 修复 XPU 通信库问题和部分算子问题包括 rnn、layer_norm_grad、yolo_box。 ([#55475](https://github.com/PaddlePaddle/Paddle/pull/55475),[#55515](https://github.com/PaddlePaddle/Paddle/pull/55515)) ([#55656](https://github.com/PaddlePaddle/Paddle/pull/55656),[#54669](https://github.com/PaddlePaddle/Paddle/pull/54669),[#55310](https://github.com/PaddlePaddle/Paddle/pull/55310) ## 海光 DCU ### Bug Fix - 修复海光 DCU 部分算子问题，包括 rnn,concat/split,fft 等。[#59402](https://github.com/PaddlePaddle/Paddle/pull/59402),[#55821](https://github.com/PaddlePaddle/Paddle/pull/55821),[#56340](https://github.com/PaddlePaddle/Paddle/pull/56340)) - 修复海光 DCU 通信库相关问题。[#57110](https://github.com/PaddlePaddle/Paddle/pull/57110) - 修复海光 DCU 编译相关问题。[#59775](https://github.com/PaddlePaddle/Paddle/pull/59775),[#55507](https://github.com/PaddlePaddle/Paddle/pull/55507),[#55612](https://github.com/PaddlePaddle/Paddle/pull/55612),[#54952](https://github.com/PaddlePaddle/Paddle/pull/54952),[#55076](https://github.com/PaddlePaddle/Paddle/pull/55076),[#56079](https://github.com/PaddlePaddle/Paddle/pull/56079),[#54874](https://github.com/PaddlePaddle/Paddle/pull/54874)) - 修复海光 DCU 对 BF16 数据类型的支持问题。[#56517](https://github.com/PaddlePaddle/Paddle/pull/56517) # 6. 环境适配采用模块化编译的方式优化了 CMake 代码的逻辑，提升了飞桨全量编译和增量编译的效率，提升了 RD 本地开发效率，同时支持了 Python3.12，CUDA12，Hopper 架构编译，并引入 Clang 等工具全面优化了代码格式。此外，将 C++单测从链接静态库的方式转变为链接动态库，减小编译体积。这些改进措施为用户提供更加流畅、高效地安装和开发体验。 - CMake 代码优化：分模块和目录编译成独立的静态库，并减少编译依赖，提升增量编译效率。[#59095](https://github.com/PaddlePaddle/Paddle/pull/59095), [#58960](https://github.com/PaddlePaddle/Paddle/pull/58960),[#56591](https://github.com/PaddlePaddle/Paddle/pull/56591),[#58484](https://github.com/PaddlePaddle/Paddle/pull/58484) - CMake 编译分层：将公共组件拆分到公有 common 库，自下而上实现飞桨架构的编译分层，提高编译效率。[#56442](https://github.com/PaddlePaddle/Paddle/pull/56442),[#54729](https://github.com/PaddlePaddle/Paddle/pull/54729),[#55733](https://github.com/PaddlePaddle/Paddle/pull/55733),[#56352](https://github.com/PaddlePaddle/Paddle/pull/56352),[#55109](https://github.com/PaddlePaddle/Paddle/pull/55109),[#54992](https://github.com/PaddlePaddle/Paddle/pull/54992),[#57698](https://github.com/PaddlePaddle/Paddle/pull/57698),[#55147](https://github.com/PaddlePaddle/Paddle/pull/55147),[#55113](https://github.com/PaddlePaddle/Paddle/pull/55113),[#56691](https://github.com/PaddlePaddle/Paddle/pull/56691),[#58618](https://github.com/PaddlePaddle/Paddle/pull/58618),[#58899](https://github.com/PaddlePaddle/Paddle/pull/58899),[#59140](https://github.com/PaddlePaddle/Paddle/pull/59140),[#59129](https://github.com/PaddlePaddle/Paddle/pull/59129),[#59222](https://github.com/PaddlePaddle/Paddle/pull/59222),[#59105](https://github.com/PaddlePaddle/Paddle/pull/59105),[#59711](https://github.com/PaddlePaddle/Paddle/pull/59711) - 第三方库离线编译：将第三方依赖库离线编译，CI/CE 系统无需每次编译重复下载第三方库，提升 CI/CE 系统运行效率。[#54344](https://github.com/PaddlePaddle/Paddle/pull/54344),[#54370](https://github.com/PaddlePaddle/Paddle/pull/54370),[#54466](https://github.com/PaddlePaddle/Paddle/pull/54466),[#54438](https://github.com/PaddlePaddle/Paddle/pull/54438),[#54388](https://github.com/PaddlePaddle/Paddle/pull/54388),[#54436](https://github.com/PaddlePaddle/Paddle/pull/54436),[#54392](https://github.com/PaddlePaddle/Paddle/pull/54392),[#54646](https://github.com/PaddlePaddle/Paddle/pull/54646),[#54380](https://github.com/PaddlePaddle/Paddle/pull/54380),[#55501](https://github.com/PaddlePaddle/Paddle/pull/55501),[#55136](https://github.com/PaddlePaddle/Paddle/pull/55136),[#54451](https://github.com/PaddlePaddle/Paddle/pull/54451),[#55631](https://github.com/PaddlePaddle/Paddle/pull/55631),[#55549](https://github.com/PaddlePaddle/Paddle/pull/55549),[#56165](https://github.com/PaddlePaddle/Paddle/pull/56165),[#54391](https://github.com/PaddlePaddle/Paddle/pull/54391),[#54614](https://github.com/PaddlePaddle/Paddle/pull/54614),[#54522](https://github.com/PaddlePaddle/Paddle/pull/54522),[#54764](https://github.com/PaddlePaddle/Paddle/pull/54764),[#54400](https://github.com/PaddlePaddle/Paddle/pull/54400),[#54322](https://github.com/PaddlePaddle/Paddle/pull/54322) - 飞桨支持 Python 3.12。[#59396](https://github.com/PaddlePaddle/Paddle/pull/59396),[#58069](https://github.com/PaddlePaddle/Paddle/pull/58069) - 使用 Clang 等工具对于源代码进行优化，提升代码质量。[#59626](https://github.com/PaddlePaddle/Paddle/pull/59626),[#55895](https://github.com/PaddlePaddle/Paddle/pull/55895),[#56632](https://github.com/PaddlePaddle/Paddle/pull/56632),[#54449](https://github.com/PaddlePaddle/Paddle/pull/54449),[#54523](https://github.com/PaddlePaddle/Paddle/pull/54523),[#54796](https://github.com/PaddlePaddle/Paddle/pull/54796),[#55847](https://github.com/PaddlePaddle/Paddle/pull/55847),[#55807](https://github.com/PaddlePaddle/Paddle/pull/55807),[#56261](https://github.com/PaddlePaddle/Paddle/pull/56261),[#57522](https://github.com/PaddlePaddle/Paddle/pull/57522),[#57868](https://github.com/PaddlePaddle/Paddle/pull/57868),[#57809](https://github.com/PaddlePaddle/Paddle/pull/57809),[#55658](https://github.com/PaddlePaddle/Paddle/pull/55658),[#58285](https://github.com/PaddlePaddle/Paddle/pull/58285),[#55491](https://github.com/PaddlePaddle/Paddle/pull/55491),[#55506](https://github.com/PaddlePaddle/Paddle/pull/55506),[#55279](https://github.com/PaddlePaddle/Paddle/pull/55279),[#55741](https://github.com/PaddlePaddle/Paddle/pull/55741),[#55894](https://github.com/PaddlePaddle/Paddle/pull/55894),[#55704](https://github.com/PaddlePaddle/Paddle/pull/55704),[#55800](https://github.com/PaddlePaddle/Paddle/pull/55800),[#55799](https://github.com/PaddlePaddle/Paddle/pull/55799),[#55983](https://github.com/PaddlePaddle/Paddle/pull/55983),[#55954](https://github.com/PaddlePaddle/Paddle/pull/55954),[#55764](https://github.com/PaddlePaddle/Paddle/pull/55764),[#56246](https://github.com/PaddlePaddle/Paddle/pull/56246),[#56219](https://github.com/PaddlePaddle/Paddle/pull/56219),[#56217](https://github.com/PaddlePaddle/Paddle/pull/56217),[#56216](https://github.com/PaddlePaddle/Paddle/pull/56216),[#56208](https://github.com/PaddlePaddle/Paddle/pull/56208),[#56134](https://github.com/PaddlePaddle/Paddle/pull/56134),[#56253](https://github.com/PaddlePaddle/Paddle/pull/56253),[#56255](https://github.com/PaddlePaddle/Paddle/pull/56255),[#56693](https://github.com/PaddlePaddle/Paddle/pull/56693),[#56692](https://github.com/PaddlePaddle/Paddle/pull/56692),[#56637](https://github.com/PaddlePaddle/Paddle/pull/56637),[#56636](https://github.com/PaddlePaddle/Paddle/pull/56636),[#56647](https://github.com/PaddlePaddle/Paddle/pull/56647),[#56218](https://github.com/PaddlePaddle/Paddle/pull/56218),[#56640](https://github.com/PaddlePaddle/Paddle/pull/56640),[#56635](https://github.com/PaddlePaddle/Paddle/pull/56635),[#55675](https://github.com/PaddlePaddle/Paddle/pull/55675),[#56601](https://github.com/PaddlePaddle/Paddle/pull/56601),[#56485](https://github.com/PaddlePaddle/Paddle/pull/56485),[#56648](https://github.com/PaddlePaddle/Paddle/pull/56648),[#56747](https://github.com/PaddlePaddle/Paddle/pull/56747),[#56676](https://github.com/PaddlePaddle/Paddle/pull/56676),[#56649](https://github.com/PaddlePaddle/Paddle/pull/56649),[#56895](https://github.com/PaddlePaddle/Paddle/pull/56895),[#56994](https://github.com/PaddlePaddle/Paddle/pull/56994),[#56904](https://github.com/PaddlePaddle/Paddle/pull/56904),[#56744](https://github.com/PaddlePaddle/Paddle/pull/56744),[#56954](https://github.com/PaddlePaddle/Paddle/pull/56954),[#57114](https://github.com/PaddlePaddle/Paddle/pull/57114),[#57343](https://github.com/PaddlePaddle/Paddle/pull/57343),[#57483](https://github.com/PaddlePaddle/Paddle/pull/57483),[#57871](https://github.com/PaddlePaddle/Paddle/pull/57871),[#57861](https://github.com/PaddlePaddle/Paddle/pull/57861),[#58028](https://github.com/PaddlePaddle/Paddle/pull/58028),[#57627](https://github.com/PaddlePaddle/Paddle/pull/57627),[#59072](https://github.com/PaddlePaddle/Paddle/pull/59072) - C++从链接静态库转变为链接动态库，减小编译体积，提升编译效率。[#59477](https://github.com/PaddlePaddle/Paddle/pull/59477),[#56630](https://github.com/PaddlePaddle/Paddle/pull/56630),[#57789](https://github.com/PaddlePaddle/Paddle/pull/57789),[#54257](https://github.com/PaddlePaddle/Paddle/pull/54257),[#59620](https://github.com/PaddlePaddle/Paddle/pull/59620),[#59384](https://github.com/PaddlePaddle/Paddle/pull/59384),[#59619](https://github.com/PaddlePaddle/Paddle/pull/59619),[#58583](https://github.com/PaddlePaddle/Paddle/pull/58583),[#58821](https://github.com/PaddlePaddle/Paddle/pull/58821),[#58710](https://github.com/PaddlePaddle/Paddle/pull/58710)，[#58619](https://github.com/PaddlePaddle/Paddle/pull/58619) - 修复源代码编译相关的问题，提升编译安装效率。[#56617](https://github.com/PaddlePaddle/Paddle/pull/56617),[#58195](https://github.com/PaddlePaddle/Paddle/pull/58195),[#56136](https://github.com/PaddlePaddle/Paddle/pull/56136),[#54540](https://github.com/PaddlePaddle/Paddle/pull/54540),[#57172](https://github.com/PaddlePaddle/Paddle/pull/57172),[#54429](https://github.com/PaddlePaddle/Paddle/pull/54429),[#55603](https://github.com/PaddlePaddle/Paddle/pull/55603),[#54807](https://github.com/PaddlePaddle/Paddle/pull/54807),[#56102](https://github.com/PaddlePaddle/Paddle/pull/56102),[#56829](https://github.com/PaddlePaddle/Paddle/pull/56829),[#56951](https://github.com/PaddlePaddle/Paddle/pull/56951),[#56555](https://github.com/PaddlePaddle/Paddle/pull/56555),[#57781](https://github.com/PaddlePaddle/Paddle/pull/57781),[#57836](https://github.com/PaddlePaddle/Paddle/pull/57836),[#58807](https://github.com/PaddlePaddle/Paddle/pull/58807),[#54535](https://github.com/PaddlePaddle/Paddle/pull/54535),[#54946](https://github.com/PaddlePaddle/Paddle/pull/54946),[#54437](https://github.com/PaddlePaddle/Paddle/pull/54437),[#54411](https://github.com/PaddlePaddle/Paddle/pull/54411),[#54411](https://github.com/PaddlePaddle/Paddle/pull/54411),[#54391](https://github.com/PaddlePaddle/Paddle/pull/54391),[#54466](https://github.com/PaddlePaddle/Paddle/pull/54466),[#54480](https://github.com/PaddlePaddle/Paddle/pull/54480),[#54480](https://github.com/PaddlePaddle/Paddle/pull/54480),[#54724](https://github.com/PaddlePaddle/Paddle/pull/54724),[#59193](https://github.com/PaddlePaddle/Paddle/pull/59193),[#54735](https://github.com/PaddlePaddle/Paddle/pull/54735),[#54812](https://github.com/PaddlePaddle/Paddle/pull/54812),[#56430](https://github.com/PaddlePaddle/Paddle/pull/56430),[#56655](https://github.com/PaddlePaddle/Paddle/pull/56655),[#56684](https://github.com/PaddlePaddle/Paddle/pull/56684),[#56774](https://github.com/PaddlePaddle/Paddle/pull/56774),[#56936](https://github.com/PaddlePaddle/Paddle/pull/56936),[#56949](https://github.com/PaddlePaddle/Paddle/pull/56949),[#56974](https://github.com/PaddlePaddle/Paddle/pull/56974),[#57171](https://github.com/PaddlePaddle/Paddle/pull/57171),[#57712](https://github.com/PaddlePaddle/Paddle/pull/57712),[#56617](https://github.com/PaddlePaddle/Paddle/pull/56617),[#58181](https://github.com/PaddlePaddle/Paddle/pull/58181),[#58253](https://github.com/PaddlePaddle/Paddle/pull/58253),[#58268](https://github.com/PaddlePaddle/Paddle/pull/58268),[#59051](https://github.com/PaddlePaddle/Paddle/pull/59051),[#59048](https://github.com/PaddlePaddle/Paddle/pull/59048),[#59081](https://github.com/PaddlePaddle/Paddle/pull/59081),[#59076](https://github.com/PaddlePaddle/Paddle/pull/59076),[#59155](https://github.com/PaddlePaddle/Paddle/pull/59155),[#59253](https://github.com/PaddlePaddle/Paddle/pull/59253),[#59347](https://github.com/PaddlePaddle/Paddle/pull/59347),[#58957](https://github.com/PaddlePaddle/Paddle/pull/58957),[#59443](https://github.com/PaddlePaddle/Paddle/pull/59443),[#58998](https://github.com/PaddlePaddle/Paddle/pull/58998),[#57574](https://github.com/PaddlePaddle/Paddle/pull/57574),[#55889](https://github.com/PaddlePaddle/Paddle/pull/55889),[#59078](https://github.com/PaddlePaddle/Paddle/pull/59078),[#55762](https://github.com/PaddlePaddle/Paddle/pull/55762),[#56252](https://github.com/PaddlePaddle/Paddle/pull/56252),[#56715](https://github.com/PaddlePaddle/Paddle/pull/56715),[#54905](https://github.com/PaddlePaddle/Paddle/pull/54905),[#56978](https://github.com/PaddlePaddle/Paddle/pull/56978),[#57032](https://github.com/PaddlePaddle/Paddle/pull/57032),[#57179](https://github.com/PaddlePaddle/Paddle/pull/57179),[#57179](https://github.com/PaddlePaddle/Paddle/pull/57179),[#58996](https://github.com/PaddlePaddle/Paddle/pull/58996),[#59915](https://github.com/PaddlePaddle/Paddle/pull/59915),[#54883](https://github.com/PaddlePaddle/Paddle/pull/54883),[#56746](https://github.com/PaddlePaddle/Paddle/pull/56746),[#57674](https://github.com/PaddlePaddle/Paddle/pull/57674),[#60117](https://github.com/PaddlePaddle/Paddle/pull/60117),[#55627](https://github.com/PaddlePaddle/Paddle/pull/55627),[#54568](https://github.com/PaddlePaddle/Paddle/pull/54568),[#54450](https://github.com/PaddlePaddle/Paddle/pull/54450),[#54513](https://github.com/PaddlePaddle/Paddle/pull/54513),[#54615](https://github.com/PaddlePaddle/Paddle/pull/54615),[#54913](https://github.com/PaddlePaddle/Paddle/pull/54913),[#54916](https://github.com/PaddlePaddle/Paddle/pull/54916),[#55148](https://github.com/PaddlePaddle/Paddle/pull/55148),[#55125](https://github.com/PaddlePaddle/Paddle/pull/55125),[#55479](https://github.com/PaddlePaddle/Paddle/pull/55479),[#55723](https://github.com/PaddlePaddle/Paddle/pull/55723),[#55831](https://github.com/PaddlePaddle/Paddle/pull/55831),[#55904](https://github.com/PaddlePaddle/Paddle/pull/55904),[#56085](https://github.com/PaddlePaddle/Paddle/pull/56085),[#56259](https://github.com/PaddlePaddle/Paddle/pull/56259),[#56366](https://github.com/PaddlePaddle/Paddle/pull/56366),[#56366](https://github.com/PaddlePaddle/Paddle/pull/56366),[#56546](https://github.com/PaddlePaddle/Paddle/pull/56546),[#56679](https://github.com/PaddlePaddle/Paddle/pull/56679),[#57222](https://github.com/PaddlePaddle/Paddle/pull/57222),[#57387](https://github.com/PaddlePaddle/Paddle/pull/57387),[#57993](https://github.com/PaddlePaddle/Paddle/pull/57993),[#59556](https://github.com/PaddlePaddle/Paddle/pull/59556),[#57931](https://github.com/PaddlePaddle/Paddle/pull/57931),[#58112](https://github.com/PaddlePaddle/Paddle/pull/58112),[#54228](https://github.com/PaddlePaddle/Paddle/pull/54228),[#56913](https://github.com/PaddlePaddle/Paddle/pull/56913),[#56993](https://github.com/PaddlePaddle/Paddle/pull/56993),[#55042](https://github.com/PaddlePaddle/Paddle/pull/55042),[#55305](https://github.com/PaddlePaddle/Paddle/pull/55305),[#55286](https://github.com/PaddlePaddle/Paddle/pull/55286),[#56634](https://github.com/PaddlePaddle/Paddle/pull/56634),[#57778](https://github.com/PaddlePaddle/Paddle/pull/57778),[#58374](https://github.com/PaddlePaddle/Paddle/pull/58374),[#58640](https://github.com/PaddlePaddle/Paddle/pull/58640),[#58822](https://github.com/PaddlePaddle/Paddle/pull/58822),[#59055](https://github.com/PaddlePaddle/Paddle/pull/59055),[#59303](https://github.com/PaddlePaddle/Paddle/pull/59303),[#59487](https://github.com/PaddlePaddle/Paddle/pull/59487),[#58400](https://github.com/PaddlePaddle/Paddle/pull/58400),[#59283](https://github.com/PaddlePaddle/Paddle/pull/59283),[#54791](https://github.com/PaddlePaddle/Paddle/pull/54791),[#59134](https://github.com/PaddlePaddle/Paddle/pull/59134),[#56206](https://github.com/PaddlePaddle/Paddle/pull/56206),[#56199](https://github.com/PaddlePaddle/Paddle/pull/56199),[#56670](https://github.com/PaddlePaddle/Paddle/pull/56670),[#58923](https://github.com/PaddlePaddle/Paddle/pull/58923) - 修复 Paddle ARM 编译相关问题。[#55416](https://github.com/PaddlePaddle/Paddle/pull/55416),[#55548](https://github.com/PaddlePaddle/Paddle/pull/55548) # Thanks to Our Contributors Azure-Tang, zhaoyinglia, From00, JZ-LIANG, xysheng-baidu, SylarTiaNII, kuizhiqing, zhiqiu, FeixLiu, liuzhenhai93, GhostScreaming, pangengzheng, xiaoyewww, wanghuancoder, ForFishes, hitywt, danleifeng, tianshuo78520a, ykkk2333, houj04, lj970926, XiaociZhang, HarperCy, cqulilujia, runzhech, RuohengMa, Caozhou1995, kangguangli, heavyrain-lzy, zyfncg, SigureMo, YuanRisheng, lchdl, LiYuRio, AndSonder, Wennie396, zhangbo9674, liudongxue01, risemeup1, phlrain, winter-wang, yuanlehome, NALLEIN, Liujie0926, yuguo-Jack, gitliuyf, zh794390558, Aurelius84, 6clc, GGBond8488, xiaoguoguo626807, Wong4j, iosmers, xiaoxiaohehe001, LielinJiang, carryyu, Difers, yangxiaoyu14, xuxinyi389, cxxly, gongshaotian, jjyaoao, lijialin03, lxd-cumt, cyber-pioneer, HydrogenSulfate, MayYouBeProsperous, Charles-hit, Patrick-Star125, ScottWong98, huangjiyi, DrRyanHuang, jinyouzhi, BeingGod, Wanglongzhi2001, yangguohao, zyt1024, longranger2, 2742195759, megemini, thisjiang, kevincheng2, zhoutianzi666, Wangzheee, ming1753, tianhaodongbd, freeliuzc, zhenyun-li, MARD1NO, RichardWooSJTU, eee4017, leo0519, csy0225, wwbitejotunn, bukejiyu, jiweibo, iamsonderr, ckl117, ronny1996, zhanglirong1999, LLee233, ZHUI, wangxn12138, zhwesky2010, Courtesy-Xs, zoooo0820, llyyxx0413, Asthestarsfalll, zxcd, pkuzyc, idontkonwher, sneaxiy, hong19860320, ZibinGuo, leolishaohao, MuShangCC, zhupengyang, shentanyue, Travis-Lee, wz1qqx, frank-oops, newway, QingshuChen, zhangyk0314, HandSomeLEEw, Shixiaowei02, zhangyuqin1998, Xing-lil, zhhsplendid, jiahy0825, xinyu-intel, MarioLulab, 0x45f, Tom-Zheng, xingmingyyj, zhangbopd, gouzil, zeroRains, BiynXu, WintersMontagne10335, wuhuachaocoding, GreatV, chenwhql, deepllz, parap1uie-s, ozogxyz, FisherWY, changeyoung98, zhiboniu, YangQun1 dynamicheart, Xreki, liugddx, Lylinnnnn, YSF-A, zzjjay, YanhuiDua, lishicheng1996, USTCKAY, abenmao, cocoshe, HermitSun, ccsuzzh, sanbuphy, enkilee, RedContritio, Liyulingyue, zrr1999, chen2016013, Galaxy1458, chalsliu, mrcangye, XieYunshen, zhiheng-liu, haohongxiang, ZzSean, JamesLim-sy, yuehuayingxueluo, niuliling123, umiswing, sijunhe, littsk, SecretXV, zhurou603, zhangjun, caizejun, yangjianfengo1, vivienfanghuagood, Xinyu302, lizexu123, yghstill, Li-fAngyU, VigiZhang, co63oc, dhanush-2501, ooooo-create, PommesPeter, zeus2x7, akshatvishu, jzhang533, Sekiro-x, gumblex, BernieHuang2008, YibinLiu666, qiuwenbogdut, XavierZXY, MqLeet, zhangting2020, mingxu1067, Ainavo, SSKlearns, yuchen202, silverling, zade23, wenxiaohahaha, NKNaN, Tsaiyue, fsczz, Tomoko-hjf, rhmaaa, zbt78, Hhankyangg, wangzhen38, zhengqiwen1997, engineer1109, onepick, qili93, Rane2021, nemonameless, DesmonDay, RachelXu7, ceci3, lyuwenyu, liuruyan, LokeZhou, shiyutang, lanxianghit, feifei-111, Sahala08, sunzhongkai588, Kaedeharai, Candy2Tang, liyongchao911, whisky-12, InsaneOnion, yoyoIcy, KongAKun, linzeyang, MuhammadNizamani, eltociear, Ligoml, LUZY0726, Windfarer, FlyingQianMM, jeng1220, junelotus, zlsh80826, Vvsmile, Frida-a, TonibMw, guoshengCS, zhink, ZhangYulongg, AlbertVan, fengxin-hello, mjp9527, entired, DanGuge.

PaddlePaddle 2.5.0 Release Notev2.5.0

? · 2023-07-25

# PaddlePaddle 2.5.0 Release Note ## 1. 重要更新 - **动静统一新架构**：实现基础算子组合的动转静加编译器执行新模式，在ResNet50&Bert模型上完成动转静、组合算子、神经网络编译器优化加速全流程。动转静完成子图fallback核心功能开发，支持动转静失败时回退到动态图训练执行；组合算子设计一套包含150多个基础算子的基础算子体系，实现python层前向算子拆分机制和支持动、静态图的反向算子拆分机制，实现70多个常用前、反向算子的拆分；CINN编译器修复正确性问题，开发关键Pass，添加手工Schedule规则，实现内核代码自动生成，ResNet50模型性能提升12%，Bert模型性能提升10%。 - **PHI算子库算子架构统一**：将原算子体系下剩余的350+算子内核全部统一到PHI算子库中，以及原算子体系中的算子定义方式也都统一为PHI算子库的算子定义形式（基于YAML配置定义算子），提升了架构统一性，降低了框架开发的理解成本；将PHI算子库依赖的Fluid头文件全部解耦，并独立编译为动态链接库，为框架的二次开发提供更轻量的算子库复用方式；继续对飞桨框架中不规范的算子以及算子内核进行规范化调整，便于开发者理解，降低了硬件的接入成本。 - **静态图新执行器全面上线**：静态图新执行器实现多项功能和性能优化，完成对原有多套旧执行器的统一和替换，成为静态图单卡和分布式训练python端入口以及动转静、控制流、CINN等后端默认使用的执行引擎，大幅提升框架调度性能，功能架构更加清晰，二次开发能力显著增强。 - **Python API 支持0维tensor**：为形状为`[1,]` 及形状为 `[]` 的张量定义了清晰的语义。 - **新的环境适配**：适配了python 3.11 与CUDA 12。 ## 2. 不兼容升级 - 飞桨API支持0维tensor。飞桨之前用shape为[1]的1维tensor来替代0维tensor，这种替代方式和当前主流习惯有差异，增加模型的开发调试成本，有时还会导致非预期错误。本版本对需支持0维tensor的376个API进行了修正，和社区广泛使用的工具如EinOps等实现。例如，在之前的情况下，模型训练中输出的loss为1维tensor，如果要取出或打印loss，往往需要使用 `loss.numpy()[0]` 这样的代码。经过本次修改后，模型训练中输出的loss为0维tensor，使用 `loss.numpy()` 即可取出或打印loss，代码简短、易懂且符合业界使用习惯。 - `paddle.fluid` API全面退场。按照上个版本已预告的计划，本次退场了1116个`paddle.fluid`API及相关内部接口，剩余少量相关内部接口会在下个版本全部清理完成。fluid API属于飞桨2.0本计划移除但考虑到兼容性等因素延缓清理的历史API，本次退场清理不会影响基于飞桨2.0开发的程序，飞桨API体系也会更加简洁易懂。 - 旧版动态图Python端代码完成清理。至此，Python端仅使用新版动态图调用C++核心逻辑。 - 为统一静态图模型数据并行的训练方式，废弃原有的单进程多卡训练方式，包括 `paddle.static.ParallelExecutor` 和 `paddle.static.CompiledProgram().with_data_parallel()` 两个接口，原因是这套接口只支持单机多卡，不支持多机多卡，且底层执行性能较差。推荐统一使用多进程多卡训练方式，即 `paddle.distributed.launch` 接口来进行数据并行的分布式训练。该升级只影响静态图，不影响动态图和动转静训练，如果使用了废弃接口，请参考 [数据并行](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/guides/06_distributed_training/cluster_quick_start_collective_cn.html) 的文档修改模型代码。[#50351](https://github.com/PaddlePaddle/Paddle/pull/50351)，[#50501](https://github.com/PaddlePaddle/Paddle/pull/50501)，[#51240](https://github.com/PaddlePaddle/Paddle/pull/51240)，[#51701](https://github.com/PaddlePaddle/Paddle/pull/51701)，[#51616](https://github.com/PaddlePaddle/Paddle/pull/51616)，[#51369](https://github.com/PaddlePaddle/Paddle/pull/51369)，[#52671](https://github.com/PaddlePaddle/Paddle/pull/52671) - 移除框架中原有的昇腾NPU和寒武纪MLU的适配代码，全部升级为CustomDevice插件式适配方式，并将昇腾NPU和寒武纪MLU的适配代码迁移至PaddleCustomDevice仓库。 ## 3. 训练框架（含分布式） ### Python API #### API 支持0维tensor - API输入支持0维tensor，涉及 `paddle.reshape`、`paddle.trace`、`paddle.linalg.norm` 等286个API。[#53208](https://github.com/PaddlePaddle/Paddle/pull/53208), [#53592](https://github.com/PaddlePaddle/Paddle/pull/53592), [#47074](https://github.com/PaddlePaddle/Paddle/pull/47074), [#53186](https://github.com/PaddlePaddle/Paddle/pull/53186), [#47677](https://github.com/PaddlePaddle/Paddle/pull/47677), [#49357](https://github.com/PaddlePaddle/Paddle/pull/49357), [#50237](https://github.com/PaddlePaddle/Paddle/pull/50237), [#46555](https://github.com/PaddlePaddle/Paddle/pull/46555), [#47219](https://github.com/PaddlePaddle/Paddle/pull/47219), [#47501](https://github.com/PaddlePaddle/Paddle/pull/47501), [#47858](https://github.com/PaddlePaddle/Paddle/pull/47858), [#47961](https://github.com/PaddlePaddle/Paddle/pull/47961), [#48058](https://github.com/PaddlePaddle/Paddle/pull/48058), [#48007](https://github.com/PaddlePaddle/Paddle/pull/48007), [#49755](https://github.com/PaddlePaddle/Paddle/pull/49755), [#51024](https://github.com/PaddlePaddle/Paddle/pull/51024), [#51566](https://github.com/PaddlePaddle/Paddle/pull/51566), [#51899](https://github.com/PaddlePaddle/Paddle/pull/51899), [#49813](https://github.com/PaddlePaddle/Paddle/pull/49813), [#47812](https://github.com/PaddlePaddle/Paddle/pull/47812), [#47849](https://github.com/PaddlePaddle/Paddle/pull/47849), [#47251](https://github.com/PaddlePaddle/Paddle/pull/47251), [#53125](https://github.com/PaddlePaddle/Paddle/pull/53125), [#53828](https://github.com/PaddlePaddle/Paddle/pull/53828), [#51265](https://github.com/PaddlePaddle/Paddle/pull/51265), [#47689](https://github.com/PaddlePaddle/Paddle/pull/47689), [#48452](https://github.com/PaddlePaddle/Paddle/pull/48452), [#49072](https://github.com/PaddlePaddle/Paddle/pull/49072), [#48638](https://github.com/PaddlePaddle/Paddle/pull/48638), [#49175](https://github.com/PaddlePaddle/Paddle/pull/49175), [#49279](https://github.com/PaddlePaddle/Paddle/pull/49279), [#50857](https://github.com/PaddlePaddle/Paddle/pull/50857), [#49805](https://github.com/PaddlePaddle/Paddle/pull/49805), [#47734](https://github.com/PaddlePaddle/Paddle/pull/47734), [#45992](https://github.com/PaddlePaddle/Paddle/pull/45992), [#49616](https://github.com/PaddlePaddle/Paddle/pull/49616), [#49959](https://github.com/PaddlePaddle/Paddle/pull/49959), [#50536](https://github.com/PaddlePaddle/Paddle/pull/50536), [#49544](https://github.com/PaddlePaddle/Paddle/pull/49544), [#49842](https://github.com/PaddlePaddle/Paddle/pull/49842), [#46909](https://github.com/PaddlePaddle/Paddle/pull/46909), [#49361](https://github.com/PaddlePaddle/Paddle/pull/49361), [#50169](https://github.com/PaddlePaddle/Paddle/pull/50169), [#48314](https://github.com/PaddlePaddle/Paddle/pull/48314), [#48735](https://github.com/PaddlePaddle/Paddle/pull/48735), [#49122](https://github.com/PaddlePaddle/Paddle/pull/49122), [#49122](https://github.com/PaddlePaddle/Paddle/pull/49122), [#49177](https://github.com/PaddlePaddle/Paddle/pull/49177), [#49501](https://github.com/PaddlePaddle/Paddle/pull/49501), [#49562](https://github.com/PaddlePaddle/Paddle/pull/49562), [#49340](https://github.com/PaddlePaddle/Paddle/pull/49340), [#49550](https://github.com/PaddlePaddle/Paddle/pull/49550), [#49596](https://github.com/PaddlePaddle/Paddle/pull/49596), [#49730](https://github.com/PaddlePaddle/Paddle/pull/49730), [#49667](https://github.com/PaddlePaddle/Paddle/pull/49667), [#49692](https://github.com/PaddlePaddle/Paddle/pull/49692), [#49854](https://github.com/PaddlePaddle/Paddle/pull/49854), [#49845](https://github.com/PaddlePaddle/Paddle/pull/49845), [#49803](https://github.com/PaddlePaddle/Paddle/pull/49803), [#49889](https://github.com/PaddlePaddle/Paddle/pull/49889), [#49904](https://github.com/PaddlePaddle/Paddle/pull/49904), [#49518](https://github.com/PaddlePaddle/Paddle/pull/49518), [#49884](https://github.com/PaddlePaddle/Paddle/pull/49884), [#49880](https://github.com/PaddlePaddle/Paddle/pull/49880), [#49862](https://github.com/PaddlePaddle/Paddle/pull/49862), [#49921](https://github.com/PaddlePaddle/Paddle/pull/49921), [#49260](https://github.com/PaddlePaddle/Paddle/pull/49260), [#49929](https://github.com/PaddlePaddle/Paddle/pull/49929), [#49570](https://github.com/PaddlePaddle/Paddle/pull/49570), [#49882](https://github.com/PaddlePaddle/Paddle/pull/49882), [#50213](https://github.com/PaddlePaddle/Paddle/pull/50213), [#49780](https://github.com/PaddlePaddle/Paddle/pull/49780), [#50271](https://github.com/PaddlePaddle/Paddle/pull/50271), [#50289](https://github.com/PaddlePaddle/Paddle/pull/50289), [#50293](https://github.com/PaddlePaddle/Paddle/pull/50293), [#49735](https://github.com/PaddlePaddle/Paddle/pull/49735), [#50433](https://github.com/PaddlePaddle/Paddle/pull/50433), [#49847](https://github.com/PaddlePaddle/Paddle/pull/49847), [#50635](https://github.com/PaddlePaddle/Paddle/pull/50635), [#50950](https://github.com/PaddlePaddle/Paddle/pull/50950), [#50947](https://github.com/PaddlePaddle/Paddle/pull/50947), [#49460](https://github.com/PaddlePaddle/Paddle/pull/49460), [#53087](https://github.com/PaddlePaddle/Paddle/pull/53087), [#51687](https://github.com/PaddlePaddle/Paddle/pull/51687), [#52185](https://github.com/PaddlePaddle/Paddle/pull/52185), [#54649](https://github.com/PaddlePaddle/Paddle/pull/54649) - API输出支持0维tensor，涉及 `paddle.sum`、`paddle.min/max`、`paddle.any/all` 等90个API。[#52891](https://github.com/PaddlePaddle/Paddle/pull/52891), [#52861](https://github.com/PaddlePaddle/Paddle/pull/52861), [#52775](https://github.com/PaddlePaddle/Paddle/pull/52775), [#52850](https://github.com/PaddlePaddle/Paddle/pull/52850), [#52843](https://github.com/PaddlePaddle/Paddle/pull/52843), [#52857](https://github.com/PaddlePaddle/Paddle/pull/52857), [#51721](https://github.com/PaddlePaddle/Paddle/pull/51721), [#53051](https://github.com/PaddlePaddle/Paddle/pull/53051), [#53192](https://github.com/PaddlePaddle/Paddle/pull/53192), [#52739](https://github.com/PaddlePaddle/Paddle/pull/52739), [#52741](https://github.com/PaddlePaddle/Paddle/pull/52741), [#53175](https://github.com/PaddlePaddle/Paddle/pull/53175), [#51889](https://github.com/PaddlePaddle/Paddle/pull/51889), [#53199](https://github.com/PaddlePaddle/Paddle/pull/53199), [#53242](https://github.com/PaddlePaddle/Paddle/pull/53242), [#53421](https://github.com/PaddlePaddle/Paddle/pull/53421) - 支持0维tensor后，修正原有不规范的代码，及对模型代码中的非规范用法进行提示和兼容。[#51562](https://github.com/PaddlePaddle/Paddle/pull/51562), [#51586](https://github.com/PaddlePaddle/Paddle/pull/51586), [#51757](https://github.com/PaddlePaddle/Paddle/pull/51757), [#52197](https://github.com/PaddlePaddle/Paddle/pull/52197), [#54117](https://github.com/PaddlePaddle/Paddle/pull/54117)。 #### new API - 新增 jacobian 和 hessian API，用于科学计算。[#53331](https://github.com/PaddlePaddle/Paddle/pull/53331) - 新增稀疏计算API。例如 `paddle.sparse.reshape`、`paddle.sparse.sum` 和 `paddle.sparse.slice` 等。[#46694](https://github.com/PaddlePaddle/Paddle/pull/46694), [#51513](https://github.com/PaddlePaddle/Paddle/pull/51513), [#53794](https://github.com/PaddlePaddle/Paddle/pull/53794), [#51406](https://github.com/PaddlePaddle/Paddle/pull/51406) - 新增其它API。例如 `paddle.optimizer.LBFGS`、`paddle.index_put` 和 `paddle.logaddexp` 等。[#53314](https://github.com/PaddlePaddle/Paddle/pull/53314), [#51912](https://github.com/PaddlePaddle/Paddle/pull/51912), [#52886](https://github.com/PaddlePaddle/Paddle/pull/52886), [#50843](https://github.com/PaddlePaddle/Paddle/pull/50843), [#47282](https://github.com/PaddlePaddle/Paddle/pull/47282), [#52284](https://github.com/PaddlePaddle/Paddle/pull/52284) ### 动态图 #### 新功能 - 新增了paddle.nn.utils.clip_grad_norm_用于支持梯度裁剪和paddle.Tensor.data_ptr用于获取Tensor数据的内存/显存的地址 [PR49935](https://github.com/PaddlePaddle/Paddle/pull/49935)[, PR48235](https://github.com/PaddlePaddle/Paddle/pull/48235), [PR49173](https://github.com/PaddlePaddle/Paddle/pull/49173) - 新增了saved_tensors_hooks机制，用于临时存放和取回用于反向计算使用的前向Tensor。 [PR45763](https://github.com/PaddlePaddle/Paddle/pull/45763), [PR46215](https://github.com/PaddlePaddle/Paddle/pull/46215), [PR48124](https://github.com/PaddlePaddle/Paddle/pull/48124) - Tensor支持了pickler，用于支持Tensor的序列化。 [PR47025](https://github.com/PaddlePaddle/Paddle/pull/47025), [PR48179](https://github.com/PaddlePaddle/Paddle/pull/48179) - 新增了调试日志，反向出现nan/inf时打印前向Python堆栈 [PR53217](https://github.com/PaddlePaddle/Paddle/pull/53217) [PR52639](https://github.com/PaddlePaddle/Paddle/pull/52639) [PR52729](https://github.com/PaddlePaddle/Paddle/pull/52729) - 新增了对expand_v2, tile, concat, assign, slice高阶微分的支持。[PR45941](https://github.com/PaddlePaddle/Paddle/pull/45941)[, PR45942](https://github.com/PaddlePaddle/Paddle/pull/45942)[, PR45940](https://github.com/PaddlePaddle/Paddle/pull/45940)[, PR45879](https://github.com/PaddlePaddle/Paddle/pull/45879), [PR45960](https://github.com/PaddlePaddle/Paddle/pull/45960) #### 功能优化 - 优化了动态图的日志打印，包括日志内容优化、VLog级别优化、报错内容优化等。[PR45783](https://github.com/PaddlePaddle/Paddle/pull/45783), [PR46349](https://github.com/PaddlePaddle/Paddle/pull/46349), [PR46934](https://github.com/PaddlePaddle/Paddle/pull/46934), [PR47724](https://github.com/PaddlePaddle/Paddle/pull/47724) - 新增了FLAGS_auto_growth_chunk_size_in_mb用于auto_growth_allocator最小chunk size的设置 [PR52204](https://github.com/PaddlePaddle/Paddle/pull/52204) #### bug fix - 修复了一些算子的bug，包括：batch_norm, slice, set_value, scale, multinomial, adam, conv, transpose2_grad, conv2d_transpose_double_grad。[PR47802](https://github.com/PaddlePaddle/Paddle/pull/47802), [PR47634](https://github.com/PaddlePaddle/Paddle/pull/47634), [PR47349](https://github.com/PaddlePaddle/Paddle/pull/47349), [PR46124](https://github.com/PaddlePaddle/Paddle/pull/46124), [PR46147](https://github.com/PaddlePaddle/Paddle/pull/46147), [PR50388](https://github.com/PaddlePaddle/Paddle/pull/50388), [PR48626](https://github.com/PaddlePaddle/Paddle/pull/48626), [PR48519](https://github.com/PaddlePaddle/Paddle/pull/48519), [PR50386](https://github.com/PaddlePaddle/Paddle/pull/50386), [PR48432](https://github.com/PaddlePaddle/Paddle/pull/48432), [PR51851](https://github.com/PaddlePaddle/Paddle/pull/51851) - 修复了PyLayer的一些错误问题。[PR51740](https://github.com/PaddlePaddle/Paddle/pull/51740), [PR47154](https://github.com/PaddlePaddle/Paddle/pull/47154), [PR47323](https://github.com/PaddlePaddle/Paddle/pull/47323), [PR54041](https://github.com/PaddlePaddle/Paddle/pull/54041), [PR48533](https://github.com/PaddlePaddle/Paddle/pull/48533) - 确保sync_batch_norm在反向有序，防止错序导致hang或精度错误。[PR52268](https://github.com/PaddlePaddle/Paddle/pull/52268), [PR52860](https://github.com/PaddlePaddle/Paddle/pull/52860), [PR52779](https://github.com/PaddlePaddle/Paddle/pull/52779) - 修复了linspace在AMP下的bug。[PR46088](https://github.com/PaddlePaddle/Paddle/pull/46088) - 修复了Python C API错误调用导致Windows崩溃的问题。[PR46833](https://github.com/PaddlePaddle/Paddle/pull/46833) - 修复了DataLoader可能遗漏删除/dev/shm的问题。[PR48511](https://github.com/PaddlePaddle/Paddle/pull/48511) - 修复了paddle.grad的一些问题。[PR47151](https://github.com/PaddlePaddle/Paddle/pull/47151) - 为不支持高阶微分的算子添加报错信息。[PR47231](https://github.com/PaddlePaddle/Paddle/pull/47231) - 为python运算符添加numpyarray的支持。[PR48229](https://github.com/PaddlePaddle/Paddle/pull/48229) - 有两处element_size 接口，删除其中之一。[PR49631](https://github.com/PaddlePaddle/Paddle/pull/49631) - 修复老动态图开VLOG崩溃问题。[PR47115](https://github.com/PaddlePaddle/Paddle/pull/47115) - XPU，d2d时，改成d2h+h2d，规避多线程问题。[PR48373](https://github.com/PaddlePaddle/Paddle/pull/48373) #### 性能优化 - Python运算符下沉到C++实现，以提升API性能, 下沉后该类API有3~6倍性能提升。[PR45811](https://github.com/PaddlePaddle/Paddle/pull/45811), [PR46326](https://github.com/PaddlePaddle/Paddle/pull/46326), [PR46329](https://github.com/PaddlePaddle/Paddle/pull/46329), [PR46520](https://github.com/PaddlePaddle/Paddle/pull/46520), [PR46542](https://github.com/PaddlePaddle/Paddle/pull/46542), [PR46565](https://github.com/PaddlePaddle/Paddle/pull/46565), [PR47060](https://github.com/PaddlePaddle/Paddle/pull/47060), [PR47077](https://github.com/PaddlePaddle/Paddle/pull/47077), [PR47174](https://github.com/PaddlePaddle/Paddle/pull/47174), [PR47315](https://github.com/PaddlePaddle/Paddle/pull/47315) - 优化了Optimizer CPU调度性能，可减少Optimizer阶段导致的GPU Gap。 [PR49787](https://github.com/PaddlePaddle/Paddle/pull/49787), [PR50188](https://github.com/PaddlePaddle/Paddle/pull/50188)[, PR51340](https://github.com/PaddlePaddle/Paddle/pull/51340), [PR49864](https://github.com/PaddlePaddle/Paddle/pull/49864), [PR50158](https://github.com/PaddlePaddle/Paddle/pull/50158), [PR50335](https://github.com/PaddlePaddle/Paddle/pull/50335) - API中可下沉到C++的逻辑，下沉到C++，以提升API性能。[PR46412](https://github.com/PaddlePaddle/Paddle/pull/46412), [PR46190](https://github.com/PaddlePaddle/Paddle/pull/46190) - 优化动态图下Python端不必要的调用逻辑，以提升API性能。[PR46221](https://github.com/PaddlePaddle/Paddle/pull/46221), [PR49473](https://github.com/PaddlePaddle/Paddle/pull/49473), [PR49574](https://github.com/PaddlePaddle/Paddle/pull/49574), [PR49589](https://github.com/PaddlePaddle/Paddle/pull/49589), [PR49612](https://github.com/PaddlePaddle/Paddle/pull/49612), [PR49717](https://github.com/PaddlePaddle/Paddle/pull/49717)[, PR49733](https://github.com/PaddlePaddle/Paddle/pull/49733), [PR49823](https://github.com/PaddlePaddle/Paddle/pull/49823)[, PR49508](https://github.com/PaddlePaddle/Paddle/pull/49508), [PR46840](https://github.com/PaddlePaddle/Paddle/pull/46840) - 优化了Allocator的使用，以提升动态图API调度性能。[PR47125](https://github.com/PaddlePaddle/Paddle/pull/47125), [PR48548](https://github.com/PaddlePaddle/Paddle/pull/48548), [PR50995](https://github.com/PaddlePaddle/Paddle/pull/50995), [PR47731](https://github.com/PaddlePaddle/Paddle/pull/47731) - 优化了fused_attention算子性能。[PR48902](https://github.com/PaddlePaddle/Paddle/pull/48902) - optimizer的_add_accumulator，如果device是CPU，且在动态图下，直接使用full初始化var。[PR48189](https://github.com/PaddlePaddle/Paddle/pull/48189) - 对反向图不必要执行的subgraph进行剪枝以提升性能。[PR47827](https://github.com/PaddlePaddle/Paddle/pull/47827) - 优化了initalizers的性能。[PR46033](https://github.com/PaddlePaddle/Paddle/pull/46033) - 新增fused dropout add算子，提升dropout 和 add 一起计算的性能。[#52903](https://github.com/PaddlePaddle/Paddle/pull/52903) ### 静态图 #### 静态图新执行器全面上线静态图新执行器实现多项功能和性能优化，完成对原有多套旧执行器的统一和替换，成为静态图单卡和分布式训练python端入口以及动转静、控制流、CINN等后端默认使用的执行引擎，大幅提升框架调度性能，功能架构更加清晰，二次开发能力显著增强。[#45913](https://github.com/PaddlePaddle/Paddle/pull/45913)，[#46025](https://github.com/PaddlePaddle/Paddle/pull/46025)，[#48911](https://github.com/PaddlePaddle/Paddle/pull/48911)，[#50239](https://github.com/PaddlePaddle/Paddle/pull/50239)，[#45696](https://github.com/PaddlePaddle/Paddle/pull/45696)，[#46092](https://github.com/PaddlePaddle/Paddle/pull/46092)，[#48158](https://github.com/PaddlePaddle/Paddle/pull/48158),[#51389](https://github.com/PaddlePaddle/Paddle/pull/51389)，[#49708](https://github.com/PaddlePaddle/Paddle/pull/49708)，[#49275](https://github.com/PaddlePaddle/Paddle/pull/49275),[#48789](https://github.com/PaddlePaddle/Paddle/pull/48789)，[#49939](https://github.com/PaddlePaddle/Paddle/pull/49939)，[#51149](https://github.com/PaddlePaddle/Paddle/pull/51149)，[#52652](https://github.com/PaddlePaddle/Paddle/pull/52652) ### 算子库 #### 自定义算子等功能增强包括：全新支持了自定义扩展机制，实现将 C++ 扩展的运算函数绑定至Python端使用，进一步提升了框架的二次开发能力；扩展支持自定义硬件上使用自定义算子机制，以满足硬件厂商实现非Paddle已有算子的需求；扩展支持了在自定义算子中实现`inplace`、`vector<Tensor>`输出、`optional<Tnesor>`输入等高阶机制；优化了自定义算子在动态图模式下的调度性能，多输入参数的算子性能提升 25.4%；为自定义算子Tensor扩展新增了常用运算符及API，支持链式调用，简化代码写法。对算子内核选择机制进行了优化；对部分算子内核进行了逻辑完善、支持数据类型增强以及性能优化；新增以及完善 XPU 内核 100+；修复各项 Bug 累计 170+。 [#49222](https://github.com/PaddlePaddle/Paddle/pull/49222), [#51773](https://github.com/PaddlePaddle/Paddle/pull/51773), [#51923](https://github.com/PaddlePaddle/Paddle/pull/51923), [#53080](https://github.com/PaddlePaddle/Paddle/pull/53080), [#50731](https://github.com/PaddlePaddle/Paddle/pull/50731), [#50563](https://github.com/PaddlePaddle/Paddle/pull/50563), [#50840](https://github.com/PaddlePaddle/Paddle/pull/50840), [#50983](https://github.com/PaddlePaddle/Paddle/pull/50983), [#51713](https://github.com/PaddlePaddle/Paddle/pull/51713), [#48733](https://github.com/PaddlePaddle/Paddle/pull/48733), [#50558](https://github.com/PaddlePaddle/Paddle/pull/50558), [#50764](https://github.com/PaddlePaddle/Paddle/pull/50764), [#51973](https://github.com/PaddlePaddle/Paddle/pull/51973), [#52216](https://github.com/PaddlePaddle/Paddle/pull/52216), [#51027](https://github.com/PaddlePaddle/Paddle/pull/51027), [#50745](https://github.com/PaddlePaddle/Paddle/pull/50745), [#50756](https://github.com/PaddlePaddle/Paddle/pull/50756), [#50886](https://github.com/PaddlePaddle/Paddle/pull/50886), [#50813](https://github.com/PaddlePaddle/Paddle/pull/50813), [#50869](https://github.com/PaddlePaddle/Paddle/pull/50869), [#51085](https://github.com/PaddlePaddle/Paddle/pull/51085), [#51646](https://github.com/PaddlePaddle/Paddle/pull/51646), [#51620](https://github.com/PaddlePaddle/Paddle/pull/51620), [#51844](https://github.com/PaddlePaddle/Paddle/pull/51844), [#52421](https://github.com/PaddlePaddle/Paddle/pull/52421), [#52872](https://github.com/PaddlePaddle/Paddle/pull/52872), [#52597](https://github.com/PaddlePaddle/Paddle/pull/52597), [#50582](https://github.com/PaddlePaddle/Paddle/pull/50582), [#52114](https://github.com/PaddlePaddle/Paddle/pull/52114), [#52915](https://github.com/PaddlePaddle/Paddle/pull/52915), [#50928](https://github.com/PaddlePaddle/Paddle/pull/50928), [#48272](https://github.com/PaddlePaddle/Paddle/pull/48272), [#48702](https://github.com/PaddlePaddle/Paddle/pull/48702), [#52191](https://github.com/PaddlePaddle/Paddle/pull/52191), [#52191](https://github.com/PaddlePaddle/Paddle/pull/52191), [#47374](https://github.com/PaddlePaddle/Paddle/pull/47374), [#47375](https://github.com/PaddlePaddle/Paddle/pull/47375), [#47378](https://github.com/PaddlePaddle/Paddle/pull/47378), [#54126](https://github.com/PaddlePaddle/Paddle/pull/54126), [#47638](https://github.com/PaddlePaddle/Paddle/pull/47638), [#47661](https://github.com/PaddlePaddle/Paddle/pull/47661), [#50606](https://github.com/PaddlePaddle/Paddle/pull/50606), [#53528](https://github.com/PaddlePaddle/Paddle/pull/53528), [#50599](https://github.com/PaddlePaddle/Paddle/pull/50599), [#51727](https://github.com/PaddlePaddle/Paddle/pull/51727), [#50825](https://github.com/PaddlePaddle/Paddle/pull/50825), [#50773](https://github.com/PaddlePaddle/Paddle/pull/50773), [#50979](https://github.com/PaddlePaddle/Paddle/pull/50979), [#53336](https://github.com/PaddlePaddle/Paddle/pull/53336), [#53555](https://github.com/PaddlePaddle/Paddle/pull/53555), [#53716](https://github.com/PaddlePaddle/Paddle/pull/53716), [#53753](https://github.com/PaddlePaddle/Paddle/pull/53753), [#53981](https://github.com/PaddlePaddle/Paddle/pull/53981), [#53977](https://github.com/PaddlePaddle/Paddle/pull/53977), [#53980](https://github.com/PaddlePaddle/Paddle/pull/53980), [#54043](https://github.com/PaddlePaddle/Paddle/pull/54043), [#54066](https://github.com/PaddlePaddle/Paddle/pull/54066), [#52866](https://github.com/PaddlePaddle/Paddle/pull/52866), [#53043](https://github.com/PaddlePaddle/Paddle/pull/53043), [#53325](https://github.com/PaddlePaddle/Paddle/pull/53325), [#54323](https://github.com/PaddlePaddle/Paddle/pull/54323), [#54367](https://github.com/PaddlePaddle/Paddle/pull/54367), [#51353](https://github.com/PaddlePaddle/Paddle/pull/51353), [#53749](https://github.com/PaddlePaddle/Paddle/pull/53749), [#50013](https://github.com/PaddlePaddle/Paddle/pull/50013), [#47570](https://github.com/PaddlePaddle/Paddle/pull/47570), [#50997](https://github.com/PaddlePaddle/Paddle/pull/50997), [#51241](https://github.com/PaddlePaddle/Paddle/pull/51241), [#49537](https://github.com/PaddlePaddle/Paddle/pull/49537) #### 算子体系架构统一具体包括：将原算子体系下剩余的350+算子内核全部统一到PHI算子库中，以及原算子体系中的算子定义方式也都统一为PHI算子库的算子定义形式（基于YAML配置定义算子），提升了架构统一性，降低了框架开发的理解成本；将PHI算子库依赖的Fluid头文件全部解耦，并独立编译为动态链接库，为框架的二次开发提供更轻量的算子库复用方式；继续对飞桨框架中不规范的算子以及算子内核进行规范化调整，便于开发者理解，降低了硬件的接入成本。 [#47856](https://github.com/PaddlePaddle/Paddle/pull/47856), [#49328](https://github.com/PaddlePaddle/Paddle/pull/49328), [#49138](https://github.com/PaddlePaddle/Paddle/pull/49138), [#52014](https://github.com/PaddlePaddle/Paddle/pull/52014), [#52044](https://github.com/PaddlePaddle/Paddle/pull/52044), [#52116](https://github.com/PaddlePaddle/Paddle/pull/52116), [#52486](https://github.com/PaddlePaddle/Paddle/pull/52486), [#52101](https://github.com/PaddlePaddle/Paddle/pull/52101), [#52882](https://github.com/PaddlePaddle/Paddle/pull/52882), [#53003](https://github.com/PaddlePaddle/Paddle/pull/53003), [#53034](https://github.com/PaddlePaddle/Paddle/pull/53034), [#51914](https://github.com/PaddlePaddle/Paddle/pull/51914), [#49116](https://github.com/PaddlePaddle/Paddle/pull/49116), [#52626](https://github.com/PaddlePaddle/Paddle/pull/52626), [#52878](https://github.com/PaddlePaddle/Paddle/pull/52878), [#52879](https://github.com/PaddlePaddle/Paddle/pull/52879), [#52880](https://github.com/PaddlePaddle/Paddle/pull/52880), [#52875](https://github.com/PaddlePaddle/Paddle/pull/52875), [#51600](https://github.com/PaddlePaddle/Paddle/pull/51600), [#51601](https://github.com/PaddlePaddle/Paddle/pull/51601), [#51590](https://github.com/PaddlePaddle/Paddle/pull/51590), [#51887](https://github.com/PaddlePaddle/Paddle/pull/51887), [#51891](https://github.com/PaddlePaddle/Paddle/pull/51891), [#52036](https://github.com/PaddlePaddle/Paddle/pull/52036), [#52130](https://github.com/PaddlePaddle/Paddle/pull/52130), [#52134](https://github.com/PaddlePaddle/Paddle/pull/52134), [#51951](https://github.com/PaddlePaddle/Paddle/pull/51951), [#51886](https://github.com/PaddlePaddle/Paddle/pull/51886), [#52274](https://github.com/PaddlePaddle/Paddle/pull/52274), [#52263](https://github.com/PaddlePaddle/Paddle/pull/52263), [#51913](https://github.com/PaddlePaddle/Paddle/pull/51913), [#52145](https://github.com/PaddlePaddle/Paddle/pull/52145), [#52347](https://github.com/PaddlePaddle/Paddle/pull/52347), [#52370](https://github.com/PaddlePaddle/Paddle/pull/52370), [#52437](https://github.com/PaddlePaddle/Paddle/pull/52437), [#52424](https://github.com/PaddlePaddle/Paddle/pull/52424), [#52231](https://github.com/PaddlePaddle/Paddle/pull/52231), [#52522](https://github.com/PaddlePaddle/Paddle/pull/52522), [#52529](https://github.com/PaddlePaddle/Paddle/pull/52529), [#52802](https://github.com/PaddlePaddle/Paddle/pull/52802), [#52799](https://github.com/PaddlePaddle/Paddle/pull/52799), [#52855](https://github.com/PaddlePaddle/Paddle/pull/52855), [#52711](https://github.com/PaddlePaddle/Paddle/pull/52711), [#52940](https://github.com/PaddlePaddle/Paddle/pull/52940), [#53309](https://github.com/PaddlePaddle/Paddle/pull/53309), [#47817](https://github.com/PaddlePaddle/Paddle/pull/47817), [#48001](https://github.com/PaddlePaddle/Paddle/pull/48001), [#48063](https://github.com/PaddlePaddle/Paddle/pull/48063), [#48049](https://github.com/PaddlePaddle/Paddle/pull/48049), [#48168](https://github.com/PaddlePaddle/Paddle/pull/48168), [#48415](https://github.com/PaddlePaddle/Paddle/pull/48415), [#48696](https://github.com/PaddlePaddle/Paddle/pull/48696), [#48970](https://github.com/PaddlePaddle/Paddle/pull/48970), [#50183](https://github.com/PaddlePaddle/Paddle/pull/50183), [#50407](https://github.com/PaddlePaddle/Paddle/pull/50407), [#50498](https://github.com/PaddlePaddle/Paddle/pull/50498), [#50419](https://github.com/PaddlePaddle/Paddle/pull/50419), [#50282](https://github.com/PaddlePaddle/Paddle/pull/50282), [#50870](https://github.com/PaddlePaddle/Paddle/pull/50870), [#50911](https://github.com/PaddlePaddle/Paddle/pull/50911), [#50865](https://github.com/PaddlePaddle/Paddle/pull/50865), [#51288](https://github.com/PaddlePaddle/Paddle/pull/51288), [#53735](https://github.com/PaddlePaddle/Paddle/pull/53735), [#47248](https://github.com/PaddlePaddle/Paddle/pull/47248), [#47787](https://github.com/PaddlePaddle/Paddle/pull/47787), [#52202](https://github.com/PaddlePaddle/Paddle/pull/52202), [#47579](https://github.com/PaddlePaddle/Paddle/pull/47579), [#49444](https://github.com/PaddlePaddle/Paddle/pull/49444), [#45772](https://github.com/PaddlePaddle/Paddle/pull/45772), [#51264](https://github.com/PaddlePaddle/Paddle/pull/51264), [#51634](https://github.com/PaddlePaddle/Paddle/pull/51634), [#51631](https://github.com/PaddlePaddle/Paddle/pull/51631), [#47385](https://github.com/PaddlePaddle/Paddle/pull/47385), [#46342](https://github.com/PaddlePaddle/Paddle/pull/46342), [#47510](https://github.com/PaddlePaddle/Paddle/pull/47510), [#47532](https://github.com/PaddlePaddle/Paddle/pull/47532), [#47702](https://github.com/PaddlePaddle/Paddle/pull/47702), [#47860](https://github.com/PaddlePaddle/Paddle/pull/47860), [#49470](https://github.com/PaddlePaddle/Paddle/pull/49470), [#50358](https://github.com/PaddlePaddle/Paddle/pull/50358), [#49121](https://github.com/PaddlePaddle/Paddle/pull/49121), [#50190](https://github.com/PaddlePaddle/Paddle/pull/50190), [#52374](https://github.com/PaddlePaddle/Paddle/pull/52374), [#52372](https://github.com/PaddlePaddle/Paddle/pull/52372), [#52375](https://github.com/PaddlePaddle/Paddle/pull/52375), [#52371](https://github.com/PaddlePaddle/Paddle/pull/52371) ### 动转静加组合算子 #### 新功能 - 组合算子添加dropout, silu, stack, relu, expand, unsqueeze, pow, squeeze, meshgrid, batch_norm, layer_norm, group_norm, instance_norm, full_like, split, split_with_num, gelu, mean, flatten, rsqrt, hadswish算子的组合规则 [#50497](https://github.com/PaddlePaddle/Paddle/pull/50497), [#50838](https://github.com/PaddlePaddle/Paddle/pull/50838), [#50861](https://github.com/PaddlePaddle/Paddle/pull/50861), [#50819](https://github.com/PaddlePaddle/Paddle/pull/50819), [#50810](https://github.com/PaddlePaddle/Paddle/pull/50810), [#51527](https://github.com/PaddlePaddle/Paddle/pull/51527), [#51070](https://github.com/PaddlePaddle/Paddle/pull/51070), [#51539](https://github.com/PaddlePaddle/Paddle/pull/51539), [#51061](https://github.com/PaddlePaddle/Paddle/pull/51061), [#49894](https://github.com/PaddlePaddle/Paddle/pull/49894), [#50422](https://github.com/PaddlePaddle/Paddle/pull/50422), [#51874](https://github.com/PaddlePaddle/Paddle/pull/51874), [#51341](https://github.com/PaddlePaddle/Paddle/pull/51341), [#50295](https://github.com/PaddlePaddle/Paddle/pull/50295), [#50298](https://github.com/PaddlePaddle/Paddle/pull/50298), [#50672](https://github.com/PaddlePaddle/Paddle/pull/50672), [#51432](https://github.com/PaddlePaddle/Paddle/pull/51432), [#51003](https://github.com/PaddlePaddle/Paddle/pull/51003) - 组合算子添加gather_nd, reduce_max, group_norm, relu, reduce_max, gather, topk, sqrt, elementwise_pow, softmax, batch_norm, prod, multiply, expand, div, relu, slice, cumsum, sigmoid, layer_norm, sin, cos, roll, instance_norm, abs, assign, tile, scatter_nd_add, erf, floor, log, silu, leaky_relu, pad算子的vjp规则 [#50966](https://github.com/PaddlePaddle/Paddle/pull/50966), [#51653](https://github.com/PaddlePaddle/Paddle/pull/51653), [#52663](https://github.com/PaddlePaddle/Paddle/pull/52663), [#51742](https://github.com/PaddlePaddle/Paddle/pull/51742), [#52203](https://github.com/PaddlePaddle/Paddle/pull/52203), [#50794](https://github.com/PaddlePaddle/Paddle/pull/50794), [#50305](https://github.com/PaddlePaddle/Paddle/pull/50305), [#50786](https://github.com/PaddlePaddle/Paddle/pull/50786), [#50679](https://github.com/PaddlePaddle/Paddle/pull/50679), [#51045](https://github.com/PaddlePaddle/Paddle/pull/51045), [#51230](https://github.com/PaddlePaddle/Paddle/pull/51230), [#51474](https://github.com/PaddlePaddle/Paddle/pull/51474), [#51283](https://github.com/PaddlePaddle/Paddle/pull/51283), [#51238](https://github.com/PaddlePaddle/Paddle/pull/51238), [#49831](https://github.com/PaddlePaddle/Paddle/pull/49831), [#51838](https://github.com/PaddlePaddle/Paddle/pull/51838), [#50771](https://github.com/PaddlePaddle/Paddle/pull/50771), [#50565](https://github.com/PaddlePaddle/Paddle/pull/50565), [#51768](https://github.com/PaddlePaddle/Paddle/pull/51768), [#51750](https://github.com/PaddlePaddle/Paddle/pull/51750), [#51748](https://github.com/PaddlePaddle/Paddle/pull/51748), [#52532](https://github.com/PaddlePaddle/Paddle/pull/52532), [#52935](https://github.com/PaddlePaddle/Paddle/pull/52935), [#50963](https://github.com/PaddlePaddle/Paddle/pull/50963), [#51430](https://github.com/PaddlePaddle/Paddle/pull/51430), [#53141](https://github.com/PaddlePaddle/Paddle/pull/53141), [#52469](https://github.com/PaddlePaddle/Paddle/pull/52469), [#50436](https://github.com/PaddlePaddle/Paddle/pull/50436), [#51059](https://github.com/PaddlePaddle/Paddle/pull/51059), [#51296](https://github.com/PaddlePaddle/Paddle/pull/51296), [#52533](https://github.com/PaddlePaddle/Paddle/pull/52533), [#53374](https://github.com/PaddlePaddle/Paddle/pull/53374) - 组合算子添加matmul, tanh, elementwise二阶微分规则 [#50452](https://github.com/PaddlePaddle/Paddle/pull/50452), [#52192](https://github.com/PaddlePaddle/Paddle/pull/52192), [#53014](https://github.com/PaddlePaddle/Paddle/pull/53014) - 组合算子添加exp, reduce_mean, softmax, divide, cast, layer_norm, prod, meshgrid, expand_as, dropout, concat, gather_nd, elementwise_max, elementwise_pow, reduce_max组合算子bf16数据类型支持 [#54263](https://github.com/PaddlePaddle/Paddle/pull/54263)， [#54236](https://github.com/PaddlePaddle/Paddle/pull/54236), [#53865](https://github.com/PaddlePaddle/Paddle/pull/53865), [#54175](https://github.com/PaddlePaddle/Paddle/pull/54175), [#54399](https://github.com/PaddlePaddle/Paddle/pull/54399) - 动转静新增控制流中的容器添加赋值语义支持 [#51248](https://github.com/PaddlePaddle/Paddle/pull/51248) - 动转静新增全图回退功能，当动转静转换失败时，可全图回退到动态图方式执行; 回退机制增加set_eval_frame接口 [#50111](https://github.com/PaddlePaddle/Paddle/pull/50111), [#52006](https://github.com/PaddlePaddle/Paddle/pull/52006) - 动转静to_static支持算子组合机制；支持被to_static装饰下使用register_hook的场景； [#49836](https://github.com/PaddlePaddle/Paddle/pull/49836), [#52948](https://github.com/PaddlePaddle/Paddle/pull/52948), [#53572](https://github.com/PaddlePaddle/Paddle/pull/53572) - 动转静to_static接口增加backend参数，可以指定为 `CINN` 或者 None，当该参数指定为 `CINN` 时，将会使用 CINN 编译器来加速训练和推理 [#52596](https://github.com/PaddlePaddle/Paddle/pull/52596) - 新增primitive接口代码自动生成功能，根据ops.yaml和legacy_ops.yaml中的算子定义；自动生成primitive接口的代码；自动生成Tensor运算接口 [#50315](https://github.com/PaddlePaddle/Paddle/pull/50315), [#49654](https://github.com/PaddlePaddle/Paddle/pull/49654), [#50642](https://github.com/PaddlePaddle/Paddle/pull/50642) - 新增算子前向组合功能，通过注册前向算子的组合规则，实现将前向算子拆分成基础算子 [#49605](https://github.com/PaddlePaddle/Paddle/pull/49605) - 新增组合算子开关，可以在shell中通过设置环境变量，实现算子按照不同方式进行拆分 [#50309](https://github.com/PaddlePaddle/Paddle/pull/50309) - 添加`OpTest`新增组合测试功能，对算子精度进行保障；添加elementwise类基础算子单测；添加batch_norm的CINN单测 [#50509](https://github.com/PaddlePaddle/Paddle/pull/50509), [#50807](https://github.com/PaddlePaddle/Paddle/pull/50807), [#52815](https://github.com/PaddlePaddle/Paddle/pull/52815) #### 功能优化 - 添加组合算子支持FP16运算和AMP O1运算；添加softmax和layer_norm算子AMP逻辑 [#52397](https://github.com/PaddlePaddle/Paddle/pull/52397), [#52598](https://github.com/PaddlePaddle/Paddle/pull/52598), [#51473](https://github.com/PaddlePaddle/Paddle/pull/51473) - 简化组合算子batch_norm的组合规则和vjp规则 [#54012](https://github.com/PaddlePaddle/Paddle/pull/54012), [#51827](https://github.com/PaddlePaddle/Paddle/pull/51827), [#51933](https://github.com/PaddlePaddle/Paddle/pull/51933), - 组合算子优化组合规则，提升含scalar组合规则的性能；优化组合算子日志打印 [#51960](https://github.com/PaddlePaddle/Paddle/pull/51960), [#50160](https://github.com/PaddlePaddle/Paddle/pull/50160) - 组合算子支持jit.save接口；新增自定义VJP规则接口 [#52344](https://github.com/PaddlePaddle/Paddle/pull/52344), [#50885](https://github.com/PaddlePaddle/Paddle/pull/50885) - 组合算子gather_grad删除overwrite参数。 [#52707](https://github.com/PaddlePaddle/Paddle/pull/52707) - 动转静代码风格清理，报错信息优化，规范日志 [#48637](https://github.com/PaddlePaddle/Paddle/pull/48637), [#46128](https://github.com/PaddlePaddle/Paddle/pull/46128), [#52527](https://github.com/PaddlePaddle/Paddle/pull/52527), [#46800](https://github.com/PaddlePaddle/Paddle/pull/46800),[#46415](https://github.com/PaddlePaddle/Paddle/pull/46415) - 动转静通过调用append backward的方式获取`grad var name`以修复高阶梯度计算时的错误 [#53250](https://github.com/PaddlePaddle/Paddle/pull/53250) - 动转静功能升级，清理to_static的临时目录以加速代码转换；增强to_static自动略过内部接口；支持在程序使用to_static装饰器 [#47102](https://github.com/PaddlePaddle/Paddle/pull/47102), [#50596](https://github.com/PaddlePaddle/Paddle/pull/50596), [#45768](https://github.com/PaddlePaddle/Paddle/pull/45768) - 动转静优化`print`函数转换以支持在组网阶段打印 Tensor 参数；升级参数收集机制 [#48672](https://github.com/PaddlePaddle/Paddle/pull/48672), [#50336](https://github.com/PaddlePaddle/Paddle/pull/50336) #### bug fix - 组合算子修复cmake编译错误；修复cuda 12测试错误；修复若干算子如meshgird, expand_as, concat, conv, arrange等错误[#49643](https://github.com/PaddlePaddle/Paddle/pull/49643), [#54622](https://github.com/PaddlePaddle/Paddle/pull/54622), [#53951](https://github.com/PaddlePaddle/Paddle/pull/53951), [#53951](https://github.com/PaddlePaddle/Paddle/pull/53951), [#53350](https://github.com/PaddlePaddle/Paddle/pull/53350), [#51486](https://github.com/PaddlePaddle/Paddle/pull/51486), [#52764](https://github.com/PaddlePaddle/Paddle/pull/52764) - 组合算子修复若干rank=1, shape=-1, amp, 多进程等场景下的bug；[#51413](https://github.com/PaddlePaddle/Paddle/pull/51413), [#51435](https://github.com/PaddlePaddle/Paddle/pull/51435), [#50518](https://github.com/PaddlePaddle/Paddle/pull/50518), [#47301](https://github.com/PaddlePaddle/Paddle/pull/47301), - 组合算子修复composite grad maker和static prim api自动代码生成bug; 修复op创建属性丢失和部分组合规则不生效的bug [#50854](https://github.com/PaddlePaddle/Paddle/pull/50854), [#51445](https://github.com/PaddlePaddle/Paddle/pull/51445), [#50780](https://github.com/PaddlePaddle/Paddle/pull/50780), [#52120](https://github.com/PaddlePaddle/Paddle/pull/52120) - 组合算子修复一些其他bug [#50086](https://github.com/PaddlePaddle/Paddle/pull/50086), [#51208](https://github.com/PaddlePaddle/Paddle/pull/51208), [#51577](https://github.com/PaddlePaddle/Paddle/pull/51577), [#53598](https://github.com/PaddlePaddle/Paddle/pull/53598), [#47500](https://github.com/PaddlePaddle/Paddle/pull/47500), [#52119](https://github.com/PaddlePaddle/Paddle/pull/52119), [#50397](https://github.com/PaddlePaddle/Paddle/pull/50397), [#50527](https://github.com/PaddlePaddle/Paddle/pull/50527), [#50788](https://github.com/PaddlePaddle/Paddle/pull/50788), [#51014](https://github.com/PaddlePaddle/Paddle/pull/51014), [#52154](https://github.com/PaddlePaddle/Paddle/pull/52154), [#52752](https://github.com/PaddlePaddle/Paddle/pull/52752) - 动转静修复dataloader, cond输入dict, transformer导入, T5模型内存泄露, grad var name解析错误等bug [#49821](https://github.com/PaddlePaddle/Paddle/pull/49821)， [#47299](https://github.com/PaddlePaddle/Paddle/pull/47299), [#50776](https://github.com/PaddlePaddle/Paddle/pull/50776), [#50883](https://github.com/PaddlePaddle/Paddle/pull/50883), [#51100](https://github.com/PaddlePaddle/Paddle/pull/51100), [#51464](https://github.com/PaddlePaddle/Paddle/pull/51464), [#51966](https://github.com/PaddlePaddle/Paddle/pull/51966), [#52110](https://github.com/PaddlePaddle/Paddle/pull/52110), [#52821](https://github.com/PaddlePaddle/Paddle/pull/52821) - 动转静修复Lazy初始化，Windows训练，is_paddle_func失效，recurrent op删除pass失败等错误 [#50785](https://github.com/PaddlePaddle/Paddle/pull/50785), [#52580](https://github.com/PaddlePaddle/Paddle/pull/52580), [#51585](https://github.com/PaddlePaddle/Paddle/pull/51585), [#51763](https://github.com/PaddlePaddle/Paddle/pull/51763), [#51763](https://github.com/PaddlePaddle/Paddle/pull/51763) #### 性能优化 - 动转静调用run_program_op的执行过程中，增加scope缓存和复用机制，避免每个step都会传入新的scope [#45813](https://github.com/PaddlePaddle/Paddle/pull/45813) ### 分布式训练 #### 动态图分布式 - 去除旧动态图分布式sharding功能API [#49334](https://github.com/PaddlePaddle/Paddle/pull/49334) - fleet升级到distributed目录 [#50834](https://github.com/PaddlePaddle/Paddle/pull/50834) - 优化分布式策略的日志打印。[#47761](https://github.com/PaddlePaddle/Paddle/pull/47761) - 重计算支持hook模式、inplace功能、stop_gradient模式，支持更灵活的使用。 [#48471](https://github.com/PaddlePaddle/Paddle/pull/48471), [#47985](https://github.com/PaddlePaddle/Paddle/pull/47985) - 数据并行 - 数据并行支持no_sync接口，用于屏蔽参数梯度通信；参数同步功能；添加scale接口，缩放参数。[#47536](https://github.com/PaddlePaddle/Paddle/pull/47536),[#51895](https://github.com/PaddlePaddle/Paddle/pull/51895),[#47519](https://github.com/PaddlePaddle/Paddle/pull/47519) - 修复数据并行下显存泄露问题。[#47369](https://github.com/PaddlePaddle/Paddle/pull/47369),[#47444](https://github.com/PaddlePaddle/Paddle/pull/47444),[#48668](https://github.com/PaddlePaddle/Paddle/pull/48668) - 支持sparse 参数梯度同步。[#52785](https://github.com/PaddlePaddle/Paddle/pull/52785) - 流水线并行 - 优化流水线性能，去除通信等待，优化调度，通信overlap。[#46209](https://github.com/PaddlePaddle/Paddle/pull/46209),[#54003](https://github.com/PaddlePaddle/Paddle/pull/54003),[#54312](https://github.com/PaddlePaddle/Paddle/pull/54312),[#53384](https://github.com/PaddlePaddle/Paddle/pull/53384),[#54310](https://github.com/PaddlePaddle/Paddle/pull/54310),[#46399](https://github.com/PaddlePaddle/Paddle/pull/46399),[#46483](https://github.com/PaddlePaddle/Paddle/pull/46483),[#46780](https://github.com/PaddlePaddle/Paddle/pull/46780),[#46116](https://github.com/PaddlePaddle/Paddle/pull/46116) - 支持自定义切分，日志打印，随机种子设置，timer耗时打印。[#53344](https://github.com/PaddlePaddle/Paddle/pull/53344), [#47670](https://github.com/PaddlePaddle/Paddle/pull/47670),[#47336](https://github.com/PaddlePaddle/Paddle/pull/47336),[#52656](https://github.com/PaddlePaddle/Paddle/pull/52656),[#53831](https://github.com/PaddlePaddle/Paddle/pull/53831) - 优化流水线调度中的显存释放逻辑，提前释放中间变量和数据。[#54557](https://github.com/PaddlePaddle/Paddle/pull/54557), [#47199](https://github.com/PaddlePaddle/Paddle/pull/47199),[#47497](https://github.com/PaddlePaddle/Paddle/pull/47497),[#48045](https://github.com/PaddlePaddle/Paddle/pull/48045),[#54672](https://github.com/PaddlePaddle/Paddle/pull/54672) - 支持流水线并行的VPP模式，模型保存。[#54196](https://github.com/PaddlePaddle/Paddle/pull/54196), [#52927](https://github.com/PaddlePaddle/Paddle/pull/52927),[#47801](https://github.com/PaddlePaddle/Paddle/pull/47801),[#45922](https://github.com/PaddlePaddle/Paddle/pull/45922),[#47242](https://github.com/PaddlePaddle/Paddle/pull/47242) - 分组切分并行 - sharding stage2 并行支持量化功能，混合并行训练，梯度累加，XPU硬件，BF16低精度计算、优化器学习率设置、offload功能、数据并行。[#47169](https://github.com/PaddlePaddle/Paddle/pull/47169),[#47535](https://github.com/PaddlePaddle/Paddle/pull/47535), [#46795](https://github.com/PaddlePaddle/Paddle/pull/46795),[#47711](https://github.com/PaddlePaddle/Paddle/pull/47711),[#48310](https://github.com/PaddlePaddle/Paddle/pull/48310),[#46846](https://github.com/PaddlePaddle/Paddle/pull/46846),[#48857](https://github.com/PaddlePaddle/Paddle/pull/48857),[#49196](https://github.com/PaddlePaddle/Paddle/pull/49196),[#49931](https://github.com/PaddlePaddle/Paddle/pull/49931),[#47114](https://github.com/PaddlePaddle/Paddle/pull/47114),[#49767](https://github.com/PaddlePaddle/Paddle/pull/49767) - sharing stage2 性能优化，支持通信计算overlap。[#46495](https://github.com/PaddlePaddle/Paddle/pull/46495),[#46894](https://github.com/PaddlePaddle/Paddle/pull/46894) - sharding stage3 支持共享参数、不可训练参数。[#48695](https://github.com/PaddlePaddle/Paddle/pull/48695),[#48577](https://github.com/PaddlePaddle/Paddle/pull/48577) - 张量模型并行 - 张量模型并行性能优化，减少stream切流对性能的影响。[#47715](https://github.com/PaddlePaddle/Paddle/pull/47715),[#51617](https://github.com/PaddlePaddle/Paddle/pull/51617) - 支持参数、优化器状体、梯度同步。[#51428](https://github.com/PaddlePaddle/Paddle/pull/51428),[#53254](https://github.com/PaddlePaddle/Paddle/pull/53254), [#53335](https://github.com/PaddlePaddle/Paddle/pull/53335),[#45803](https://github.com/PaddlePaddle/Paddle/pull/45803),[#46303](https://github.com/PaddlePaddle/Paddle/pull/46303),[#52293](https://github.com/PaddlePaddle/Paddle/pull/52293) - 优化张量模型并行算子，如c_embedding、softmax_with_corss_entropy。[#53197](https://github.com/PaddlePaddle/Paddle/pull/53197),[#53547](https://github.com/PaddlePaddle/Paddle/pull/53547),[#53541](https://github.com/PaddlePaddle/Paddle/pull/53541),[#52789](https://github.com/PaddlePaddle/Paddle/pull/52789),[#46491](https://github.com/PaddlePaddle/Paddle/pull/46491),[#52742](https://github.com/PaddlePaddle/Paddle/pull/52742),[#53419](https://github.com/PaddlePaddle/Paddle/pull/53419) - Launch启动 - 支持分布式Launch功能，保存独立日志。[#53207](https://github.com/PaddlePaddle/Paddle/pull/53207),[#50405](https://github.com/PaddlePaddle/Paddle/pull/50405) - 新增框架打印环境变量功能，日志覆盖功能，日志返回，环境检查，便于debug环境变量的改动。[#53243](https://github.com/PaddlePaddle/Paddle/pull/53243),[#53243](https://github.com/PaddlePaddle/Paddle/pull/53243), [#51803](https://github.com/PaddlePaddle/Paddle/pull/51803), [#53990](https://github.com/PaddlePaddle/Paddle/pull/53990) - 通信库 - 增加自定义混合并行通信组，拓扑结构信息打印，自定义通信拓扑顺序。[#47021](https://github.com/PaddlePaddle/Paddle/pull/47021),[#54000](https://github.com/PaddlePaddle/Paddle/pull/54000),[#51781](https://github.com/PaddlePaddle/Paddle/pull/51781) - 去除通信库对Place信息依赖 [#47857](https://github.com/PaddlePaddle/Paddle/pull/47857) - 增加通信库对GLOO算子支持，支持send/recv/gather。 [#52221](https://github.com/PaddlePaddle/Paddle/pull/52221), [#52334](https://github.com/PaddlePaddle/Paddle/pull/52334),[#49084](https://github.com/PaddlePaddle/Paddle/pull/49084) - 禁止通信算子的反向计算。[#47636](https://github.com/PaddlePaddle/Paddle/pull/47636) - 新增通信库静态shape check，帮助判别通信量是否匹配。[#48256](https://github.com/PaddlePaddle/Paddle/pull/48256),[#48915](https://github.com/PaddlePaddle/Paddle/pull/48915),[#48646](https://github.com/PaddlePaddle/Paddle/pull/48646) - 支持通信python object类型，BF16类型，alltoall，reduce，allgather，group call，global gather，broadcast，scatter通信方式，XPU设备通信支持。[#51765](https://github.com/PaddlePaddle/Paddle/pull/51765),[#45844](https://github.com/PaddlePaddle/Paddle/pull/45844),[#48059](https://github.com/PaddlePaddle/Paddle/pull/48059),[#48115](https://github.com/PaddlePaddle/Paddle/pull/48115), [#48339](https://github.com/PaddlePaddle/Paddle/pull/48339),[#49252](https://github.com/PaddlePaddle/Paddle/pull/49252),[#49451](https://github.com/PaddlePaddle/Paddle/pull/49451),[#50085](https://github.com/PaddlePaddle/Paddle/pull/50085),[#50701](https://github.com/PaddlePaddle/Paddle/pull/50701),[#48208](https://github.com/PaddlePaddle/Paddle/pull/48208),[#48736](https://github.com/PaddlePaddle/Paddle/pull/48736),[#51762](https://github.com/PaddlePaddle/Paddle/pull/51762),[#52495](https://github.com/PaddlePaddle/Paddle/pull/52495),[#53514](https://github.com/PaddlePaddle/Paddle/pull/53514),[#48232](https://github.com/PaddlePaddle/Paddle/pull/48232),[#49896](https://github.com/PaddlePaddle/Paddle/pull/49896),[#49941](https://github.com/PaddlePaddle/Paddle/pull/49941),[#45584](https://github.com/PaddlePaddle/Paddle/pull/45584) - 新增对计算流通信功能。[#46182](https://github.com/PaddlePaddle/Paddle/pull/46182),[#46023](https://github.com/PaddlePaddle/Paddle/pull/46023),[#46295](https://github.com/PaddlePaddle/Paddle/pull/46295),[#46761](https://github.com/PaddlePaddle/Paddle/pull/46761),[#47481](https://github.com/PaddlePaddle/Paddle/pull/47481),[#47740](https://github.com/PaddlePaddle/Paddle/pull/47740),[#47976](https://github.com/PaddlePaddle/Paddle/pull/47976),[#48163](https://github.com/PaddlePaddle/Paddle/pull/48163),[#48396](https://github.com/PaddlePaddle/Paddle/pull/48396),[#48308](https://github.com/PaddlePaddle/Paddle/pull/48308),[#47110](https://github.com/PaddlePaddle/Paddle/pull/47110),[#53089](https://github.com/PaddlePaddle/Paddle/pull/53089) - 优化通信库TCP建联时间。[#49810](https://github.com/PaddlePaddle/Paddle/pull/49810),[#47184](https://github.com/PaddlePaddle/Paddle/pull/47184) #### 自动并行 - 静态图半自动并行功能完善： - 新增多个算子的FLOPs计算函数，并新增基于FLOPs的计算Cost建模 [#48083](https://github.com/PaddlePaddle/Paddle/pull/48083),[#47978](https://github.com/PaddlePaddle/Paddle/pull/47978),[#47595](https://github.com/PaddlePaddle/Paddle/pull/47595),[#48083](https://github.com/PaddlePaddle/Paddle/pull/48083),[#48084](https://github.com/PaddlePaddle/Paddle/pull/48084),[#47816](https://github.com/PaddlePaddle/Paddle/pull/47816) - 接口易用性提升，完善 DistAttr, Process Mesh, Engine API、信息打印、输入输出等模块；执行Engine新增cost接口，可用于理论分析模型运行的时间和显存开销 [#47503](https://github.com/PaddlePaddle/Paddle/pull/47503),[#46416](https://github.com/PaddlePaddle/Paddle/pull/46416),[#46554](https://github.com/PaddlePaddle/Paddle/pull/46554), [#46633](https://github.com/PaddlePaddle/Paddle/pull/46633),[#49214](https://github.com/PaddlePaddle/Paddle/pull/49214),[#53848](https://github.com/PaddlePaddle/Paddle/pull/53848),[#46552](https://github.com/PaddlePaddle/Paddle/pull/46552), [#47043](https://github.com/PaddlePaddle/Paddle/pull/47043), [#49665](https://github.com/PaddlePaddle/Paddle/pull/49665), [#52912](https://github.com/PaddlePaddle/Paddle/pull/52912), [#45776](https://github.com/PaddlePaddle/Paddle/pull/45776), [#47263](https://github.com/PaddlePaddle/Paddle/pull/47263) - 优化Pass的通用性和易用性升级，支持更多场景、减少Pass预分析耗时 [#46519](https://github.com/PaddlePaddle/Paddle/pull/46519),[#47358](https://github.com/PaddlePaddle/Paddle/pull/47358),[#46391](https://github.com/PaddlePaddle/Paddle/pull/46391), [#51035](https://github.com/PaddlePaddle/Paddle/pull/51035) - 调试能力增强，添加分布式随机性控制机制和混合并行精度对齐工具 [#52903](https://github.com/PaddlePaddle/Paddle/pull/52903),[#49865](https://github.com/PaddlePaddle/Paddle/pull/49865) - 支持推理生成任务组网的自动切分, 适配生成模型中的控制流、conditional block等特殊用法 [#46771](https://github.com/PaddlePaddle/Paddle/pull/46771), [#54067](https://github.com/PaddlePaddle/Paddle/pull/54067) - 完善grad_clip，支持了数据并行场景下的负载均衡。[#49510](https://github.com/PaddlePaddle/Paddle/pull/49510), [#49249](https://github.com/PaddlePaddle/Paddle/pull/49249) - 静态图半自动并行性能提升： - 新增 Sharding Pass 自动化通信Fuse 和多流通信功能，GPT 6.7B 模型两机上吞吐性能提升 26% [#48604](https://github.com/PaddlePaddle/Paddle/pull/48604), [#47180](https://github.com/PaddlePaddle/Paddle/pull/47180),[#46180](https://github.com/PaddlePaddle/Paddle/pull/46180) - 新增 Recompute 优化策略调优功能，支持根据显存和模型大小选择最优 recompute checkpoint 设置 [#48608](https://github.com/PaddlePaddle/Paddle/pull/48608),[#47846](https://github.com/PaddlePaddle/Paddle/pull/47846),[#49010](https://github.com/PaddlePaddle/Paddle/pull/49010) - 流水线并行新增 1F1B 调度优化 Pass [#54260](https://github.com/PaddlePaddle/Paddle/pull/54260), [#45915](https://github.com/PaddlePaddle/Paddle/pull/45915) - 数据并行优化，支持融合通信和通信计算Overlap 等优化, GPT 1.3B模型内性能提升 5% [#48092](https://github.com/PaddlePaddle/Paddle/pull/48092),[#45643](https://github.com/PaddlePaddle/Paddle/pull/45643),[#49744](https://github.com/PaddlePaddle/Paddle/pull/49744), [#47578](https://github.com/PaddlePaddle/Paddle/pull/47578) - 优化 Reshard模块concate 性能，减少部分场景下concate 次数。[#47809](https://github.com/PaddlePaddle/Paddle/pull/47809) - 混合精度优化Pass性能升级, 支持 BF16 低精度, 适配 while 循环控制流的自动混合并行等 [#51285](https://github.com/PaddlePaddle/Paddle/pull/51285),[#51147](https://github.com/PaddlePaddle/Paddle/pull/51147), [#49219](https://github.com/PaddlePaddle/Paddle/pull/49219), [#49079](https://github.com/PaddlePaddle/Paddle/pull/49079) - 静态图全自动并行功能完善： - 新增基于规则的全自动搜索策略 [#51859](https://github.com/PaddlePaddle/Paddle/pull/51859),[#51908](https://github.com/PaddlePaddle/Paddle/pull/51908),[#52053](https://github.com/PaddlePaddle/Paddle/pull/52053),[#48316](https://github.com/PaddlePaddle/Paddle/pull/48316),[#48464](https://github.com/PaddlePaddle/Paddle/pull/48464), [#52041](https://github.com/PaddlePaddle/Paddle/pull/52041) - 自动并行建模能力完善，丰富单节点内拓扑建模、通信量建模等。 [#52723](https://github.com/PaddlePaddle/Paddle/pull/52723),[#46387](https://github.com/PaddlePaddle/Paddle/pull/46387),[#47043](https://github.com/PaddlePaddle/Paddle/pull/47043) #### 参数服务器 - 清空ps目录下all列表，其中API不暴露 [#51289](https://github.com/PaddlePaddle/Paddle/pull/51289) - 清理cvm算子 [#48989](https://github.com/PaddlePaddle/Paddle/pull/48989) - GPUPS新增对AFS支持。[#46611](https://github.com/PaddlePaddle/Paddle/pull/46611) - PGLBOX2.0 日志降级、修复dense参数卡住问题、修复barrier不生效的问题、增加 get_epoch_finish python端接口[#49946](https://github.com/PaddlePaddle/Paddle/pull/49946),[#50166](https://github.com/PaddlePaddle/Paddle/pull/50166),[#50349](https://github.com/PaddlePaddle/Paddle/pull/50349) - GPUPs运行切换到指定模式。[#51115](https://github.com/PaddlePaddle/Paddle/pull/51115) - GPUPS加入benchmark。[#49587](https://github.com/PaddlePaddle/Paddle/pull/49587),[#49649](https://github.com/PaddlePaddle/Paddle/pull/49649) - GPUPS优化器选择问题修复，修复reader读取问题，修复RPC编译问题。 [#47026](https://github.com/PaddlePaddle/Paddle/pull/47026),[#47192](https://github.com/PaddlePaddle/Paddle/pull/47192),[#49878](https://github.com/PaddlePaddle/Paddle/pull/49878), [#46356](https://github.com/PaddlePaddle/Paddle/pull/46356),[#46575](https://github.com/PaddlePaddle/Paddle/pull/46575),[#49389](https://github.com/PaddlePaddle/Paddle/pull/49389),[#46258](https://github.com/PaddlePaddle/Paddle/pull/46258),[#50136](https://github.com/PaddlePaddle/Paddle/pull/50136) - 增加rocksdb编译方式。[#46074](https://github.com/PaddlePaddle/Paddle/pull/46074) ### CUDA #### 新功能 - 新增对CUDA 12.0的编译支持，并修复相关单测 ([#49539](https://github.com/PaddlePaddle/Paddle/pull/49539), [#54542](https://github.com/PaddlePaddle/Paddle/pull/54542)) - 新增CUDNN Frontend API的编译支持及相关单测，可以使用`WITH_CUDNN_FRONTEND=ON` 的编译选项进行开启。([#47524](https://github.com/PaddlePaddle/Paddle/pull/47524), [#47612](https://github.com/PaddlePaddle/Paddle/pull/47612)) #### 功能优化 - 混合精度策略及精度优化： - 新增及优化了框架200余个算子的FP16、BF16数据类型支持，包括logsumexp，reduce_max，cumprod，sync_batch_norm，compare类OP等，并对所有FP16、BF16算子进行了精度优化及单测覆盖，针对低精度算子完善单测框架功能，确保在大模型训推过程中精度无损。([#51193](https://github.com/PaddlePaddle/Paddle/pull/51193), [#51114](https://github.com/PaddlePaddle/Paddle/pull/51114), [#45817](https://github.com/PaddlePaddle/Paddle/pull/45817), [#52862](https://github.com/PaddlePaddle/Paddle/pull/52862), [#52919](https://github.com/PaddlePaddle/Paddle/pull/52919), [#52921](https://github.com/PaddlePaddle/Paddle/pull/52921), [#46413](https://github.com/PaddlePaddle/Paddle/pull/46413), [#48205](https://github.com/PaddlePaddle/Paddle/pull/48205), [#54193](https://github.com/PaddlePaddle/Paddle/pull/54193), [#48041](https://github.com/PaddlePaddle/Paddle/pull/48041), [#48121](https://github.com/PaddlePaddle/Paddle/pull/48121), [#46364](https://github.com/PaddlePaddle/Paddle/pull/46364), [#51153](https://github.com/PaddlePaddle/Paddle/pull/51153), [#53023](https://github.com/PaddlePaddle/Paddle/pull/53023), [#53079](https://github.com/PaddlePaddle/Paddle/pull/53079), [#53137](https://github.com/PaddlePaddle/Paddle/pull/53137), [#46212](https://github.com/PaddlePaddle/Paddle/pull/46212), [#50908](https://github.com/PaddlePaddle/Paddle/pull/50908), [#52555](https://github.com/PaddlePaddle/Paddle/pull/52555), [#51582](https://github.com/PaddlePaddle/Paddle/pull/51582), [#47897](https://github.com/PaddlePaddle/Paddle/pull/47897), [#45601](https://github.com/PaddlePaddle/Paddle/pull/45601), [#53522](https://github.com/PaddlePaddle/Paddle/pull/53522), [#52666](https://github.com/PaddlePaddle/Paddle/pull/52666), [#50101](https://github.com/PaddlePaddle/Paddle/pull/50101), [#48315](https://github.com/PaddlePaddle/Paddle/pull/48315), [#50847](https://github.com/PaddlePaddle/Paddle/pull/50847), [#50905](https://github.com/PaddlePaddle/Paddle/pull/50905), [#50906](https://github.com/PaddlePaddle/Paddle/pull/50906), [#50909](https://github.com/PaddlePaddle/Paddle/pull/50909), [#50916](https://github.com/PaddlePaddle/Paddle/pull/50916), [#50917](https://github.com/PaddlePaddle/Paddle/pull/50917), [#50920](https://github.com/PaddlePaddle/Paddle/pull/50920), [#50919](https://github.com/PaddlePaddle/Paddle/pull/50919), [#50904](https://github.com/PaddlePaddle/Paddle/pull/50904), [#50918](https://github.com/PaddlePaddle/Paddle/pull/50918), [#50938](https://github.com/PaddlePaddle/Paddle/pull/50938), [#50858](https://github.com/PaddlePaddle/Paddle/pull/50858), [#50933](https://github.com/PaddlePaddle/Paddle/pull/50933), [#50945](https://github.com/PaddlePaddle/Paddle/pull/50945), [#50936](https://github.com/PaddlePaddle/Paddle/pull/50936), [#51168](https://github.com/PaddlePaddle/Paddle/pull/51168), [#51493](https://github.com/PaddlePaddle/Paddle/pull/51493), [#50924](https://github.com/PaddlePaddle/Paddle/pull/50924), [#50923](https://github.com/PaddlePaddle/Paddle/pull/50923), [#50926](https://github.com/PaddlePaddle/Paddle/pull/50926), [#50925](https://github.com/PaddlePaddle/Paddle/pull/50925), [#50930](https://github.com/PaddlePaddle/Paddle/pull/50930), [#53284](https://github.com/PaddlePaddle/Paddle/pull/53284), [#53286](https://github.com/PaddlePaddle/Paddle/pull/53286), [#53285](https://github.com/PaddlePaddle/Paddle/pull/53285), [#50976](https://github.com/PaddlePaddle/Paddle/pull/50976), [#50915](https://github.com/PaddlePaddle/Paddle/pull/50915), [#50915](https://github.com/PaddlePaddle/Paddle/pull/50915), [#48192](https://github.com/PaddlePaddle/Paddle/pull/48192), [#50993](https://github.com/PaddlePaddle/Paddle/pull/50993)， [#50998](https://github.com/PaddlePaddle/Paddle/pull/50998), [#51380](https://github.com/PaddlePaddle/Paddle/pull/51380), [#51137](https://github.com/PaddlePaddle/Paddle/pull/51137), [#51106](https://github.com/PaddlePaddle/Paddle/pull/51106), [#51197](https://github.com/PaddlePaddle/Paddle/pull/51197), [#51159](https://github.com/PaddlePaddle/Paddle/pull/51159), [#51552](https://github.com/PaddlePaddle/Paddle/pull/51552), [#51151](https://github.com/PaddlePaddle/Paddle/pull/51151), [#51005](https://github.com/PaddlePaddle/Paddle/pull/51005), [#51565](https://github.com/PaddlePaddle/Paddle/pull/51565), [#51036](https://github.com/PaddlePaddle/Paddle/pull/51036), [#51185](https://github.com/PaddlePaddle/Paddle/pull/51185), [#51791](https://github.com/PaddlePaddle/Paddle/pull/51791), [#51083](https://github.com/PaddlePaddle/Paddle/pull/51083), [#51694](https://github.com/PaddlePaddle/Paddle/pull/51694), [#51689](https://github.com/PaddlePaddle/Paddle/pull/51689), [#51009](https://github.com/PaddlePaddle/Paddle/pull/51009), [#51051](https://github.com/PaddlePaddle/Paddle/pull/51051), [#51532](https://github.com/PaddlePaddle/Paddle/pull/51532), [#51978](https://github.com/PaddlePaddle/Paddle/pull/51978), [#51903](https://github.com/PaddlePaddle/Paddle/pull/51903), [#51888](https://github.com/PaddlePaddle/Paddle/pull/51888), [#52016](https://github.com/PaddlePaddle/Paddle/pull/52016), [#52035](https://github.com/PaddlePaddle/Paddle/pull/52035), [#52184](https://github.com/PaddlePaddle/Paddle/pull/52184), [#52018](https://github.com/PaddlePaddle/Paddle/pull/52018), [#51787](https://github.com/PaddlePaddle/Paddle/pull/51787), [#51640](https://github.com/PaddlePaddle/Paddle/pull/51640), [#52172](https://github.com/PaddlePaddle/Paddle/pull/52172), [#52193](https://github.com/PaddlePaddle/Paddle/pull/52193), [#51160](https://github.com/PaddlePaddle/Paddle/pull/51160), [#51809](https://github.com/PaddlePaddle/Paddle/pull/51809), [#51678](https://github.com/PaddlePaddle/Paddle/pull/51678), [#52158](https://github.com/PaddlePaddle/Paddle/pull/52158), [#51015](https://github.com/PaddlePaddle/Paddle/pull/51015), [#52240](https://github.com/PaddlePaddle/Paddle/pull/52240), [#52276](https://github.com/PaddlePaddle/Paddle/pull/52276), [#52233](https://github.com/PaddlePaddle/Paddle/pull/52233), [#52220](https://github.com/PaddlePaddle/Paddle/pull/52220), [#52107](https://github.com/PaddlePaddle/Paddle/pull/52107), [#52282](https://github.com/PaddlePaddle/Paddle/pull/52282), [#52311](https://github.com/PaddlePaddle/Paddle/pull/52311), [#52315](https://github.com/PaddlePaddle/Paddle/pull/52315), [#52357](https://github.com/PaddlePaddle/Paddle/pull/52357), [#52256](https://github.com/PaddlePaddle/Paddle/pull/52256), [#51649](https://github.com/PaddlePaddle/Paddle/pull/51649), [#52413](https://github.com/PaddlePaddle/Paddle/pull/52413), [#52369](https://github.com/PaddlePaddle/Paddle/pull/52369), [#51837](https://github.com/PaddlePaddle/Paddle/pull/51837), [#52112](https://github.com/PaddlePaddle/Paddle/pull/52112), [#51819](https://github.com/PaddlePaddle/Paddle/pull/51819), [#52388](https://github.com/PaddlePaddle/Paddle/pull/52388), [#52411](https://github.com/PaddlePaddle/Paddle/pull/52411), [#52521](https://github.com/PaddlePaddle/Paddle/pull/52521), [#51300](https://github.com/PaddlePaddle/Paddle/pull/51300), [#51117](https://github.com/PaddlePaddle/Paddle/pull/51117), [#52380](https://github.com/PaddlePaddle/Paddle/pull/52380), [#52317](https://github.com/PaddlePaddle/Paddle/pull/52317), [#51263](https://github.com/PaddlePaddle/Paddle/pull/51263), [#52668](https://github.com/PaddlePaddle/Paddle/pull/52668), [#52259](https://github.com/PaddlePaddle/Paddle/pull/52259), [#50999](https://github.com/PaddlePaddle/Paddle/pull/50999), [#52407](https://github.com/PaddlePaddle/Paddle/pull/52407), [#52288](https://github.com/PaddlePaddle/Paddle/pull/52288), [#52845](https://github.com/PaddlePaddle/Paddle/pull/52845), [#50953](https://github.com/PaddlePaddle/Paddle/pull/50953), [#52667](https://github.com/PaddlePaddle/Paddle/pull/52667), [#52582](https://github.com/PaddlePaddle/Paddle/pull/52582), [#52426](https://github.com/PaddlePaddle/Paddle/pull/52426), [#51884](https://github.com/PaddlePaddle/Paddle/pull/51884), [#52630](https://github.com/PaddlePaddle/Paddle/pull/52630), [#52136](https://github.com/PaddlePaddle/Paddle/pull/52136), [#52604](https://github.com/PaddlePaddle/Paddle/pull/52604), [#51615](https://github.com/PaddlePaddle/Paddle/pull/51615), [#51275](https://github.com/PaddlePaddle/Paddle/pull/51275), [#52898](https://github.com/PaddlePaddle/Paddle/pull/52898), [#52918](https://github.com/PaddlePaddle/Paddle/pull/52918), [#52572](https://github.com/PaddlePaddle/Paddle/pull/52572), [#52683](https://github.com/PaddlePaddle/Paddle/pull/52683), [#52956](https://github.com/PaddlePaddle/Paddle/pull/52956), [#52963](https://github.com/PaddlePaddle/Paddle/pull/52963), [#52954](https://github.com/PaddlePaddle/Paddle/pull/52954), [#52444](https://github.com/PaddlePaddle/Paddle/pull/52444), [#52314](https://github.com/PaddlePaddle/Paddle/pull/52314), [#52887](https://github.com/PaddlePaddle/Paddle/pull/52887), [#52195](https://github.com/PaddlePaddle/Paddle/pull/52195), [#53100](https://github.com/PaddlePaddle/Paddle/pull/53100), [#52961](https://github.com/PaddlePaddle/Paddle/pull/52961), [#52953](https://github.com/PaddlePaddle/Paddle/pull/52953), [#53111](https://github.com/PaddlePaddle/Paddle/pull/53111), [#53549](https://github.com/PaddlePaddle/Paddle/pull/53549), [#53736](https://github.com/PaddlePaddle/Paddle/pull/53736), [#52920](https://github.com/PaddlePaddle/Paddle/pull/52920), [#53195](https://github.com/PaddlePaddle/Paddle/pull/53195), [#53535](https://github.com/PaddlePaddle/Paddle/pull/53535), [#53876](https://github.com/PaddlePaddle/Paddle/pull/53876), [#53785](https://github.com/PaddlePaddle/Paddle/pull/53785), [#53722](https://github.com/PaddlePaddle/Paddle/pull/53722), [#54285](https://github.com/PaddlePaddle/Paddle/pull/54285), [#54232](https://github.com/PaddlePaddle/Paddle/pull/54232), [#53922](https://github.com/PaddlePaddle/Paddle/pull/53922), [#47277](https://github.com/PaddlePaddle/Paddle/pull/47277), [#50811](https://github.com/PaddlePaddle/Paddle/pull/50811), [#54571](https://github.com/PaddlePaddle/Paddle/pull/54571), [#50129](https://github.com/PaddlePaddle/Paddle/pull/50129), [#50340](https://github.com/PaddlePaddle/Paddle/pull/50340), [#50848](https://github.com/PaddlePaddle/Paddle/pull/50848), [#50849](https://github.com/PaddlePaddle/Paddle/pull/50849), [#50868](https://github.com/PaddlePaddle/Paddle/pull/50868), [#50878](https://github.com/PaddlePaddle/Paddle/pull/50878), [#50929](https://github.com/PaddlePaddle/Paddle/pull/50929), [#50939](https://github.com/PaddlePaddle/Paddle/pull/50939), [#50973](https://github.com/PaddlePaddle/Paddle/pull/50973), [#50913](https://github.com/PaddlePaddle/Paddle/pull/50913), [#51145](https://github.com/PaddlePaddle/Paddle/pull/51145), [#51090](https://github.com/PaddlePaddle/Paddle/pull/51090), [#51098](https://github.com/PaddlePaddle/Paddle/pull/51098), [#51094](https://github.com/PaddlePaddle/Paddle/pull/51094), [#51216](https://github.com/PaddlePaddle/Paddle/pull/51216), [#51736](https://github.com/PaddlePaddle/Paddle/pull/51736), [#51684](https://github.com/PaddlePaddle/Paddle/pull/51684), [#51925](https://github.com/PaddlePaddle/Paddle/pull/51925), [#54030](https://github.com/PaddlePaddle/Paddle/pull/54030), [#50700](https://github.com/PaddlePaddle/Paddle/pull/50700), [#52264](https://github.com/PaddlePaddle/Paddle/pull/52264), [#51069](https://github.com/PaddlePaddle/Paddle/pull/51069), [#51101](https://github.com/PaddlePaddle/Paddle/pull/51101), [#51286](https://github.com/PaddlePaddle/Paddle/pull/51286), [#53582](https://github.com/PaddlePaddle/Paddle/pull/53582),[#49869](https://github.com/PaddlePaddle/Paddle/pull/49869))) - 混合精度策略（AMP）优化：在混合精度训练的易用性、精度稳定性及可调试性方面进行了全面的升级和优化，能够更好的支持大模型训练加速。易用性方面统一了动静态图API，并新增model.float()、model.float16()、model.bfloat16()等转换接口；精度稳定性方面增强了针对BF16类型的策略自动调整，优化了黑名单设置，增强了优化器算子Adagrad、Adamax、Adadelta、RMSProp等对 multi_precision 功能的支持，在O2模式下，完善了master grad机制，并新增类型提升机制，以及新增参数对特定模块使用float32计算以保障精度；在可调式性方面，新增paddle.amp.debugging 模块，提供算子统计、异常值检测、精度对比等功能。( [#50132](https://github.com/PaddlePaddle/Paddle/pull/50132), [#50078](https://github.com/PaddlePaddle/Paddle/pull/50078), [#50131](https://github.com/PaddlePaddle/Paddle/pull/50131), [#49705](https://github.com/PaddlePaddle/Paddle/pull/49705), [#52936](https://github.com/PaddlePaddle/Paddle/pull/52936), [#52871](https://github.com/PaddlePaddle/Paddle/pull/52871), [#53289](https://github.com/PaddlePaddle/Paddle/pull/53289), [#53362](https://github.com/PaddlePaddle/Paddle/pull/53362), [#54240](https://github.com/PaddlePaddle/Paddle/pull/54240), [#53768](https://github.com/PaddlePaddle/Paddle/pull/53768), [#48041](https://github.com/PaddlePaddle/Paddle/pull/48041), [#47672](https://github.com/PaddlePaddle/Paddle/pull/47672), [#48843](https://github.com/PaddlePaddle/Paddle/pull/48843), [#49391](https://github.com/PaddlePaddle/Paddle/pull/49391), [#51635](https://github.com/PaddlePaddle/Paddle/pull/51635), [#45541](https://github.com/PaddlePaddle/Paddle/pull/45541), [#53742](https://github.com/PaddlePaddle/Paddle/pull/53742), [#51020](https://github.com/PaddlePaddle/Paddle/pull/51020), [#51063](https://github.com/PaddlePaddle/Paddle/pull/51063), [#52514](https://github.com/PaddlePaddle/Paddle/pull/52514), [#50940](https://github.com/PaddlePaddle/Paddle/pull/50940), [#52936](https://github.com/PaddlePaddle/Paddle/pull/52936), [#53439](https://github.com/PaddlePaddle/Paddle/pull/53439), [#53712](https://github.com/PaddlePaddle/Paddle/pull/53712), [#48238](https://github.com/PaddlePaddle/Paddle/pull/48238), [#52215](https://github.com/PaddlePaddle/Paddle/pull/52215), [#53012](https://github.com/PaddlePaddle/Paddle/pull/53012), [#52918](https://github.com/PaddlePaddle/Paddle/pull/52918), [#54571](https://github.com/PaddlePaddle/Paddle/pull/54571)) - GroupNorm算子新增对NHWC数据格式的支持 ([#47533](https://github.com/PaddlePaddle/Paddle/pull/47533)) - index_put算子新增对bool和int的混合数据类型支持 ([#54195](https://github.com/PaddlePaddle/Paddle/pull/54195)) - 新增sparse.is_nan API 用于判断sparse tensor中是否含有NaN元素。 ([#51513](https://github.com/PaddlePaddle/Paddle/pull/51513)) #### bug fix - 修复trace、roll、dropout_nd、log_softmax等多个算子计算出错、栈溢出，以及部分单测问题。([#50243](https://github.com/PaddlePaddle/Paddle/pull/50243), [#52012](https://github.com/PaddlePaddle/Paddle/pull/52012), [#53795](https://github.com/PaddlePaddle/Paddle/pull/53795), [#53149](https://github.com/PaddlePaddle/Paddle/pull/53149), [#53654](https://github.com/PaddlePaddle/Paddle/pull/53654), [#51054](https://github.com/PaddlePaddle/Paddle/pull/51054), [#49373](https://github.com/PaddlePaddle/Paddle/pull/49373), [#53038](https://github.com/PaddlePaddle/Paddle/pull/53038)) - 修复conv算子穷举搜索在部分场景不生效的问题。([#47065](https://github.com/PaddlePaddle/Paddle/pull/47065)) - 修复collective_reduce_scatter等算子在A100上出现timeout的问题。([#54513](https://github.com/PaddlePaddle/Paddle/pull/54513)) - 修复FusedLinear单测中属性错误的问题。 ([#50359](https://github.com/PaddlePaddle/Paddle/pull/50359)) - 修复在使用Profiler时可能出现的OOM等问题 ([#46089](https://github.com/PaddlePaddle/Paddle/pull/46089)) #### 性能提升 - 进一步优化框架大量算子的GPU Kernel以及eigen实现方式，包括max_pool3d, dropout, adaptive_pooling, depthwise_conv2d、transpose, eigh, broadcast类计算，reduce类计算，prelu，logsumexp，以及sparse类算子等，在更多配置场景下达到更优性能。([#45820](https://github.com/PaddlePaddle/Paddle/pull/45820), [#45959](https://github.com/PaddlePaddle/Paddle/pull/45959), [#45934](https://github.com/PaddlePaddle/Paddle/pull/45934), [#46332](https://github.com/PaddlePaddle/Paddle/pull/46332), [#46287](https://github.com/PaddlePaddle/Paddle/pull/46287), [#47233](https://github.com/PaddlePaddle/Paddle/pull/47233), [#48855](https://github.com/PaddlePaddle/Paddle/pull/48855), [#48560](https://github.com/PaddlePaddle/Paddle/pull/48560), [#49419](https://github.com/PaddlePaddle/Paddle/pull/49419), [#49748](https://github.com/PaddlePaddle/Paddle/pull/49748), [#50348](https://github.com/PaddlePaddle/Paddle/pull/50348), [#52401](https://github.com/PaddlePaddle/Paddle/pull/52401), [#51131](https://github.com/PaddlePaddle/Paddle/pull/51131), [#51141](https://github.com/PaddlePaddle/Paddle/pull/51141), [#51479](https://github.com/PaddlePaddle/Paddle/pull/51479), [#51835](https://github.com/PaddlePaddle/Paddle/pull/51835), [#52509](https://github.com/PaddlePaddle/Paddle/pull/52509), [#52482](https://github.com/PaddlePaddle/Paddle/pull/52482), [#52700](https://github.com/PaddlePaddle/Paddle/pull/52700), [#53112](https://github.com/PaddlePaddle/Paddle/pull/53112), [#53659](https://github.com/PaddlePaddle/Paddle/pull/53659), [#53658](https://github.com/PaddlePaddle/Paddle/pull/53658), [#53154](https://github.com/PaddlePaddle/Paddle/pull/53154), [#54071](https://github.com/PaddlePaddle/Paddle/pull/54071), [#53622](https://github.com/PaddlePaddle/Paddle/pull/53622), [#52952](https://github.com/PaddlePaddle/Paddle/pull/52952), [#46046](https://github.com/PaddlePaddle/Paddle/pull/46046), [#46119](https://github.com/PaddlePaddle/Paddle/pull/46119), [#45946](https://github.com/PaddlePaddle/Paddle/pull/45946), [#47212](https://github.com/PaddlePaddle/Paddle/pull/47212), [#47791](https://github.com/PaddlePaddle/Paddle/pull/47791), [#47454](https://github.com/PaddlePaddle/Paddle/pull/47454), [#45230](https://github.com/PaddlePaddle/Paddle/pull/45230), [#48899](https://github.com/PaddlePaddle/Paddle/pull/48899), [#33051](https://github.com/PaddlePaddle/Paddle/pull/33051), [#49040](https://github.com/PaddlePaddle/Paddle/pull/49040), [#48992](https://github.com/PaddlePaddle/Paddle/pull/48992), [#49086](https://github.com/PaddlePaddle/Paddle/pull/49086), [#50808](https://github.com/PaddlePaddle/Paddle/pull/50808), [#46431](https://github.com/PaddlePaddle/Paddle/pull/46431), [#50931](https://github.com/PaddlePaddle/Paddle/pull/50931), [#48056](https://github.com/PaddlePaddle/Paddle/pull/48056), [#46071](https://github.com/PaddlePaddle/Paddle/pull/46071), [#49231](https://github.com/PaddlePaddle/Paddle/pull/49231), [#38660](https://github.com/PaddlePaddle/Paddle/pull/38660), [#50287](https://github.com/PaddlePaddle/Paddle/pull/50287), [#46111](https://github.com/PaddlePaddle/Paddle/pull/46111), [#46997](https://github.com/PaddlePaddle/Paddle/pull/46997), [#45854](https://github.com/PaddlePaddle/Paddle/pull/45854), [#47738](https://github.com/PaddlePaddle/Paddle/pull/47738), [#48635](https://github.com/PaddlePaddle/Paddle/pull/48635), [#50353](https://github.com/PaddlePaddle/Paddle/pull/50353), [#50362](https://github.com/PaddlePaddle/Paddle/pull/50362), [#51934](https://github.com/PaddlePaddle/Paddle/pull/51934), [#54045](https://github.com/PaddlePaddle/Paddle/pull/54045), [#46679](https://github.com/PaddlePaddle/Paddle/pull/46679), [#52093](https://github.com/PaddlePaddle/Paddle/pull/52093), [#52969](https://github.com/PaddlePaddle/Paddle/pull/52969)) - 提供更多融合算子实现，以及相关融合Pass，如fused_feed_forward，gather-gemm-scatter，matmul + bias，layernorm_shift_partition + element_add，elementwise类融合等模式，进一步提升使用该模式的模型性能。( [#50423](https://github.com/PaddlePaddle/Paddle/pull/50423), [#50091](https://github.com/PaddlePaddle/Paddle/pull/50091), [#50364](https://github.com/PaddlePaddle/Paddle/pull/50364), [#53017](https://github.com/PaddlePaddle/Paddle/pull/53017), [#50755](https://github.com/PaddlePaddle/Paddle/pull/50755), [#50050](https://github.com/PaddlePaddle/Paddle/pull/50050), [#47099](https://github.com/PaddlePaddle/Paddle/pull/47099), [#48848](https://github.com/PaddlePaddle/Paddle/pull/48848), [#49383](https://github.com/PaddlePaddle/Paddle/pull/49383), [#50809](https://github.com/PaddlePaddle/Paddle/pull/50809), [#52361](https://github.com/PaddlePaddle/Paddle/pull/52361), [#52028](https://github.com/PaddlePaddle/Paddle/pull/52028), [#48439](https://github.com/PaddlePaddle/Paddle/pull/48439), [#49009](https://github.com/PaddlePaddle/Paddle/pull/49009), [#51427](https://github.com/PaddlePaddle/Paddle/pull/51427), [#52731](https://github.com/PaddlePaddle/Paddle/pull/52731), [#51805](https://github.com/PaddlePaddle/Paddle/pull/51805)) #### 文档 - 修复index_put文档中的错误 ([#53727](https://github.com/PaddlePaddle/Paddle/pull/53727)) ### Intermediate Representation 为了飞桨IR体系存在的稳定性、降低研发成本问题，孵化了飞桨新的IR体系，完成了基础的数据结构定义、算子定义生成和执行体系适配。为了更好的支持科学计算场景的高阶需求，完成了silu、cast等算子的高阶适配。 - 完成了IR数据数据结构定义，包含类型系统，算子定义；打通了和phi kernel的执行适配。[#51112](https://github.com/PaddlePaddle/Paddle/pull/51112)， [#51992](https://github.com/PaddlePaddle/Paddle/pull/51992), [#50412](https://github.com/PaddlePaddle/Paddle/pull/50412), [#53557](https://github.com/PaddlePaddle/Paddle/pull/53557), [#53953](https://github.com/PaddlePaddle/Paddle/pull/53953), [#50959](https://github.com/PaddlePaddle/Paddle/pull/50959), [#54250](https://github.com/PaddlePaddle/Paddle/pull/54250), [#54197](https://github.com/PaddlePaddle/Paddle/pull/54197), [#54289](https://github.com/PaddlePaddle/Paddle/pull/54289), [#51636](https://github.com/PaddlePaddle/Paddle/pull/51636), [#52846](https://github.com/PaddlePaddle/Paddle/pull/52846), [#53988](https://github.com/PaddlePaddle/Paddle/pull/53988), [#54143](https://github.com/PaddlePaddle/Paddle/pull/54143), [#54035](https://github.com/PaddlePaddle/Paddle/pull/54035), [#54052](https://github.com/PaddlePaddle/Paddle/pull/54052), [#54340](https://github.com/PaddlePaddle/Paddle/pull/54340), [#54356](https://github.com/PaddlePaddle/Paddle/pull/54356), [#54068](https://github.com/PaddlePaddle/Paddle/pull/54068), [#53894](https://github.com/PaddlePaddle/Paddle/pull/53894), [#53707](https://github.com/PaddlePaddle/Paddle/pull/53707), [#54185](https://github.com/PaddlePaddle/Paddle/pull/54185), [#54031](https://github.com/PaddlePaddle/Paddle/pull/54031), [#54220](https://github.com/PaddlePaddle/Paddle/pull/54220), [#54275](https://github.com/PaddlePaddle/Paddle/pull/54275), [#54281](https://github.com/PaddlePaddle/Paddle/pull/54281), [#54186](https://github.com/PaddlePaddle/Paddle/pull/54186), [#54259](https://github.com/PaddlePaddle/Paddle/pull/54259), [#54124](https://github.com/PaddlePaddle/Paddle/pull/54124), [#54292](https://github.com/PaddlePaddle/Paddle/pull/54292), [#48068](https://github.com/PaddlePaddle/Paddle/pull/48068), [#53978](https://github.com/PaddlePaddle/Paddle/pull/53978) - 完善pass基础设置，包含基础的pass定义，pass注册管理等。 [#54023](https://github.com/PaddlePaddle/Paddle/pull/54023),[#54170](https://github.com/PaddlePaddle/Paddle/pull/54170), [#54170](https://github.com/PaddlePaddle/Paddle/pull/54170), [#54308](https://github.com/PaddlePaddle/Paddle/pull/54308), [#54348](https://github.com/PaddlePaddle/Paddle/pull/54348), [#54385](https://github.com/PaddlePaddle/Paddle/pull/54385) - 完善高阶算子的适配，主要包含基础模块改造和silu、cast算子适配等。 [#52005](https://github.com/PaddlePaddle/Paddle/pull/52005), [#53425](https://github.com/PaddlePaddle/Paddle/pull/53425), [#53417](https://github.com/PaddlePaddle/Paddle/pull/53417), [#53417](https://github.com/PaddlePaddle/Paddle/pull/53417), [#53498](https://github.com/PaddlePaddle/Paddle/pull/53498), [#53171](https://github.com/PaddlePaddle/Paddle/pull/53171), [#53632](https://github.com/PaddlePaddle/Paddle/pull/53632), [#53605](https://github.com/PaddlePaddle/Paddle/pull/53605), [#53746](https://github.com/PaddlePaddle/Paddle/pull/53746), [#53874](https://github.com/PaddlePaddle/Paddle/pull/53874), [#54164](https://github.com/PaddlePaddle/Paddle/pull/54164), [#45888](https://github.com/PaddlePaddle/Paddle/pull/45888), [#46024](https://github.com/PaddlePaddle/Paddle/pull/46024), [#46446](https://github.com/PaddlePaddle/Paddle/pull/46446), [#46960](https://github.com/PaddlePaddle/Paddle/pull/46960) ### CINN编译器 #### 新功能 - 新增CINN对0D-Tensor的支持，目前为配合主框架升级，暂时采用增加pass的临时方案进行支持，后续会对该方案进行替换升级。 ([#53382](https://github.com/PaddlePaddle/Paddle/pull/53382), [#53955](https://github.com/PaddlePaddle/Paddle/pull/53955), [#54064](https://github.com/PaddlePaddle/Paddle/pull/54064), [#54118](https://github.com/PaddlePaddle/Paddle/pull/54118), [#54216](https://github.com/PaddlePaddle/Paddle/pull/54216), [#53454](https://github.com/PaddlePaddle/Paddle/pull/53454)) - 新增CINN对int8/uint8/int16/uint16/bf16等数据类型的支持 ([#50566](https://github.com/PaddlePaddle/Paddle/pull/50566), [#53637](https://github.com/PaddlePaddle/Paddle/pull/53637)) - 新增CINN expand算子的支持 ([#46776](https://github.com/PaddlePaddle/Paddle/pull/46776)) - 新增CINN对PaddleInference的支持. ([#45009](https://github.com/PaddlePaddle/Paddle/pull/45009)) #### 功能优化 - CINN编译器，传递skip_gc_vars属性到CINN子图；CINN为skip_gc_vars添加fetch算子 [#49471](https://github.com/PaddlePaddle/Paddle/pull/49471), [#49553](https://github.com/PaddlePaddle/Paddle/pull/49553) - CINN编译器，conv2d和conv2d_grad默认不使用cinn算子 [#51645](https://github.com/PaddlePaddle/Paddle/pull/51645) - 将 build_cinn_pass 添加到 BuildStrategy，以便于在动转静中使用 ([#49496](https://github.com/PaddlePaddle/Paddle/pull/49496)) - 增加reshape算子在组合算子机制下的单测 ([#51276](https://github.com/PaddlePaddle/Paddle/pull/51276)) - 主框架联编CINN的版本从固定commit改为develop ([#49775](https://github.com/PaddlePaddle/Paddle/pull/49775)) - 为CINN设置默认Target参数 ([#50182](https://github.com/PaddlePaddle/Paddle/pull/50182)) #### bug fix - 修复CINN符号化过程中拓扑排序后的出现的算子顺序不一致的问题。 ([#52556](https://github.com/PaddlePaddle/Paddle/pull/52556)) - 修复一些算子计算错误、精度下降，以及单测相关问题 ([#53859](https://github.com/PaddlePaddle/Paddle/pull/53859), [#54261](https://github.com/PaddlePaddle/Paddle/pull/54261), [#46801](https://github.com/PaddlePaddle/Paddle/pull/46801), [#53676](https://github.com/PaddlePaddle/Paddle/pull/53676), [#53772](https://github.com/PaddlePaddle/Paddle/pull/53772)) - 修复CINN对float16类型支持的问题。([#48249](https://github.com/PaddlePaddle/Paddle/pull/48249)) - 修复build_cinn_pass中的问题。 ([#46843](https://github.com/PaddlePaddle/Paddle/pull/46843)) - 修复了组合算子+动转静在开启CINN时，出现反向因误被GC而导致的无数据区的问题 ([#50116](https://github.com/PaddlePaddle/Paddle/pull/50116)) - 修复编译器dropout amp出错，组合算子跑resnet出错，inplace变量未找到等问题 [#51688](https://github.com/PaddlePaddle/Paddle/pull/51688), [#52813](https://github.com/PaddlePaddle/Paddle/pull/52813), [#51769](https://github.com/PaddlePaddle/Paddle/pull/51769) #### 性能提升 - 优化reshape相关融合策略 ([#53066](https://github.com/PaddlePaddle/Paddle/pull/53066)) - 优化BuildCINNPass的性能 ([#49696](https://github.com/PaddlePaddle/Paddle/pull/49696)) - 优化子图检测模块的性能 ([#45040](https://github.com/PaddlePaddle/Paddle/pull/45040), [#46937](https://github.com/PaddlePaddle/Paddle/pull/46937)) ### 硬件接入 #### CustomDevice - 训练侧新增分布式策略 MP/Sharding/PP/MoE 以及 recompute 重计算功能的支持，推理侧新增分布式策略MP的支持，支持通过CustomDevice接入的硬件昇腾NPU和寒武纪MLU无需修改任何代码即可自动继承CustomDevice新增的所有分布式策略。 [#52872](https://github.com/PaddlePaddle/Paddle/pull/52872), [#54384](https://github.com/PaddlePaddle/Paddle/pull/54384), [#53220](https://github.com/PaddlePaddle/Paddle/pull/53220), [#54572](https://github.com/PaddlePaddle/Paddle/pull/54572), [#54573](https://github.com/PaddlePaddle/Paddle/pull/54573), [#54676](https://github.com/PaddlePaddle/Paddle/pull/54676), [#53044](https://github.com/PaddlePaddle/Paddle/pull/53044), [#53719](https://github.com/PaddlePaddle/Paddle/pull/53719), [#53701](https://github.com/PaddlePaddle/Paddle/pull/53701), [#53702](https://github.com/PaddlePaddle/Paddle/pull/53702), [#53703](https://github.com/PaddlePaddle/Paddle/pull/53703) - 新增API paddle.device.is_compiled_with_custom_device，方便用户判断当前环境是否支持某硬件的插件式设备后端 [#49271](https://github.com/PaddlePaddle/Paddle/pull/49721) - 增加环境变量 CUSTOM_DEVICE_BLACK_LIST 设置，支持黑名单内的算子自动异构到CPU上运行 [#50409](https://github.com/PaddlePaddle/Paddle/pull/50409), [#50666](https://github.com/PaddlePaddle/Paddle/pull/50666) - 优化 CustomDevice 性能，减少对runtime中get_device_count接口的调用次数 [#46963](https://github.com/PaddlePaddle/Paddle/pull/46963) #### 昆仑芯XPU - 训练侧使用了新版动态图并新增分布式策略 MP/Sharding/PP 以及 recompute 重计算功能,通信库通信的支持；推理侧新增分布式策略MP的支持，并增加对XPU FasterTransformer 算子加速库的支持；[#49531](https://github.com/PaddlePaddle/Paddle/pull/49531), [#49815](https://github.com/PaddlePaddle/Paddle/pull/49815), [#48897](https://github.com/PaddlePaddle/Paddle/pull/48897), [#50717](https://github.com/PaddlePaddle/Paddle/pull/50717), [#51082](https://github.com/PaddlePaddle/Paddle/pull/51082), [#49757](https://github.com/PaddlePaddle/Paddle/pull/49757), [#51399](https://github.com/PaddlePaddle/Paddle/pull/51399), [#50329](https://github.com/PaddlePaddle/Paddle/pull/50329), [#48369](https://github.com/PaddlePaddle/Paddle/pull/48369), [#47838](https://github.com/PaddlePaddle/Paddle/pull/47838),[#48076](https://github.com/PaddlePaddle/Paddle/pull/48076),[#47882](https://github.com/PaddlePaddle/Paddle/pull/47882),[#48961](https://github.com/PaddlePaddle/Paddle/pull/48961),[#49043](https://github.com/PaddlePaddle/Paddle/pull/49043),[#49749](https://github.com/PaddlePaddle/Paddle/pull/49749),[#49806](https://github.com/PaddlePaddle/Paddle/pull/49806),[#53427](https://github.com/PaddlePaddle/Paddle/pull/53427),[#48470](https://github.com/PaddlePaddle/Paddle/pull/48470),[#49207](https://github.com/PaddlePaddle/Paddle/pull/49207),[#52296](https://github.com/PaddlePaddle/Paddle/pull/52296),[#51785](https://github.com/PaddlePaddle/Paddle/pull/51785),[#47168](https://github.com/PaddlePaddle/Paddle/pull/47168),[#47445](https://github.com/PaddlePaddle/Paddle/pull/47445),[#50200](https://github.com/PaddlePaddle/Paddle/pull/50200),[#49934](https://github.com/PaddlePaddle/Paddle/pull/49934),[#50792](https://github.com/PaddlePaddle/Paddle/pull/50792),[#52228](https://github.com/PaddlePaddle/Paddle/pull/52228),[#53337](https://github.com/PaddlePaddle/Paddle/pull/53337),[#53389](https://github.com/PaddlePaddle/Paddle/pull/53389),[#53496](https://github.com/PaddlePaddle/Paddle/pull/53496),[#53609](https://github.com/PaddlePaddle/Paddle/pull/53609),[#53697](https://github.com/PaddlePaddle/Paddle/pull/53697),[#53496](https://github.com/PaddlePaddle/Paddle/pull/53496),[#53720](https://github.com/PaddlePaddle/Paddle/pull/53720),[#53734](https://github.com/PaddlePaddle/Paddle/pull/53734),[#54172](https://github.com/PaddlePaddle/Paddle/pull/54172),[PR46227](https://github.com/PaddlePaddle/Paddle/pull/46227) ## 4. 部署方向（Paddle Inference） ### 新功能 - 支持Paddle TensorRT多个子图TensorRT engine 或者不同Predictor的之间的TensorRT engine共享显存，以便节约显存。[#45842](https://github.com/PaddlePaddle/Paddle/pull/45842) [#47631](https://github.com/PaddlePaddle/Paddle/pull/47631) - C++ API增加获取输入Tensor的Shape和数据类型接口，增加获取输出Tensor的Shape和数据类型接口。C API增加SetExecStream、EnableMkldnnInt8等C++已有接口，用于服务化部署。 [#49758](https://github.com/PaddlePaddle/Paddle/pull/49758) - 新增paddle.inference.Predictor.register_output_hook()接口，可支持调试时打印GPU推理下每层的输出，同时也支持在While等控制流模型中使用。注意此接口不支持Paddle-TensorRT。[#54433](https://github.com/PaddlePaddle/Paddle/pull/54433) ，[#47050](https://github.com/PaddlePaddle/Paddle/pull/47050) ， [#54254](https://github.com/PaddlePaddle/Paddle/pull/54254) 。 - Paddle Inference推理的Predictor接口支持paddle::Tensor作为输入和输出，以便用户直接复用飞桨动态图做推理前、后处理。 ([#50445](https://github.com/PaddlePaddle/Paddle/pull/50445)) - 增强Paddle TensorRT动态shape运行能力，config.enable_tuned_tensorrt_dynamic_shape()接口，不传任何参数时，在运行时构建TensorRT Engine。不再需要先收集shape信息再运行，但为了避免运行时的重新构建，需要在前几次运行时，覆盖最小及最大Shape的情况， [#52162](https://github.com/PaddlePaddle/Paddle/pull/52162) 。 - Paddle-TensorRT支持NHWC格式的模型输入，[#49633](https://github.com/PaddlePaddle/Paddle/pull/49633) 。 - 扩展config.Exp_DisableTensorRtOPs接口通过指定Tensor变量的名字来禁止进入TensorRT，[#49497](https://github.com/PaddlePaddle/Paddle/pull/49497) 。 ### 功能优化 - GPU混合精度推理(非Paddle TensorRT场景)功能增强，Config.enable_use_gpu增强可设置精度类型。 [#47993](https://github.com/PaddlePaddle/Paddle/pull/47993) - 支持double类型输入进行推理， [#51786](https://github.com/PaddlePaddle/Paddle/pull/51786) 。 - 由于TensorRT 算子不支持INT64类型导致模型中存在INT64数据类型式运行失败问题，Paddle-TensorRT做了增强，当模型中包含INT64数据类型时，进行自动转换，降低到INT32类型运行。 [#45547](https://github.com/PaddlePaddle/Paddle/pull/45547) - Paddle-TensorRT支持更多算子进入TensorRT推理，包含： - expand_v2，gather_nd，rsqrt，sign，not，onehot，arg_min，temporal_shift，expend_as_v2，setvalue，index_select，round，acosh，square，reduce_max，not_equal，reduce_min，reduce_prod，grid_sampler，elementwise_mod，pad3d ，greater_equal，bitwise，cumsum，matmul_v2，reciprocal，where，bmm，take_along_axis，less_than，greater_than， logical_or， logical_xor， logical_and， less_equal，range，reduce_all，reduce_any ，fill_any_like ，pow - [#47002](https://github.com/PaddlePaddle/Paddle/pull/47002) , [#47589](https://github.com/PaddlePaddle/Paddle/pull/47589) ，[#48223](https://github.com/PaddlePaddle/Paddle/pull/48223) ，[#48557](https://github.com/PaddlePaddle/Paddle/pull/48557) ， [#48655](https://github.com/PaddlePaddle/Paddle/pull/48655) ， [#49113](https://github.com/PaddlePaddle/Paddle/pull/49113) ， [#51207](https://github.com/PaddlePaddle/Paddle/pull/51207) ，[#51028](https://github.com/PaddlePaddle/Paddle/pull/51028) ，[#50341](https://github.com/PaddlePaddle/Paddle/pull/50341) ，[#51498](https://github.com/PaddlePaddle/Paddle/pull/51498) ，[#48534](https://github.com/PaddlePaddle/Paddle/pull/48534) ，[#48684](https://github.com/PaddlePaddle/Paddle/pull/48684) ， [#49393](https://github.com/PaddlePaddle/Paddle/pull/49393) ， [#49615](https://github.com/PaddlePaddle/Paddle/pull/49615) ，[#50934](https://github.com/PaddlePaddle/Paddle/pull/50934) ，[#50974](https://github.com/PaddlePaddle/Paddle/pull/50974)，[#50986](https://github.com/PaddlePaddle/Paddle/pull/50986) ， [#52000](https://github.com/PaddlePaddle/Paddle/pull/52000) ，[#51971](https://github.com/PaddlePaddle/Paddle/pull/51971) ， [#52518](https://github.com/PaddlePaddle/Paddle/pull/52518) ，[#44918](https://github.com/PaddlePaddle/Paddle/pull/44918) ，[#48230](https://github.com/PaddlePaddle/Paddle/pull/48230) ，[#47820](https://github.com/PaddlePaddle/Paddle/pull/47820) ， [#46877](https://github.com/PaddlePaddle/Paddle/pull/46877) ， [#48358](https://github.com/PaddlePaddle/Paddle/pull/48358) ， [#48592](https://github.com/PaddlePaddle/Paddle/pull/48592) ，[#48697](https://github.com/PaddlePaddle/Paddle/pull/48697) , [#53088](https://github.com/PaddlePaddle/Paddle/pull/53088) ， [#47974](https://github.com/PaddlePaddle/Paddle/pull/47974) ， [#53462](https://github.com/PaddlePaddle/Paddle/pull/53462) - 增强Paddle-TensorRT映射算子strided_slice，instance_norm，prelu，argmax，cast，nearest_interp_v2，elementwise，bilinear实现，[#46819](https://github.com/PaddlePaddle/Paddle/pull/46819) ，[#47998](https://github.com/PaddlePaddle/Paddle/pull/47998) ，[#48043](https://github.com/PaddlePaddle/Paddle/pull/48043) ，[#48998](https://github.com/PaddlePaddle/Paddle/pull/48998) ， [#49675](https://github.com/PaddlePaddle/Paddle/pull/49675) , [#47495](https://github.com/PaddlePaddle/Paddle/pull/47495) - Paddle-TensorRT部分算子(scale, square, sum, swish, expand_as_v2, prelu, gelu, hard_swish, hard_sigmoid, leaky_relu,softmax, stack, clip, cast, flatten_contiguous_range，unary，equal, elementwise_op) 支持0维Tensor，[#53660](https://github.com/PaddlePaddle/Paddle/pull/53660) ，[#53627](https://github.com/PaddlePaddle/Paddle/pull/53627) ， [#53634](https://github.com/PaddlePaddle/Paddle/pull/53634) ， [#53714](https://github.com/PaddlePaddle/Paddle/pull/53714) ， [#53729](https://github.com/PaddlePaddle/Paddle/pull/53729) ，[#53769](https://github.com/PaddlePaddle/Paddle/pull/53769) ，[#53506](https://github.com/PaddlePaddle/Paddle/pull/53506) ，[#53704](https://github.com/PaddlePaddle/Paddle/pull/53704) - 支持GCC12 + CUDA 12.0以下版本编译， [#50106](https://github.com/PaddlePaddle/Paddle/pull/50106) - Paddle-TensorRT的DeformableConv插件支持动态Shape输入，[#50698](https://github.com/PaddlePaddle/Paddle/pull/50698) - Paddle-TensorRT增加lookup_table算子的插件支持， [#46613](https://github.com/PaddlePaddle/Paddle/pull/46613) - 新增config.enable_low_precision_io()接口支持Paddle-TensorRT场景下低精度类型输入， [#52485](https://github.com/PaddlePaddle/Paddle/pull/52485) - Paddle-TensorRT的LayerNorm插件支持FP16计算， [#45043](https://github.com/PaddlePaddle/Paddle/pull/45043) - Predictor的输入数据paddle_infer::Tensor支持bool类型，[#49388](https://github.com/PaddlePaddle/Paddle/pull/49388) - Paddle-TensorRT增强Convolution实现采用ConvolutionNd，[#47653](https://github.com/PaddlePaddle/Paddle/pull/47653) - conv2d_fusion融合算子支持NHWC格式，[#49047](https://github.com/PaddlePaddle/Paddle/pull/49047) - 调整C++推理库下Phi算子相关目录结构，[#53091](https://github.com/PaddlePaddle/Paddle/pull/53091) - 当TensorRT序列化和加载版本不匹配时，支持重新构建TensorRT Engine，而不是报错，[#50775](https://github.com/PaddlePaddle/Paddle/pull/50775) 。 - 优化Paddle-TensorRT运行时打印日志信息，[#50181](https://github.com/PaddlePaddle/Paddle/pull/50181) - 基于oneDNN的CPU推理支持elementwise的0维Tensor输入，[#51656](https://github.com/PaddlePaddle/Paddle/pull/51656) - 清理和规范化Paddle-TensorRT的FC、matmul、matmul_v2算子的支持，统一升级到使用TensorRT的IMatrixMultiplyLayer进行支持，[#52222](https://github.com/PaddlePaddle/Paddle/pull/52222) ### 性能提升 - 支持多个lookup_tables进入Paddle-TensorRT的Embedding+Eltwise+LayerNorm的融合 [#46243](https://github.com/PaddlePaddle/Paddle/pull/46243) ，[#46230](https://github.com/PaddlePaddle/Paddle/pull/46230) - 增加MoE融合Phi算子，提升MoE模型性能推理性能， [#48703](https://github.com/PaddlePaddle/Paddle/pull/48703) - 在INT8量化推理的场景下，Paddle-TensorRT 插件fallback到FP16计算而不是FP32计算，[#50554](https://github.com/PaddlePaddle/Paddle/pull/50554) - 优化推理时内存、显存， [#49051](https://github.com/PaddlePaddle/Paddle/pull/49051) ， [#49046](https://github.com/PaddlePaddle/Paddle/pull/49046) ，[#53930](https://github.com/PaddlePaddle/Paddle/pull/53930) - Layout排布优化Pass增强， [#52997](https://github.com/PaddlePaddle/Paddle/pull/52997) - 支持对算子Shape推断进行缓存，提升模型推理性能， [#48312](https://github.com/PaddlePaddle/Paddle/pull/48312) - 使用half2指令优化bias+add+relu融合，[#49048](https://github.com/PaddlePaddle/Paddle/pull/49048) - 使用向量化操作优化多个输入的Concat Kernel，[#49540](https://github.com/PaddlePaddle/Paddle/pull/49540) - 基于CUTLASS实现Convolution、Depthwise Convolution及相关融合算子，提升推理速度。 [#47989](https://github.com/PaddlePaddle/Paddle/pull/47989) ，[#50603](https://github.com/PaddlePaddle/Paddle/pull/50603) ，[#51792](https://github.com/PaddlePaddle/Paddle/pull/51792) ，[#50603](https://github.com/PaddlePaddle/Paddle/pull/50603) - Paddle-TensorRT支持FlashAttention的插件，提升StableDiffusion等模型的推理速度，[#49438](https://github.com/PaddlePaddle/Paddle/pull/49438) 。 - 增加Transpose+LayerNorm的融合PASS，提升StableDiffusion等模型的推理速度，[#50082](https://github.com/PaddlePaddle/Paddle/pull/50082) 。 - 增加Elementwise+Transpose的融合，[#50081](https://github.com/PaddlePaddle/Paddle/pull/50081) - 优化Paddle-TensorRT Group Norm插件实现，[#49160](https://github.com/PaddlePaddle/Paddle/pull/49160) - Config.EnableTensorRtEngine()接口增加use_cuda_graph参数，可以支持开启CUDA Graph，注意在使用时，需要保证模型输入shape不变，可以降低运行时耗时，[#53406](https://github.com/PaddlePaddle/Paddle/pull/53406) - 支持对Reshape的inplace操作减少模型运行时的拷贝耗时， [#49146](https://github.com/PaddlePaddle/Paddle/pull/49146) - 基于oneDNN优化LayerNorm kernel实现，[#47782](https://github.com/PaddlePaddle/Paddle/pull/47782) - 基于oneDNN支持quantize+transpose 以及 transpose+dequantize融合，[#49509](https://github.com/PaddlePaddle/Paddle/pull/49509) - CPU推理下当开启MKLDNN时，默认开启FC相关的融合Pass，提升性能，[#45704](https://github.com/PaddlePaddle/Paddle/pull/45704) - CPU的OneDNN推理支持suqeeze2 + transpose2融合，[#47592](https://github.com/PaddlePaddle/Paddle/pull/47592) ### XPU推理提升和性能优化 - 新增 ExpRunWithRuntimeConfig 接口与 XpuRuntimeConfig 允许推理期间设置外部流、L3 cache 等参数；GetExecStream 接口支持获得昆仑外部流对象；输入、输出支持昆仑设备内存减少 D2H 和 H2D 开销，[#53334](https://github.com/PaddlePaddle/Paddle/pull/53334)、 [#52466](https://github.com/PaddlePaddle/Paddle/pull/52466)、 [#53240](https://github.com/PaddlePaddle/Paddle/pull/53240) - 新增 multi-encoder, fused_multi_transformer 算子和融合 pass，提升 ERNIE 和 Transformer 类模型性能，[#50570](https://github.com/PaddlePaddle/Paddle/pull/50570)、[#51346](https://github.com/PaddlePaddle/Paddle/pull/51346)、 [#50499](https://github.com/PaddlePaddle/Paddle/pull/50499)、[#53982](https://github.com/PaddlePaddle/Paddle/pull/53982)、[#50759](https://github.com/PaddlePaddle/Paddle/pull/50759)、[#51571](https://github.com/PaddlePaddle/Paddle/pull/51571)、 [#53144](https://github.com/PaddlePaddle/Paddle/pull/53144)、[#53306](https://github.com/PaddlePaddle/Paddle/pull/53306) - 优化BeamSearch性能，当beam_size=1 时对 write_read_array, gather 等细粒度算子进行变换、去除和融合提升模型性能，[#53130](https://github.com/PaddlePaddle/Paddle/pull/53130) - 多个相同输入的 stack 算子变换为支持 broadcast 的 unsqueeze 算子，unsquee/squeeze 支持 inplace 计算， [#52099](https://github.com/PaddlePaddle/Paddle/pull/52099) - 新增支持导出适用于昆仑芯的多卡推理模型， [#50490](https://github.com/PaddlePaddle/Paddle/pull/50490) - 新增 embedding_with_eltwise_add 融合 pass 及算子 phi kernel，减小显存占用并提升推理性能， [#50590](https://github.com/PaddlePaddle/Paddle/pull/50590) - interpolate 类算子 phi kernel 支持 FP16， [#52358](https://github.com/PaddlePaddle/Paddle/pull/52358) - argmax 算子支持 INT32 类型输出， [#51303](https://github.com/PaddlePaddle/Paddle/pull/51303) - 修复开启混合精度推理模式后, 保存序列化模型时只有model文件时的报错， [#52994](https://github.com/PaddlePaddle/Paddle/pull/52994) - 修复 instance_norm 在 scale 和 bias 为空时出现的段错误， [#52627](https://github.com/PaddlePaddle/Paddle/pull/52627) - conv_transpose 算子支持 FP16，[#53626](https://github.com/PaddlePaddle/Paddle/pull/53626) - 添加 yolo_box_xpu 融合 pass 及算子 phi kernel，优化 YOLO 模型通用子结构， [#54163](https://github.com/PaddlePaddle/Paddle/pull/54163) - 添加 conv2d_xpu 融合 pass 以及算子 phi kernel，并支持FP16推理，优化卷积操作推理耗时，[#52247](https://github.com/PaddlePaddle/Paddle/pull/52247) ，[#53626](https://github.com/PaddlePaddle/Paddle/pull/53626) - 添加 sigmoid_elementmul 通用融合 pass，融合为 swish 算子以匹配 conv2d_fusion pass 提升 YOLO 模型推理性能， [#53580](https://github.com/PaddlePaddle/Paddle/pull/53580) - 添加 act_add 融合 pass 及算子 phi kernel 提升推理性能，[#53965](https://github.com/PaddlePaddle/Paddle/pull/53965) - 添加 fold_interp_outsize 融合 pass 提升推理性能， [#54245](https://github.com/PaddlePaddle/Paddle/pull/54245) - 解决当FC存在共享 weight 时因重复融合导致结果错误的问题。 [#51108](https://github.com/PaddlePaddle/Paddle/pull/51108)、[#51039](https://github.com/PaddlePaddle/Paddle/pull/51039) - 删除算子仅用于训练的 op_device 属性，防止在推理期间错误的选择训练时的 place， [#51029](https://github.com/PaddlePaddle/Paddle/pull/51029) - 支持优化后模型的保存，允许再次推理时跳过 PASS优化减少第一次推理时间， [#53696](https://github.com/PaddlePaddle/Paddle/pull/53696) - 解决算子 Kernel 的 CPUPlace 输入被强制拷贝到 XPU 而导致的计算错误问题， [#51306](https://github.com/PaddlePaddle/Paddle/pull/51306) - subblock 支持参数 H2D 提前拷贝以提升推理性能。[#51876](https://github.com/PaddlePaddle/Paddle/pull/51876) - 修复昆仑芯 2 代芯片输出激活的 scale 存储空间大小。 [#53505](https://github.com/PaddlePaddle/Paddle/pull/53505) - 新执行器昆仑芯 D2D 拷贝支持异步执行， [#51876](https://github.com/PaddlePaddle/Paddle/pull/51876) - 删除只有一个输入的 concat 算子，[#52304](https://github.com/PaddlePaddle/Paddle/pull/52304) - lookup_table_v2 支持 FP16 删除冗余 cast 算子， [#52888](https://github.com/PaddlePaddle/Paddle/pull/52888) - 控制流While算子支持缓存scope，降低每次新建scope 的开销， [#52628](https://github.com/PaddlePaddle/Paddle/pull/52628) - scatter 新增支持 FP16，删除冗余 cast 算子以及某一个输入为 1 的 elementwise_mul 算子。[#52831](https://github.com/PaddlePaddle/Paddle/pull/52831) ### 模型量化 - 动态图量化功能全面升级 - 新增动态图模型下量化训练的API为 ```paddle.quantization.QAT``` ，支持通过配置传入量化相关参数，简化量化训练使用流程和二次开发难度 ([#49398](https://github.com/PaddlePaddle/Paddle/pull/49398)) - 新增离线量化的API为 ```paddle.quantization.PTQ``` ，支持量化模型导出成推理支持的模型格式 ([#50107](https://github.com/PaddlePaddle/Paddle/pull/50107)) - 新增STUB算子，在训练过程中模拟实际的量化操作([#50510](https://github.com/PaddlePaddle/Paddle/pull/50510)) - 支持量化训练模型加载离线量化模型的参数，支持更多算子量化，包含matmul, scale,conv1d，[#47892](https://github.com/PaddlePaddle/Paddle/pull/47892)， [#45911](https://github.com/PaddlePaddle/Paddle/pull/45911)，[#48912](https://github.com/PaddlePaddle/Paddle/pull/48912) - 支持静态图量化训练的混合并行训练，[#52219](https://github.com/PaddlePaddle/Paddle/pull/52219) - 修复动态图量化过程中的问题： - 导出量化训练模型时候重复插入量化节点，[#48751](https://github.com/PaddlePaddle/Paddle/pull/48751) - 修复给模型输入插入量化节点的问题，[#49926](https://github.com/PaddlePaddle/Paddle/pull/49926) ## 5. 环境适配为提升源码编译效率，完善和推广setuptools + ninja编译方式，提升开发效率，CPU场景下，全量编译耗时减少20min，编译速度提升24.52%，GPU场景下全量编译耗时减少22min，编译速度提升29.31%; 为了适配较为主流的开发环境，飞桨在源码编译支持了gcc12编译和C++17标准，适配了最新的Python3.11 和 CUDA12；代码质量完成了编译warning的清理，提升编译体验；第三方依赖层级，为减少依赖冲突，升级了底层的protobuf版本，并清理了一些低版本依赖库的废弃属性和老旧的代码格式，并移除了对于python2.x的支持。 - ninja编译适配，提升编译速度。[#52433](https://github.com/PaddlePaddle/Paddle/pull/52433),[#48932](https://github.com/PaddlePaddle/Paddle/pull/48932),[#49420](https://github.com/PaddlePaddle/Paddle/pull/49420),[#48435](https://github.com/PaddlePaddle/Paddle/pull/48435),[#49303](https://github.com/PaddlePaddle/Paddle/pull/49303),[#49448](https://github.com/PaddlePaddle/Paddle/pull/49448),[#49838](https://github.com/PaddlePaddle/Paddle/pull/49838),[#50067](https://github.com/PaddlePaddle/Paddle/pull/50067),[#52796](https://github.com/PaddlePaddle/Paddle/pull/52796),[#50431](https://github.com/PaddlePaddle/Paddle/pull/50431),[#49181](https://github.com/PaddlePaddle/Paddle/pull/49181),[#48867](https://github.com/PaddlePaddle/Paddle/pull/48867),[#48490](https://github.com/PaddlePaddle/Paddle/pull/48490),[#48211](https://github.com/PaddlePaddle/Paddle/pull/48211),[#49499](https://github.com/PaddlePaddle/Paddle/pull/49499),[#53076](https://github.com/PaddlePaddle/Paddle/pull/53076) - setuptools编译打包一体化适配。[#48770](https://github.com/PaddlePaddle/Paddle/pull/48770),[#46957](https://github.com/PaddlePaddle/Paddle/pull/46957),[#49583](https://github.com/PaddlePaddle/Paddle/pull/49583),[#47602](https://github.com/PaddlePaddle/Paddle/pull/47602),[#48301](https://github.com/PaddlePaddle/Paddle/pull/48301),[#50800](https://github.com/PaddlePaddle/Paddle/pull/50800),[#42575](https://github.com/PaddlePaddle/Paddle/pull/42575)),[#49826](https://github.com/PaddlePaddle/Paddle/pull/49826),[#49002](https://github.com/PaddlePaddle/Paddle/pull/49002),[#51443](https://github.com/PaddlePaddle/Paddle/pull/51443),[#51528](https://github.com/PaddlePaddle/Paddle/pull/51528),[#52621](https://github.com/PaddlePaddle/Paddle/pull/52621),[#52465](https://github.com/PaddlePaddle/Paddle/pull/52465) - python 3.11 支持。[#42126](https://github.com/PaddlePaddle/Paddle/pull/42126),[#54273](https://github.com/PaddlePaddle/Paddle/pull/54273),[[#51350](https://github.com/PaddlePaddle/Paddle/pull/51350) - gcc12 支持。[#52960](https://github.com/PaddlePaddle/Paddle/pull/52960),[#52265](https://github.com/PaddlePaddle/Paddle/pull/52265),[#46546](https://github.com/PaddlePaddle/Paddle/pull/46546),[#52318](https://github.com/PaddlePaddle/Paddle/pull/52318),[#46808](https://github.com/PaddlePaddle/Paddle/pull/46808),[#47466](https://github.com/PaddlePaddle/Paddle/pull/47466),[#52083](https://github.com/PaddlePaddle/Paddle/pull/52083),[#48176](https://github.com/PaddlePaddle/Paddle/pull/48176),[#49423](https://github.com/PaddlePaddle/Paddle/pull/49423),[#49452](https://github.com/PaddlePaddle/Paddle/pull/49452),[#51037](https://github.com/PaddlePaddle/Paddle/pull/51037),[#52007](https://github.com/PaddlePaddle/Paddle/pull/52007),[#52441](https://github.com/PaddlePaddle/Paddle/pull/52441),[#52085](https://github.com/PaddlePaddle/Paddle/pull/52085),[#50817](https://github.com/PaddlePaddle/Paddle/pull/50817),[#52646](https://github.com/PaddlePaddle/Paddle/pull/52646),[#50777](https://github.com/PaddlePaddle/Paddle/pull/50777),[#53288](https://github.com/PaddlePaddle/Paddle/pull/53288),[#54009](https://github.com/PaddlePaddle/Paddle/pull/54009) - c++17标准支持。[#53345](https://github.com/PaddlePaddle/Paddle/pull/53345),[#53892](https://github.com/PaddlePaddle/Paddle/pull/53892),[#54282](https://github.com/PaddlePaddle/Paddle/pull/54282),[#49017](https://github.com/PaddlePaddle/Paddle/pull/49017),[#47635](https://github.com/PaddlePaddle/Paddle/pull/47635),[#54258](https://github.com/PaddlePaddle/Paddle/pull/54258) - cuda12支持。[#52285](https://github.com/PaddlePaddle/Paddle/pull/52285),[#49592](https://github.com/PaddlePaddle/Paddle/pull/49592),[#52232](https://github.com/PaddlePaddle/Paddle/pull/52232),[#52654](https://github.com/PaddlePaddle/Paddle/pull/52654),[#54641](https://github.com/PaddlePaddle/Paddle/pull/54641) - CodeStyle。[#45909](https://github.com/PaddlePaddle/Paddle/pull/45909),[#47772](https://github.com/PaddlePaddle/Paddle/pull/47772),[#48538](https://github.com/PaddlePaddle/Paddle/pull/48538),[#49522](https://github.com/PaddlePaddle/Paddle/pull/49522),[#47264](https://github.com/PaddlePaddle/Paddle/pull/47264),[#49558](https://github.com/PaddlePaddle/Paddle/pull/49558) - 编译Warning消除。[#47163](https://github.com/PaddlePaddle/Paddle/pull/47163),[#47216](https://github.com/PaddlePaddle/Paddle/pull/47216),[#47309](https://github.com/PaddlePaddle/Paddle/pull/47309)，[#47252](https://github.com/PaddlePaddle/Paddle/pull/47252)，[#47341](https://github.com/PaddlePaddle/Paddle/pull/47341)，[#47399](https://github.com/PaddlePaddle/Paddle/pull/47399)，[#47513](https://github.com/PaddlePaddle/Paddle/pull/47513)，[#47558](https://github.com/PaddlePaddle/Paddle/pull/47558)，[#47706](https://github.com/PaddlePaddle/Paddle/pull/47706)，[#52717](https://github.com/PaddlePaddle/Paddle/pull/52717)，[#51203](https://github.com/PaddlePaddle/Paddle/pull/51203)，[#51336](https://github.com/PaddlePaddle/Paddle/pull/51336)，[#51608](https://github.com/PaddlePaddle/Paddle/pull/51608)，[#51633](https://github.com/PaddlePaddle/Paddle/pull/51633),[#46644](https://github.com/PaddlePaddle/Paddle/pull/46644),[#53092](https://github.com/PaddlePaddle/Paddle/pull/53092),[#53185](https://github.com/PaddlePaddle/Paddle/pull/53185),[#53246](https://github.com/PaddlePaddle/Paddle/pull/53246),[#53650](https://github.com/PaddlePaddle/Paddle/pull/53650),[#53683](https://github.com/PaddlePaddle/Paddle/pull/53683),[#53687](https://github.com/PaddlePaddle/Paddle/pull/53687),[#53886](https://github.com/PaddlePaddle/Paddle/pull/53886),[#53689](https://github.com/PaddlePaddle/Paddle/pull/53689),[#53679](https://github.com/PaddlePaddle/Paddle/pull/53679),[#53681](https://github.com/PaddlePaddle/Paddle/pull/53681),[#53532](https://github.com/PaddlePaddle/Paddle/pull/53532),[#47137](https://github.com/PaddlePaddle/Paddle/pull/47137),[#47045](https://github.com/PaddlePaddle/Paddle/pull/47045),[#52186](https://github.com/PaddlePaddle/Paddle/pull/52186),[#52490](https://github.com/PaddlePaddle/Paddle/pull/52490),[#53924](https://github.com/PaddlePaddle/Paddle/pull/53924),[#53938](https://github.com/PaddlePaddle/Paddle/pull/53938),[#53945](https://github.com/PaddlePaddle/Paddle/pull/53945),[#53851](https://github.com/PaddlePaddle/Paddle/pull/53851),[#53847](https://github.com/PaddlePaddle/Paddle/pull/53847),[#53818](https://github.com/PaddlePaddle/Paddle/pull/53818),[#53931](https://github.com/PaddlePaddle/Paddle/pull/53931) - 支持protobuf升级。[#49875](https://github.com/PaddlePaddle/Paddle/pull/49875),[#48495](https://github.com/PaddlePaddle/Paddle/pull/48495),[#49673](https://github.com/PaddlePaddle/Paddle/pull/49673),[#52499](https://github.com/PaddlePaddle/Paddle/pull/52499),[#51161](https://github.com/PaddlePaddle/Paddle/pull/51161),[#49168](https://github.com/PaddlePaddle/Paddle/pull/49168) - 支持第三方库离线编译。[#54326](https://github.com/PaddlePaddle/Paddle/pull/54326),[#54370](https://github.com/PaddlePaddle/Paddle/pull/54370),[#54335](https://github.com/PaddlePaddle/Paddle/pull/54335),[#54346](https://github.com/PaddlePaddle/Paddle/pull/54346),[#53744](https://github.com/PaddlePaddle/Paddle/pull/53744),[#54319](https://github.com/PaddlePaddle/Paddle/pull/54319),[#53915](https://github.com/PaddlePaddle/Paddle/pull/53915) - phi独立编译头文件依赖解耦。[#50456](https://github.com/PaddlePaddle/Paddle/pull/50456),[#47088](https://github.com/PaddlePaddle/Paddle/pull/47088),[#52573](https://github.com/PaddlePaddle/Paddle/pull/52573),[#52651](https://github.com/PaddlePaddle/Paddle/pull/52651) - Python2.x 退场。[#48685](https://github.com/PaddlePaddle/Paddle/pull/48685) ## 6. 安全 - 修复了诸如空指针使用、非法地址访问、内存越界、除0、Python IndexError等问题。[PR49976](https://github.com/PaddlePaddle/Paddle/pull/49976), [ PR49993](https://github.com/PaddlePaddle/Paddle/pull/49993)[, PR49942](https://github.com/PaddlePaddle/Paddle/pull/49942), [PR49965](https://github.com/PaddlePaddle/Paddle/pull/49965)[, PR50000](https://github.com/PaddlePaddle/Paddle/pull/50000)[, PR50005](https://github.com/PaddlePaddle/Paddle/pull/50005)[, PR49953](https://github.com/PaddlePaddle/Paddle/pull/49953)[, PR49995](https://github.com/PaddlePaddle/Paddle/pull/49995)[, PR49974](https://github.com/PaddlePaddle/Paddle/pull/49974)[, PR50015](https://github.com/PaddlePaddle/Paddle/pull/50015)[, PR50010](https://github.com/PaddlePaddle/Paddle/pull/50010), [PR49979](https://github.com/PaddlePaddle/Paddle/pull/49979), [PR49994](https://github.com/PaddlePaddle/Paddle/pull/49994), [PR49977](https://github.com/PaddlePaddle/Paddle/pull/49977)[, PR49968](https://github.com/PaddlePaddle/Paddle/pull/49968), [PR49984](https://github.com/PaddlePaddle/Paddle/pull/49984)[, PR49958](https://github.com/PaddlePaddle/Paddle/pull/49958)[, PR50008](https://github.com/PaddlePaddle/Paddle/pull/50008)[, PR51714](https://github.com/PaddlePaddle/Paddle/pull/51714), [PR51847](https://github.com/PaddlePaddle/Paddle/pull/51847), [PR51034](https://github.com/PaddlePaddle/Paddle/pull/51034)[, PR51088](https://github.com/PaddlePaddle/Paddle/pull/51088)[, PR51091](https://github.com/PaddlePaddle/Paddle/pull/51091)[, PR51092](https://github.com/PaddlePaddle/Paddle/pull/51092), [PR49966](https://github.com/PaddlePaddle/Paddle/pull/49966), [PR49656](https://github.com/PaddlePaddle/Paddle/pull/49656), [PR52161](https://github.com/PaddlePaddle/Paddle/pull/52161), [PR49548](https://github.com/PaddlePaddle/Paddle/pull/49548), [PR49546](https://github.com/PaddlePaddle/Paddle/pull/49546), [PR49547](https://github.com/PaddlePaddle/Paddle/pull/49547), [PR49549](https://github.com/PaddlePaddle/Paddle/pull/49549), [PR51850](https://github.com/PaddlePaddle/Paddle/pull/51850) ## Thanks to our Contributors This release contains contributions from: 1want2sleep, 201716010711, 404988613, 5u13, 6clc, Ackeraa, Aganlengzi, ahahahahahaha, Ainavo, Allen Guo, andyj, Asthestarsfalll, Aurelius84, Ayuan, BellaZYL, Bjmw3, Bo Zhang, bukejiyu, caozhou, carryyu, Ccc, ccrrong, ceci3, chalsliu, Chang Xu, CHANGer, Charles-hit, Chen Weihang, chenjian, Chenxiao Niu, chenxiao120660, chenxujun, Chitsing KUI, cifar10, co63oc, CollaborativeFiltering, csy0225, cxxly, cyber-pioneer, cyberslack_lee, czr-gc, Dandelight, danleifeng, Danyang Zhang, dasen, denglianbin, Difer, dongfangshenzhu, DrowFish19, duanboqiang, duanyanhui, engineer, engineer1109, Epsilon Luoo, feifei-111, Feiyu Chan, Feng Ni, feng_shuai, Fisher, FlyingQianMM, Frank Lin, Galaxy1458, GaoYuYang, gaoziyuan, gem5, GGBond8488, Ghost Screaming, gongenlei, gouzil, Guanghua Yu, Guo Sheng, Guoxia Wang, Hamid Zare, Hanchiao, handiz, Haohongxiang, haosicheng, haozi, Happyd99, heliqi, hellockx, hellolllw, heyanru, hg-1099255210, hh-qiao, hjyp, hong, HongyuJia, houj04, hua-zi, Huang Jiyi, Huang Zhengjie, huangjiyi, huangjun12, Hui Zhang, Huihuang Zheng, Hulek, hwa, HydrogenSulfate, Ikko Eltociear Ashimine, iLeGend, Infinity_lee, Infrared1029, Jacek Czaja, jakpiase, james, jameszhang, Jiabin Yang, jiahongyu, jiangcheng, jiangfan06, Jianghai, jiaqianjing, jingsongliu, JingZhuangzhuang, jjyaoao, joanna.wozna.intel, junxiu777, Jx-qi, JYChen, JZ-LIANG, jzhang533, Kai Song, Kai Xing, Kaipeng Deng, Kang Zhao, kangguangli, Kevin吴嘉文, Kim, Kim Yann, knamg, kuizhiqing, lanxianghit, Leding Li, Leo Chen, Leo Guo, levi131, Li Min, Li-fAngyU, Ligoml, lijialin03, lijin23, limingshu, Lin Manhui, LinearTemporalLogic, Linjie Chen, lishicheng1996, Little-chick, littleforest, liu zhengxi, liulinduo, liuruyan, liuzhenhai93, LiYuRio, lj970926, LokeZhou, LoneRanger, lubiu, Lucas, lugimzzz, Lux et Veritas, lxsbupt, LyndonKong, lzy, lzydev, Mahmoud Ashraf, Manan Goel, Maple Xie, Matsumoto Ruko, mayang002, MayYouBeProsperous, megemini, mengziheng, Meteor Liu, mhy, mhy-666, Ming-Xu Huang, ming1753, minghaoBD, mjxs, Moqim, Mountagha, Mr.Juice, mrcangye, NetPunk, Netpunk, nihao, niuliling123, Nyakku Shigure, OccupyMars2025, Ouyang Chao, pangengzheng, pangyoki, parap1uie-s, Paulina Gacek, Piotr Paturej, PommesPeter, PPGitub, PPPPzhang, PuQing, Qi Li, Qi Shao, QingshuChen, qipengh, qizhaoaoe, Rayman, RedContritio, RichardWooSJTU, risemeup1, Roc, ronnywang, Ruibiao Chen, Ruibin Cheung, RuohengMa, Ryan, SaltFish11, Sanbu, Scotty, scotty, seemingwang, Shaojie WANG, ShenLiang, shentanyue, Shijie, Shuangchi He, Siming Dai, Sing_chan, sneaxiy, Sonder, sprouteer, Sqhttwl, sunli, superwinner1, supplyout, SylarTiaNII, Sylwester Fraczek, Sławomir Siwek, taixiurong, Tao Luo, Taylor-Layrose, TeFeng Chen, Thomas Young, thunder95, Thunderbrook, Tian, Tian Zheng, tiancaishaonvjituizi, tianshuo78520a, tifa, Tinson Lai, Tomasz Socha, Tony Cao, ucsk, umiswing, ustiniankw, Vegetable dog, Vigi Zhang, Vvsmile, Wang Bojun, Wang Xin, Wang Xinyu, wangfengsheng1999, wangguanqun, wangguanzhong, wanghuancoder, wangna11BD, wangshengxiang, wangxiaoning, wangxinxin08, Wangzheee, WangZhen, wangzhen38, wasupandceacar, wawltor, Wei Shengyu, Weilong Wu, weishengying, Wen Sun, wenbin, wentao yu, wenzhe.wang, westfish, whisky-12, whs, Wilber, will-jl944, winter-wang, Winters Montagne, WJJ1995, wuhuachaocoding, wuyefeilin, wz1qqx, XiangGao, xiaoguoguo626807, xiaohemaikoo, xiaoluomi, xiaoting, xiaoxiaohehe001, Xiaoxu Chen, xiaoyuanzi914, Xinger, Xinyu Chen, xiongkun, xjmxyt, xu98bin, xysheng-baidu, yangguohao, yangjianfengo1, YangQun, YangZhou, yeliang2258, YepKong, Yichen Zhang, yikaikkk, Yiqun Liu, yjphhw, ykkk2333, Young-Flash, yu wentao, Yuang Liu, Yuanle Liu, YuanRisheng, yuchen202, yuehuayingxueluo, YuhangLi, Yulong Ao, YUNSHEN XIE, yunyaoXYY, YuRonan, zachary sun, ZeKai Zhou, Zenghui Yuan, zengshao0622, Zero Rains, Zhan Rongrui, Zhang Jun, Zhang Na, Zhang Ting, Zhang Zheng, zhangbo9674, ZhangDY-6483, zhangkaihuo, zhangxin81, zhangyikun02, zhangyingying520, zhangyuqin1998, zhaocaibei123, zhaoyingli, Zhen Wang, Zheng-Bicheng, Zhenghai Zhang, Zheng_Bicheng, zhenyun, Zhibao Li, zhiboniu, Zhong Hui, Zhou Wei, ZhouMengLei1999, zhoutianzi666, zhouzj, zhupengyang, zhurou603, zhuyipin, zhwesky2010, ziyoujiyi, zlsh80826, Zman, zmxdream, zqw_1997, Zuza Gawrysiak, zxcd, zyfncg, ZZK, zzk0, 丁一, 傅剑寒, 六个骨头, 卢林, 周周周, 姜永久, 学渣戊, 张春乔, 张正海, 柠檬味~, 王明冬, 石晓伟, 超级码牛, 陈沧夜, 骑马小猫