SiLU/Swish activation function
SiLU is added as a non-fusable activation, similar to Sine.
The main motivation for this is to expand the range of supported activations, especially since SiLU is the default choice for some flow matching and diffusion model architectures.
I haven't been able to thoroughly test this, but, since it is a simple change, this PR should be fast to review.
I have only empirical good results from the sample code of fitting to einstein's image, shown below.
[einstein_training_silu.zip](https://github.com/user-attachments/files/21616360/einstein_training_silu.zip)
合并状态:未合并 1 条评论