add some new ops, fix some operators and add batch operations to certain operators.

[leejet]

I tested it in the sd.cpp project, and everything works well.

[FSSRepo]

@leejet I believe that before merging this, the respective Metal kernels should be created to maintain compatibility, although it doesn't really help much since sd.cpp doesn't benefit from the Metal backend as matrix operations are very slow.

[leejet]

OK, I will find time to add support for Metal backend.

[leejet]

@FSSRepo I have implemented Metal backend support for ggml_arange and ggml_timestep_embedding, and tested it with the test-arrange and test-timestep-embedding test cases to ensure it works correctly. Could you find some time to review and merge this PR?

[FSSRepo]

Ok, I will test this PR with latest sd.cpp

[slaren]

• Does arange affect the performance significantly? Are there any downsides to constructing the tensor in a CPU buffer and copying it to the backend as an input?
• Does timestep_embedding have any use outside of SD? And again, would it hurt performance significantly if it was implemented on the CPU and copied to the backend as an input?

[leejet]

These two operations won't significantly improve performance, but in stable video diffusion, it's necessary to perform these operations multiple times within the UNet network. Additionally, the parameters may change with each execution. If I don't execute these operations when computing the graph, I would have to remember the tensors produced by each operation and their corresponding parameters. Then, during the ggml_gallocr_alloc_graph, I'd have to copy the data, making the process more complex. Implementing this through ggml operators would make the code more elegant.

[slaren]

It would be good to have tests for these operators in test-backend-ops.

[ggerganov]

After adding tests to test-backend-ops we can merge

[ggerganov]

What was the reason to change these normalization to be inside the loop?

[leejet]

Because previously, it calculated the sum of (v * v) and then divided by (ne00 * ne01 * step), I was concerned that in certain situations, it might lead to overflow or precision issues. Therefore, I changed it to calculate the sum of (v * v)/(ne00 * ne01 * step) to obtain the variance.

[ggerganov]

It shouldn't cause problems since we accumulate into double (i.e. ggml_float). I've introduced a separate accumulator per row and restored the normalization to be at the end in order to avoid extra division in the loop

[leejet]

I have modified the code based on the suggestions from the review and added tests to test-backend-ops. There are two uncertain points still awaiting feedback from the maintainer.