Improve token usage recording

[antonpirker]

Update token usage recording to work if the LLM is calling them prompt_tokens or input_tokens. Same for completion_tokens and output_tokens. Records also cached and reasoning tokens usage.

Because the signature of a helper function was changed, other AI integrations also have changes.

[codecov]

❌ 101 Tests Failed:

Tests completed	Failed	Passed	Skipped
24112	101	24011	5814

View the top 3 failed test(s) by shortest run time

tests.integrations.anthropic.test_anthropic::test_add_ai_data_to_span_with_input_json_delta

Stack Traces | 0.067s run time

.../integrations/anthropic/test_anthropic.py:814: in test_add_ai_data_to_span_with_input_json_delta
    assert span._measurements.get("ai_prompt_tokens_used")["value"] == 10
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E   TypeError: 'NoneType' object is not subscriptable

tests.integrations.anthropic.test_anthropic::test_add_ai_data_to_span_with_input_json_delta

Stack Traces | 0.078s run time

.../integrations/anthropic/test_anthropic.py:814: in test_add_ai_data_to_span_with_input_json_delta
    assert span._measurements.get("ai_prompt_tokens_used")["value"] == 10
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E   TypeError: 'NoneType' object is not subscriptable

tests.integrations.huggingface_hub.test_huggingface_hub::test_streaming_chat_completion[True-False-True]

Stack Traces | 0.085s run time

.../integrations/huggingface_hub/test_huggingface_hub.py:137: in test_streaming_chat_completion
    assert span["measurements"]["ai_total_tokens_used"]["value"] == 10
           ^^^^^^^^^^^^^^^^^^^^
E   KeyError: 'measurements'

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}