New integration - CometLogger

[Lothiraldan]

Context

This PR introduces support for logging metrics to Comet https://www.comet.com/site/.

Changelog

• Add torchtune.utils.metric_logging.CometLogger
• Add unit tests
• Update documentation and reference

Test plan

• run pre-commit hooks and linters (make sure you've first installed via pre-commit install)
• [ x add unit tests for any new functionality
• [ ]xupdate docstrings for any new or updated methods or classes
• run unit tests via pytest tests
• run recipe tests via pytest tests -m integration_test

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/910

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[facebook-github-bot]

Hi @Lothiraldan!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at [email protected]. Thanks!

[kartikayk]

@Lothiraldan thanks so much for the PR! Really excited about new community integrations. Do you mind sharing more context on why this logger is needed and what problem it solves?

Highlighting ecosystem integrations is one of the core principles of torchtune and something we believe makes the ecosystem stronger.

We started with integrations for WandB and Tensorboard since these have come up as feature requests in the user conversations we had. Generally, we'd like to get more user signal into different integrations before we move them to the core repo.

Our general mode of operation is:

• Land integration in a fork
• Highlight the integration in the new community integration section we're adding to the README (PR should be up in a day or 2)
• As more folks look at this and we get requests to merge this into main, we can pull in this change

Let me know if this makes sense.

[sherpan]

Hi, Jumping in since @Lothiraldan is on PTO! My name is Sid and I'm a ML Growth Engineer at Comet.

We opened the PR since we had two enterprise customers asking for it. They’re currently using Transformers and looking to switch to torchtune. We’re happy to connect them directly to the torchtune team if that helps.

There’s >100k researchers using Comet on cloud and given how disruptive TorchTune is we expect a lot of them will want to switch but will be blocked. The rest of the core training frameworks usually define a logging Abstract Base Class and we create/maintain a Comet Subclass for logging metrics and assets to Comet. It’s worked well in the Pytorch Lightning, Transformers, Composer libraries: all which have multiple logging options for users.

We could create a fork and ask our Comet users to go there but i think it adds another layer of complexity for our mutual users and customers. Would really appreciate if you can consider a similar approach to other frameworks and happy to help on the implementation side.

[Lothiraldan]

Hi @kartikayk, sorry I didn't introduce myself. I'm Boris Integration Product Manager at Comet. We started working on the Torchtune integrations after getting questions about it from our customers.

I was able to catch up with my team on your suggestion. I am not a big fan of landing the integration inside a fork as it would require users to install transformers in a different way to use the integration. My team suggested that we package the Torchtune integration inside our Python SDK at first. Users would then be able to use the integration like this:

    metric_logger:
      _component_: comet_ml.integration.torchtune.CometLogger
      # the Comet project to log to
      project: comet-examples-torchtune

This would have two advantages when we want to move the integration to the Torchtune repository:

• Users would keep installing the same libraries (just transformers and comet_ml).
• Users would only need to update the _component_ path to torchtune.utils.metric_logging.CometLogger.

Please let me know what you think about this technical solution.

Having a dedicated section for integrations in the Readme makes sense. Here is I think a good example of how another library highlights its integrations: https://github.com/ultralytics/ultralytics?tab=readme-ov-file#integrations (the picture is big but helps to understand where each integration kicks in. It can probably be added later when you have more integrations). I'm not 100% sure that Torchtune users would understand the difference between integrations that live inside the Torchtune repository vs in an external repository. For example, we document all of our integrations the same no matter where the code lives: https://www.comet.com/docs/v2/integrations/overview/.