Get more information from the profile page

[Roughsketch]

There is currently a lot of information that is simply not scraped from the page. This issue will attempt to keep track of things that are missing and their progress.

• Profile
  • Race
  • Clan
  • Gender
  • Nameday
  • Guardian
  • City-state
  • Grand Company
  • Current class
  • Current class level
  • Currently worn items
  • All class levels
• Attributes
  • HP
  • MP
  • Current gear attributes
• Class/Job
  • XP in each class
  • Level in each class
• Mounts/Minions
  • List of available mounts
  • List of available minions

[Roughsketch]

A slight update to this, upon looking closer it looks like current class, minions, and mounts will not be possible to get without some fancy workaround. All of these use only images or javascript to display information, and thus there's nothing we can parse out of the doc itself.

tl;dr, if we want to implement current class, minions, and mounts we will need to verify that the ids of images they use stays constant and also find a way to map these in a way where we can get notified if things are added.

There is a slight workaround we can do for minions and mounts, but it would require an obscene amount of requests and I'd like to keep those to a minimum. For reference though, it looks like we can get individual information by calling a URL like <profile_url>/mount/tooltip/<mount_id>. Note, replacing mount with minion gets minions but the format for all this is the same.

For example: https://na.finalfantasyxiv.com/lodestone/character/11908971/mount/tooltip/9045c5c5d5d181ee495f0e76af07d6d93c9f0f13

This information is stored in the doc itself if you iterate through every mount__list_icon node and take the data-tooltip_href attribute. Since profiles with lots of mounts and minions could result in hundreds of individual calls I think this is not ethical to implement.

That being said, if the ids for mounts and minions stay the same between profiles, we could potentially make a table of mount/minion ids and simply map them to their respective names. I don't know where we can get all this information though, let alone keep it up to date. If there is some API or master list we can parse, I can throw that into a test and have the CI notify me of changes. From what I know right now, that sounds like the only way minions/mounts are getting into this library.

[mcronce]

If the IDs are constant, one option would be to have a codegen step that goes through a list of known IDs, pulls down the tooltips for each one, scrapes them, and puts that data down in a .rs file. Could technically be a build.rs step too. It would be a little janky, but it would cut down on runtime requests, and it would certainly be fast at runtime - just an in-static-memory lookup instead of a network call.

FWIW, looks like the mount IDs are constant. My Company Chocobo mount's tooltip is https://na.finalfantasyxiv.com/lodestone/character/30924141/mount/tooltip/9045c5c5d5d181ee495f0e76af07d6d93c9f0f13

[Roughsketch]

If possible I'd like to stop any excess requests, including at build time. In this case if the IDs are constant my best solution would be to manually keep track of IDs and add them to the library manually just like how basically everything else would be updated. For instance, there was no automatic way to add blue mage when it was released, that had to be done manually.

The issue with this is I have no way to consistently or accurately get the ID used on the website of every minion and mount which makes it very hard to actually keep a list.