future of sqlite3 in doxygen

[abathur]

Hi @jcarrano,

I noticed this repo on a search I occasionally run for new code interacting with Doxygen's sqlite3 database. It didn't look like you're actually using it--just that you mention it?

In any case, I wanted to give you a heads up that I'm working on a pretty big update to Doxygen's sqlite3 generator. If I can get the contribution accepted, the schema (and quality of the data it holds) may change in a future Doxygen version.

Also, are you aware of Breathe (https://github.com/michaeljones/breathe)? At a glance, it sounds like you're both working to leverage Doxygen's output, via XML, to generate Sphinx documentation.

[jcarrano]

Hi @abathur

Thank you very much for tackling this issue, and for letting me know. Do you have a link to the issue/pr/thread?

This repo sprung out of my efforts to use sphinx for RIOT's documentation. See this pr.

Of course my first approach was to use Breathe, which unfortunately suffers from some hard-to-fix perfomance issues, partly caused by minidom and partly by Breathe's design (it has some kind of DSL for filtering and stuff). The result is that the build exhausts ReadTheDocs's build resources. It even takes ages in my 8-core i7 desktop box.

Apart from performance, I'm really enjoying doing XML transforms. I feel I would need a huge amount of python to get the same effect.

I think a relational model is much more suited to the kind of data that Doxygen generates than XML. While source code is hierarchic (in fact is has a tree structure), the docs are full of cross references, which makes it more like a graph. For my use case I only needed the DB for indexing, though having more data would mean that I wouldn't have to repeatedly load and parse XML (multiple times) to extract entities.

If a future Doxygen update brings [stable] SQL support I will definitively take a look. It would improve Sphinx startup times, as I would not have to load all XML and also memory consumption because then I could count on having the DB as a file, which could be shared between parallel-build workers. As I mentioned in the README, whatever support there is now is not very much documented, so I played it safe and went with XML.

[abathur]

I don't have a PR for it quite yet, but I'll try to remember to drop a link here when I do.

This sounds familiar. :) I've been on the reverse yak-shaving adventure, where I'm working on going from Sphinx -> Doxygen -> sqlite3.

There are tradeoffs between the output formats. Relational makes some things easier, but sometimes it also means things that are fairly intuitive to extract from the XML need a fairly long query on the SQL side.

Another aside, in case it's useful: When I was skimming your code I noticed something about dumping the SQL statements to pickle the database, and it reminded me of something a CLI utility I like (https://github.com/dinedal/textql) does. In normal operation, it'll load a CSV or TSV into an sqlite3 in-memory database so that you can query it, and then the output the query rows in CSV/TSV format, so you can pipe it to other utilities or save it out to a new file. It also has a flag to make it save out the sqlite3 file for continued use.