Is is possible to test & benchmark each component in a standalone way?

[JigaoLuo]

Hi.
My question is: is it possible now to test and benchmark each component in a standalone way?

Be more specific: There is a popular way to import google test and google benchmark to test a single standalone component like ART Indexing or a Hash Join Operator.
I see in DuckDB the unit test and benchmark cases are all built upon on a SQL interface.
Yes SQL interface is crucial for a DBMS.
But a component-wise unit test and benchmark keep the noise of other components away, which might be helpful to debug and do profiling.

[Mytherin]

We avoid doing component-wise testing and benchmarking. First, it requires writing and maintaining lots of code, mocks, etc. that have no other purpose than testing and benchmarking. This huge extra set of code means changing anything later on becomes much more painful, since not only does the code have to be rewritten, the tests need to be rewritten as well. SQL is perfect for testing since there is a very specific contract (this query returns this result) that the system needs to handle anyway. Adding SQL tests does not cause any lock-in to current code, which means implementations are easy to change or swap around. Sometimes forcing specific code to be executed or tested is useful, but working around that by adding a PRAGMA is much easier than adding all the code, mocks, etc required for component-wise testing.

"Keeping the noise of other components away" for benchmarking sounds good, but in practice it means you might be optimizing entirely irrelevant sections of code. For example, if you spend weeks optimizing a method (e.g. ART::Search) and make it 10x faster, but in end-to-end code all time is spend on tuple reconstruction and this speed-up only makes the actual queries 1% faster, it might have been more productive to spend time elsewhere. If the noise of other components is so significant, maybe the optimization isn't that relevant to begin with.