Compiling a Windows DLL With Support for Parquet Queries

[SartorialOffense]

I have been poking around in the repo and issues trying to determine if it is feasible to compile a Windows DLL with the Parquet extension included with the ultimate goal of using it in a .NET project. My knowledge of C++ is from a brown book I read 1993, so I am slightly behind the times!

But I was able to compile the full build on Windows just using what I found in your Windows (64 Bit) target . After fixing a minor path separator issue in parquet-amalgamation.py, I am able to run this and can find the 4 generated files in src/amalgamation. From there I thought I would be able to compile but I am having some issues.

If I simply:

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_GENERATOR_PLATFORM=x64 -DBUILD_ICU_EXTENSION=1 -DBUILD_PARQUET_EXTENSION=1 -DBUILD_TPCH_EXTENSION=1 -DBUILD_TPCDS_EXTENSION=1 -DBUILD_FTS_EXTENSION=1 -DBUILD_REST=1 -DJDBC_DRIVER=1 -DBUILD_VISUALIZER_EXTENSION=1 -DBUILD_ODBC_DRIVER=1 -DDISABLE_UNITY=1 -DAMALGAMATION_BUILD=1 cmake --build . --config Release

I get lots of errors complaining about redefinitions:

C:\git\duckdb\src\amalgamation\duckdb.hpp(14511,24): error C2011: 'duckdb::CreateTableInfo': 'struct' type redefinition [C:\git\duckdb\src\duckdb_static.vcxproj]

I am assuming this means it is getting confused by the types in the amalgamation and the main code. So, I tried building my own make file with just the needed files:

cmake_minimum_required(VERSION 3.10) project(duckdb_side_of_parquet) include_directories(${PROJECT_SOURCE_DIR}) add_library(duckdb_side_of_parquet SHARED duckdb.cpp parquet-amalgamation.cpp)

But this is giving me headaches with inconsistent linkages making me think this isn't prepped for being a windows DLL:

C:\dd\duckdb.cpp(36341,42): warning C4273: 'INVALID': inconsistent dll linkage [C:\dd\duckdb_side_of_parquet.vcxproj] C:\dd\duckdb.hpp(1021,38): message : see previous definition of 'public: static duckdb::LogicalType const duckdb::LogicalType::INVALID' [C:\dd\duckdb_side_of_parquet.vcxproj] C:\dd\duckdb.cpp(36341,42): error C2491: 'duckdb::LogicalType::INVALID': definition of dllimport static data member not allowed [C:\dd\duckdb_side_of_parquet.vcxproj]

Any suggestions on what path might be the most fruitful here? Using DuckDB to query Parquet files is going to save me some serious work so I don't mind banding my head against the wall for a bit more - but I assume someone else has already thought about this a bit more and might be able to tell me if there is ongoing work or if such a thing is possible for a C++ novice to tackle.

Thanks!

[SartorialOffense]

EDIT: This actually does work - I just had my head somewhere it normally isn't supposed to be. Have not tested the Parquet extension though.

So I did make some progress on this. I was finally able to create a DLL from amalgamated files the with this CMakeList file:

cmake_minimum_required(VERSION 3.10)

project(duckdb_side_of_parquet)

include (GenerateExportHeader)
add_definitions(-DDUCKDB) # not sure if this makes a differencer 
add_definitions(/bigobj)
add_definitions(-DDUCKDB_BUILD_LIBRARY)
add_definitions(-DWIN32)

include_directories(${PROJECT_SOURCE_DIR})
add_library(duckdb_side_of_parquet SHARED duckdb.cpp parquet-amalgamation.cpp)

I dumped the PDB file contents and verified that at least there are symbols from both amalgamated files there. But, when I try and actually use in a .NET application (C#), I am unable to use any of them.

[DllImport("duckdb_side_of_parquet.dll", CallingConvention = CallingConvention.Cdecl, EntryPoint = "duckdb_open")]
public static extern DuckDBState DuckDBOpen(string path, out DuckDBDatabase database);

Yields:

Unable to load DLL 'duckdb_side_of_parquet.dll' or one of its dependencies: The specified module could not be found. (0x8007007E)

So the calling convention is off or something.

[SartorialOffense]

So I added my own end point to load the parquet extension:

__declspec(dllexport) void load_parquet_ext(duckdb_database *database)
{
    auto wrapper = (DatabaseData *)database;
    DuckDB db = *wrapper->database;
    db.LoadExtension<ParquetExtension>();
}

It compiles and even runs. I am having some stability issues with the created DLL's even without calling (or including) this endpoint so I need to sort that out as well.

[SartorialOffense]

No actually it seems to work just fine after a reboot. Might have been an issue with running the app in Visual Studio preview. But I think the extensions should each have a DLL with a signature like:

__declspec(dllexport) void load_extension(duckdb_database *database)
{
auto wrapper = (DatabaseData *)database;
DuckDB db = *wrapper->database;
db.LoadExtension<...Extension>();
}