Skip to content

Add provider metadata discovery#187

Open
PrzemyslawKlys wants to merge 42 commits into
masterfrom
feature/metadata-discovery-no-smo
Open

Add provider metadata discovery#187
PrzemyslawKlys wants to merge 42 commits into
masterfrom
feature/metadata-discovery-no-smo

Conversation

@PrzemyslawKlys

Copy link
Copy Markdown
Member

Summary

  • add provider-neutral metadata models for databases, tables/views, columns, and indexes
  • implement native catalog-query discovery across SQL Server, PostgreSQL, MySQL, Oracle, and SQLite without adding an SMO dependency
  • expose a thin Get-DbaXMetadata PowerShell cmdlet and document metadata examples

Validation

  • dotnet build .\DbaClientX.sln -c Release --framework net8.0
  • dotnet test .\DbaClientX.Tests\DbaClientX.Tests.csproj -c Release --framework net8.0 --no-build

Notes

  • Module\Build\Build-Module.ps1 is currently blocked before code execution by a PSPublishModule Gallery version parsing error: build ('-2147483648') must be a non-negative value.

Comment thread DbaClientX.SQLite/SQLite.Metadata.cs Fixed
@codecov

codecov Bot commented Jun 22, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 46.55082% with 1015 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.86%. Comparing base (b232813) to head (fa81868).

Files with missing lines Patch % Lines
DbaClientX.SqlServer/SqlServer.Management.cs 31.38% 188 Missing ⚠️
DbaClientX.MySql/MySql.Metadata.cs 0.00% 135 Missing ⚠️
DbaClientX.Oracle/Oracle.Metadata.cs 0.00% 121 Missing ⚠️
DbaClientX.PostgreSql/PostgreSql.Metadata.cs 0.00% 113 Missing ⚠️
DbaClientX.SqlServer/SqlServer.Metadata.cs 0.00% 113 Missing ⚠️
...SqlServer/Management/SqlServerManagementMappers.cs 50.45% 94 Missing and 16 partials ⚠️
DbaClientX.PowerShell/CmdletGetDbaXMetadata.cs 0.00% 78 Missing ⚠️
...lServer/Management/SqlServerManagementScripting.cs 89.26% 10 Missing and 35 partials ⚠️
...ntX.PowerShell/CmdletGetDbaXSqlServerManagement.cs 0.00% 38 Missing ⚠️
DbaClientX.SQLite/SQLite.Metadata.cs 81.48% 19 Missing and 1 partial ⚠️
... and 12 more
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #187      +/-   ##
==========================================
- Coverage   50.65%   49.86%   -0.79%     
==========================================
  Files          99      130      +31     
  Lines        7970     9869    +1899     
  Branches     1428     1609     +181     
==========================================
+ Hits         4037     4921     +884     
- Misses       3547     4501     +954     
- Partials      386      447      +61     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ec51d8c5a8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread DbaClientX.Oracle/Oracle.Metadata.cs Outdated
Comment thread DbaClientX.PostgreSql/PostgreSql.Metadata.cs Outdated
Comment thread DbaClientX.SQLite/SQLite.Metadata.cs Outdated
Comment thread DbaClientX.SqlServer/SqlServer.Metadata.cs Outdated
Comment thread DbaClientX.SQLite/SQLite.Metadata.cs Outdated
Comment thread DbaClientX.SqlServer/SqlServer.Metadata.cs Outdated
Comment thread DbaClientX.SQLite/SQLite.Metadata.cs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1cf319d3f3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread DbaClientX.PostgreSql/PostgreSql.Metadata.cs
Comment thread DbaClientX.SQLite/SQLite.Metadata.cs
Comment thread DbaClientX.Oracle/Oracle.Metadata.cs Outdated
Comment thread DbaClientX.PostgreSql/PostgreSql.Metadata.cs Outdated
Comment thread DbaClientX.PostgreSql/PostgreSql.Metadata.cs
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@PrzemyslawKlys

Copy link
Copy Markdown
Member Author

@codex please review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 91ecc373a8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread DbaClientX.SQLite/SQLite.Metadata.cs Outdated
Comment thread DbaClientX.SqlServer/SqlServer.Metadata.cs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 90bc39a2b1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread DbaClientX.Oracle/Oracle.Metadata.cs Outdated
Comment thread DbaClientX.Oracle/Oracle.Metadata.cs Outdated
Comment thread DbaClientX.SQLite/SQLite.Metadata.cs
Comment thread DbaClientX.MySql/MySql.Metadata.cs Outdated
Comment thread DbaClientX.SqlServer/SqlServer.Metadata.cs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b903a59633

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread DbaClientX.Oracle/Oracle.Metadata.cs
Comment thread DbaClientX.SQLite/SQLite.Metadata.cs Outdated
Comment thread DbaClientX.PostgreSql/PostgreSql.Metadata.cs Outdated
Comment thread DbaClientX.Oracle/Oracle.Metadata.cs Outdated
Comment thread DbaClientX.SqlServer/SqlServer.Metadata.cs
Comment thread DbaClientX.PostgreSql/PostgreSql.Metadata.cs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 32c3f42ced

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread DbaClientX.SQLite/SQLite.Metadata.cs Outdated
Comment thread DbaClientX.PostgreSql/PostgreSql.Metadata.cs Outdated
Comment thread DbaClientX.Oracle/Oracle.Metadata.cs Outdated
Comment thread DbaClientX.PostgreSql/PostgreSql.Metadata.cs
Comment thread DbaClientX.SqlServer/SqlServer.Metadata.cs Outdated
Comment thread DbaClientX.SqlServer/SqlServer.Metadata.cs Outdated
Comment thread DbaClientX.MySql/MySql.Metadata.cs Outdated
Comment thread DbaClientX.MySql/MySql.Metadata.cs
Comment thread DbaClientX.SqlServer/SqlServer.Metadata.cs
Comment thread DbaClientX.Oracle/Oracle.Metadata.cs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3cef7e6b19

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread DbaClientX.Oracle/Oracle.Metadata.cs Outdated
Comment thread DbaClientX.Oracle/Oracle.Metadata.cs Outdated
Comment thread DbaClientX.Oracle/Oracle.Metadata.cs Outdated
Comment thread DbaClientX.MySql/MySql.Metadata.cs
Comment thread DbaClientX.Core/Metadata/DbaColumnInfo.cs
Comment thread DbaClientX.Oracle/Oracle.Metadata.cs
Comment thread DbaClientX.SqlServer/SqlServer.Metadata.cs
Comment thread DbaClientX.PostgreSql/PostgreSql.Metadata.cs Outdated
Comment thread DbaClientX.MySql/MySql.Metadata.cs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9590f35605

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread DbaClientX.PostgreSql/PostgreSql.Metadata.cs Outdated
Comment thread DbaClientX.Oracle/Oracle.Metadata.cs Outdated
Comment thread DbaClientX.PostgreSql/PostgreSql.Metadata.cs
Comment thread DbaClientX.Oracle/Oracle.Metadata.cs
Comment thread DbaClientX.PostgreSql/PostgreSql.Metadata.cs Outdated
Comment thread DbaClientX.PostgreSql/PostgreSql.Metadata.cs Outdated
Comment thread DbaClientX.MySql/MySql.Metadata.cs

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 697080dfd0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread DbaClientX.Oracle/Oracle.Metadata.cs Outdated
Comment thread DbaClientX.SQLite/SQLite.Metadata.cs
Comment thread DbaClientX.MySql/MySql.Metadata.cs
Comment thread DbaClientX.Oracle/Oracle.Metadata.cs

private static DataTableReader ReadSingleRow(params (string Name, Type Type, object Value)[] columns)
{
var table = new DataTable();
Comment on lines +25 to +31
foreach (SqlServerScriptInfo? postCreateScript in tableGroups.Select(BuildTablePostCreateScript))
{
if (postCreateScript != null)
{
scripts.Add(postCreateScript);
}
}

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fa81868f63

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +127 to +132
CASE
WHEN o.type IN ('FN', 'FS') THEN
CASE
WHEN ty.schema_id IS NOT NULL AND SCHEMA_NAME(ty.schema_id) <> 'sys' THEN CONCAT(SCHEMA_NAME(ty.schema_id), '.', ty.name)
ELSE ty.name
END

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve SQL Server return type modifiers

For scalar functions declared with modifiers, such as RETURNS decimal(10,2) or RETURNS nvarchar(50), this branch reports only ty.name, so GetRoutines returns decimal/nvarchar and loses the precision, scale, or length. The fresh issue in this version is that the projection now reads the return row but only emits the bare type name; since DbaRoutineInfo has no separate fields for those modifiers and other providers return full result type text, schema comparison or recreation can change the function signature.

Useful? React with 👍 / 👎.

i.index_type,
CASE WHEN i.uniqueness = 'UNIQUE' THEN 1 ELSE 0 END AS is_unique,
CASE WHEN c.constraint_type = 'P' THEN 1 ELSE 0 END AS is_primary_key,
CASE WHEN ie.column_expression IS NULL THEN ic.column_name ELSE NULL END AS column_name,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Avoid CASE on Oracle LONG index expressions

Fresh evidence in the current patch is that ie.column_expression is now used inside a CASE, but Oracle documents ALL_IND_EXPRESSIONS.COLUMN_EXPRESSION as a LONG, and LONG values cannot be used in normal SQL expressions without raising ORA-00997. On Oracle, GetIndexes can therefore fail when this SELECT is parsed instead of returning index metadata; fetch/convert the expression separately or avoid testing the LONG column in SQL.

Useful? React with 👍 / 👎.

NULL AS is_visible,
NULL AS prefix_length,
CASE WHEN ii.cid = -2 THEN im.sql ELSE NULL END AS expression,
CASE WHEN instr(lower(im.sql), ' where ') > 0 THEN substr(im.sql, instr(lower(im.sql), ' where ') + 7) ELSE NULL END AS filter_definition

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve formatted SQLite partial-index predicates

This extracts the partial-index predicate only when the stored CREATE INDEX text contains the exact substring ' where '. For a valid partial index created with common formatting such as CREATE INDEX ix ON Users(Name)\nWHERE Name IS NOT NULL, instr(...) returns 0 and FilterDefinition becomes null, so schema comparison/recreation silently treats the filtered index as unfiltered.

Useful? React with 👍 / 👎.

TABLE_NAME AS object_name,
CASE WHEN TABLE_TYPE = 'VIEW' THEN 'View' ELSE 'Table' END AS object_kind
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_SCHEMA = COALESCE(@schema, DATABASE())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Handle MySQL connections without a default schema

When the MySQL connection string omits a default database, DATABASE() is NULL, so this predicate becomes TABLE_SCHEMA = NULL and GetTables(connectionString) returns no tables even though schemas are visible and schema is optional. The same COALESCE(@schema, DATABASE()) pattern is used by the related metadata queries, so callers must already know and pass a schema to discover anything; list accessible non-system schemas or report a clear requirement when neither schema nor a current database is available.

Useful? React with 👍 / 👎.

Comment on lines +17 to +21
=> ExecuteMetadata(connectionString, SqlServerAgentJobsManagementQuery, SqlServerManagementMappers.MapAgentJob, new Dictionary<string, object?>
{
["@jobName"] = jobName,
["@includeDisabled"] = includeDisabled ? 1 : 0
});

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Gate direct Agent metadata on catalog support

On SQL Server targets without SQL Agent catalog tables, such as Azure SQL Database, this direct reader executes the msdb.dbo.sysjobs query unconditionally, while the inventory path already probes SupportsAgentCatalog before reading jobs. Get-DbaXSqlServerManagement -Type AgentJob therefore fails with an invalid-object SQL error instead of returning an empty/unsupported result; apply the same guard to the direct Agent job/step/schedule readers.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant