Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MSSQL Migration not working - ERROR MSSQL ERROR: %dbsqlexec fail #1220

Closed
5 tasks
haj92 opened this issue Oct 15, 2020 · 3 comments · May be fixed by #1243
Closed
5 tasks

MSSQL Migration not working - ERROR MSSQL ERROR: %dbsqlexec fail #1220

haj92 opened this issue Oct 15, 2020 · 3 comments · May be fixed by #1243

Comments

@haj92
Copy link

haj92 commented Oct 15, 2020

  • Pgloader Version:
    pgloader version 3.6.48d8ed0 compiled with SBCL 2.0.1.debian

  • FreeTDS:
    Compile-time settings (established with the "configure" script)
    Version: freetds v1.2.5
    freetds.conf directory: /usr/local/etc
    MS db-lib source compatibility: no
    Sybase binary compatibility: no
    Thread safety: yes
    iconv library: yes
    TDS version: 7.4
    iODBC: no
    unixodbc: yes
    SSPI "trusted" logins: no
    Kerberos: no
    OpenSSL: no
    GnuTLS: no
    MARS: yes

Also tried it with TDS Version 7.3

  • MSSQL:
    it is working for me in test:
    Microsoft SQL Server 2019 (RTM) - 15.0.2000.5 (X64) Sep 24 2019 13:48:23 Copyright (C) 2019 Microsoft Corporation Express Edition (64-bit) on Windows Server 2016 Standard 10.0 <X64> (Build 14393: ) (Hypervisor)
    But not in PRD env:
    Microsoft SQL Server 2017 (RTM-CU17) (KB4515579) - 14.0.3238.1 (X64) Sep 13 2019 15:49:57 Copyright (C) 2017 Microsoft Corporation Standard Edition (64-bit) on Windows Server 2016 Datacenter 10.0 <X64> (Build 14393: ) (Hypervisor)

  • Logs in debug mode:

    `~$ ~/pgloader/build/bin/pgloader -d ~/pgloader/mssql2postgresql.load
    pgloader version 3.6.48d8ed0
    compiled with SBCL 2.0.1.debian
    sb-impl::default-external-format :UTF-8
    tmpdir: #P"/tmp/pgloader/"
    2020-10-15T08:12:49.006000Z NOTICE Starting pgloader, log system is ready.
    2020-10-15T08:12:49.019000Z INFO Starting monitor
    2020-10-15T08:12:49.024000Z LOG pgloader version "3.6.48d8ed0"
    2020-10-15T08:12:49.076000Z INFO Parsed command:
    load database
    from ***
    into ***

including only table names like '***' in schema 'dbo'

set work_mem to '16MB', maintenance_work_mem to '512 MB'

before load do $$ drop schema if exists dbo cascade; $$;

2020-10-15T08:12:49.187000Z DEBUG CONNECTED TO #<PGLOADER.PGSQL:PGSQL-CONNECTION *** {1007694B83}>
2020-10-15T08:12:49.187000Z DEBUG SET client_encoding TO 'utf8'
2020-10-15T08:12:49.188000Z DEBUG SET work_mem TO '16MB'
2020-10-15T08:12:49.189000Z DEBUG SET maintenance_work_mem TO '512 MB'
2020-10-15T08:12:49.189000Z DEBUG SET application_name TO 'pgloader'
2020-10-15T08:12:49.420000Z NOTICE Executing SQL block for before load
2020-10-15T08:12:49.493000Z DEBUG CONNECTED TO #<PGLOADER.PGSQL:PGSQL-CONNECTION *** {1007694B83}>
2020-10-15T08:12:49.494000Z DEBUG SET client_encoding TO 'utf8'
2020-10-15T08:12:49.494000Z DEBUG SET work_mem TO '16MB'
2020-10-15T08:12:49.496000Z DEBUG SET maintenance_work_mem TO '512 MB'
2020-10-15T08:12:49.497000Z DEBUG SET application_name TO 'pgloader'
2020-10-15T08:12:49.505000Z DEBUG BEGIN
2020-10-15T08:12:49.507000Z SQL drop schema if exists dbo cascade;
2020-10-15T08:12:49.516000Z LOG Migrating from #<MSSQL-CONNECTION *** {10076936A3}>
2020-10-15T08:12:49.516000Z LOG Migrating into #<PGSQL-CONNECTION *** {1007694B83}>
Max connections reached, increase value of TDS_MAX_CONN
2020-10-15T08:12:51.091000Z SQL MSSQL: sending query: -- params: dbname
-- table-type-name
-- including
-- filter-list-to-where-clause including
-- excluding
-- filter-list-to-where-clause excluding
select c.TABLE_SCHEMA,
c.TABLE_NAME,
c.COLUMN_NAME,
c.DATA_TYPE,
CASE
WHEN c.COLUMN_DEFAULT LIKE '((%' AND c.COLUMN_DEFAULT LIKE '%))' THEN
CASE
WHEN SUBSTRING(c.COLUMN_DEFAULT,3,len(c.COLUMN_DEFAULT)-4) = 'newid()' THEN 'GENERATE_UUID'
WHEN SUBSTRING(c.COLUMN_DEFAULT,3,len(c.COLUMN_DEFAULT)-4) LIKE 'convert(%varchar%,getdate(),%)' THEN 'today'
WHEN SUBSTRING(c.COLUMN_DEFAULT,3,len(c.COLUMN_DEFAULT)-4) = 'getdate()' THEN 'CURRENT_TIMESTAMP'
WHEN SUBSTRING(c.COLUMN_DEFAULT,3,len(c.COLUMN_DEFAULT)-4) LIKE '''%''' THEN SUBSTRING(c.COLUMN_DEFAULT,4,len(c.COLUMN_DEFAULT)-6)
ELSE SUBSTRING(c.COLUMN_DEFAULT,3,len(c.COLUMN_DEFAULT)-4)
END
WHEN c.COLUMN_DEFAULT LIKE '(%' AND c.COLUMN_DEFAULT LIKE '%)' THEN
CASE
WHEN SUBSTRING(c.COLUMN_DEFAULT,2,len(c.COLUMN_DEFAULT)-2) = 'newid()' THEN 'GENERATE_UUID'
WHEN SUBSTRING(c.COLUMN_DEFAULT,2,len(c.COLUMN_DEFAULT)-2) LIKE 'convert(%varchar%,getdate(),%)' THEN 'today'
WHEN SUBSTRING(c.COLUMN_DEFAULT,2,len(c.COLUMN_DEFAULT)-2) = 'getdate()' THEN 'CURRENT_TIMESTAMP'
WHEN SUBSTRING(c.COLUMN_DEFAULT,2,len(c.COLUMN_DEFAULT)-2) LIKE '''%''' THEN SUBSTRING(c.COLUMN_DEFAULT,3,len(c.COLUMN_DEFAULT)-4)
ELSE SUBSTRING(c.COLUMN_DEFAULT,2,len(c.COLUMN_DEFAULT)-2)
END
ELSE c.COLUMN_DEFAULT
END,
c.IS_NULLABLE,
COLUMNPROPERTY(object_id(c.TABLE_NAME), c.COLUMN_NAME, 'IsIdentity'),
c.CHARACTER_MAXIMUM_LENGTH,
c.NUMERIC_PRECISION,
c.NUMERIC_PRECISION_RADIX,
c.NUMERIC_SCALE,
c.DATETIME_PRECISION,
c.CHARACTER_SET_NAME,
c.COLLATION_NAME

from INFORMATION_SCHEMA.COLUMNS c
     join INFORMATION_SCHEMA.TABLES t
          on c.TABLE_SCHEMA = t.TABLE_SCHEMA
         and c.TABLE_NAME = t.TABLE_NAME

where c.TABLE_CATALOG = ''
and t.TABLE_TYPE = 'BASE TABLE'
and ((c.table_schema = 'dbo' and c.table_name LIKE '
'))

order by c.table_schema, c.table_name, c.ordinal_position;
2020-10-15T08:12:51.332000Z SQL MSSQL: sending query: -- params: including
-- filter-list-to-where-clause including
-- excluding
-- filter-list-to-where-clause excluding
select schema_name(schema_id) as SchemaName,
o.name as TableName,
REPLACE(i.name, '.', '_') as IndexName,
co.[name] as ColumnName,
i.is_unique,
i.is_primary_key,
i.filter_definition

from sys.indexes i
     join sys.objects o on i.object_id = o.object_id
     join sys.index_columns ic on ic.object_id = i.object_id
         and ic.index_id = i.index_id
     join sys.columns co on co.object_id = i.object_id
         and co.column_id = ic.column_id

where schema_name(schema_id) not in ('dto', 'sys')
and ((schema_name(schema_id) = 'dbo' and o.name LIKE '***'))

order by SchemaName,
o.[name],
i.[name],
ic.is_included_column,
ic.key_ordinal;
2020-10-15T08:12:51.549000Z SQL MSSQL: sending query: -- params: dbname
-- including
-- filter-list-to-where-clause including
-- excluding
-- filter-list-to-where-clause excluding
SELECT
REPLACE(KCU1.CONSTRAINT_NAME, '.', '_') AS 'CONSTRAINT_NAME'
, KCU1.TABLE_SCHEMA AS 'TABLE_SCHEMA'
, KCU1.TABLE_NAME AS 'TABLE_NAME'
, KCU1.COLUMN_NAME AS 'COLUMN_NAME'
, KCU2.TABLE_SCHEMA AS 'UNIQUE_TABLE_SCHEMA'
, KCU2.TABLE_NAME AS 'UNIQUE_TABLE_NAME'
, KCU2.COLUMN_NAME AS 'UNIQUE_COLUMN_NAME'
, RC.UPDATE_RULE AS 'UPDATE_RULE'
, RC.DELETE_RULE AS 'DELETE_RULE'

FROM INFORMATION_SCHEMA.REFERENTIAL_CONSTRAINTS RC
     JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE KCU1
          ON KCU1.CONSTRAINT_CATALOG = RC.CONSTRAINT_CATALOG
             AND KCU1.CONSTRAINT_SCHEMA = RC.CONSTRAINT_SCHEMA
             AND KCU1.CONSTRAINT_NAME = RC.CONSTRAINT_NAME
     JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE KCU2
          ON KCU2.CONSTRAINT_CATALOG = RC.UNIQUE_CONSTRAINT_CATALOG
             AND KCU2.CONSTRAINT_SCHEMA = RC.UNIQUE_CONSTRAINT_SCHEMA
             AND KCU2.CONSTRAINT_NAME = RC.UNIQUE_CONSTRAINT_NAME

WHERE KCU1.ORDINAL_POSITION = KCU2.ORDINAL_POSITION
AND KCU1.TABLE_CATALOG = ''
AND KCU1.CONSTRAINT_CATALOG = '
'
AND KCU1.CONSTRAINT_SCHEMA NOT IN ('dto', 'sys')
AND KCU1.TABLE_SCHEMA NOT IN ('dto', 'sys')
AND KCU2.TABLE_SCHEMA NOT IN ('dto', 'sys')

     and ((kcu1.table_schema = 'dbo' and kcu1.table_name LIKE '***'))

ORDER BY KCU1.CONSTRAINT_NAME, KCU1.ORDINAL_POSITION;
2020-10-15T08:12:52.267000Z ERROR MSSQL ERROR: %dbsqlexec fail
2020-10-15T08:12:52.268000Z LOG You might need to review the FreeTDS protocol version in your freetds.conf file, see http://www.freetds.org/userguide/choosingtdsprotocol.htm
2020-10-15T08:12:52.268000Z LOG report summary reset
table name errors read imported bytes total time read write


  before load          0          1          1                     0.090s

fetch meta data 0 0 0 0.000s



2020-10-15T08:12:52.322000Z INFO Stopping monitor`

  • Running the last sql query casing the fail in a client:

Msg 4104 (severity 16, state 1) from COFPT-MSSQL01\INFLN10501 Line 29: "The multi-part identifier "kcu1.table_schema" could not be bound." Msg 4104 (severity 16, state 1) from COFPT-MSSQL01\INFLN10501 Line 29: "The multi-part identifier "kcu1.table_name" could not be bound."

Thanks in advance for any support in the matter!

@haj92
Copy link
Author

haj92 commented Oct 15, 2020

Hello @dimitri

I don't know if this might solve it but if we update the last line in the WHERE clause in the query it runs in MSSQL client:

Old:

  SELECT
           REPLACE(KCU1.CONSTRAINT_NAME, '.', '_') AS 'CONSTRAINT_NAME'
         , KCU1.TABLE_SCHEMA AS 'TABLE_SCHEMA'
         , KCU1.TABLE_NAME AS 'TABLE_NAME'
         , KCU1.COLUMN_NAME AS 'COLUMN_NAME'
         , KCU2.TABLE_SCHEMA AS 'UNIQUE_TABLE_SCHEMA'
         , KCU2.TABLE_NAME AS 'UNIQUE_TABLE_NAME'
         , KCU2.COLUMN_NAME AS 'UNIQUE_COLUMN_NAME'
         , RC.UPDATE_RULE AS 'UPDATE_RULE'
         , RC.DELETE_RULE AS 'DELETE_RULE'

    FROM INFORMATION_SCHEMA.REFERENTIAL_CONSTRAINTS RC
         JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE KCU1
              ON KCU1.CONSTRAINT_CATALOG = RC.CONSTRAINT_CATALOG
                 AND KCU1.CONSTRAINT_SCHEMA = RC.CONSTRAINT_SCHEMA
                 AND KCU1.CONSTRAINT_NAME = RC.CONSTRAINT_NAME
         JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE KCU2
              ON KCU2.CONSTRAINT_CATALOG = RC.UNIQUE_CONSTRAINT_CATALOG
                 AND KCU2.CONSTRAINT_SCHEMA = RC.UNIQUE_CONSTRAINT_SCHEMA
                 AND KCU2.CONSTRAINT_NAME = RC.UNIQUE_CONSTRAINT_NAME

   WHERE KCU1.ORDINAL_POSITION = KCU2.ORDINAL_POSITION
         AND KCU1.TABLE_CATALOG = 'ptln105'
         AND KCU1.CONSTRAINT_CATALOG = 'ptln105'
         AND KCU1.CONSTRAINT_SCHEMA NOT IN ('dto', 'sys')
         AND KCU1.TABLE_SCHEMA NOT IN ('dto', 'sys')
         AND KCU2.TABLE_SCHEMA NOT IN ('dto', 'sys')

         and ((kcu1.table_schema = 'dbo' and kcu1.table_name LIKE 'ttccom000800'))


ORDER BY KCU1.CONSTRAINT_NAME, KCU1.ORDINAL_POSITION;

Update:

  SELECT
           REPLACE(KCU1.CONSTRAINT_NAME, '.', '_') AS 'CONSTRAINT_NAME'
         , KCU1.TABLE_SCHEMA AS 'TABLE_SCHEMA'
         , KCU1.TABLE_NAME AS 'TABLE_NAME'
         , KCU1.COLUMN_NAME AS 'COLUMN_NAME'
         , KCU2.TABLE_SCHEMA AS 'UNIQUE_TABLE_SCHEMA'
         , KCU2.TABLE_NAME AS 'UNIQUE_TABLE_NAME'
         , KCU2.COLUMN_NAME AS 'UNIQUE_COLUMN_NAME'
         , RC.UPDATE_RULE AS 'UPDATE_RULE'
         , RC.DELETE_RULE AS 'DELETE_RULE'

    FROM INFORMATION_SCHEMA.REFERENTIAL_CONSTRAINTS RC
         JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE KCU1
              ON KCU1.CONSTRAINT_CATALOG = RC.CONSTRAINT_CATALOG
                 AND KCU1.CONSTRAINT_SCHEMA = RC.CONSTRAINT_SCHEMA
                 AND KCU1.CONSTRAINT_NAME = RC.CONSTRAINT_NAME
         JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE KCU2
              ON KCU2.CONSTRAINT_CATALOG = RC.UNIQUE_CONSTRAINT_CATALOG
                 AND KCU2.CONSTRAINT_SCHEMA = RC.UNIQUE_CONSTRAINT_SCHEMA
                 AND KCU2.CONSTRAINT_NAME = RC.UNIQUE_CONSTRAINT_NAME

   WHERE KCU1.ORDINAL_POSITION = KCU2.ORDINAL_POSITION
         AND KCU1.TABLE_CATALOG = 'ptln105'
         AND KCU1.CONSTRAINT_CATALOG = 'ptln105'
         AND KCU1.CONSTRAINT_SCHEMA NOT IN ('dto', 'sys')
         AND KCU1.TABLE_SCHEMA NOT IN ('dto', 'sys')
         AND KCU2.TABLE_SCHEMA NOT IN ('dto', 'sys')

         and ((KCU1.table_schema = 'dbo' and KCU1.table_name LIKE 'ttccom000800'))

Appreciate any help on the matter

@haj92
Copy link
Author

haj92 commented Oct 16, 2020

For those of you who faced similar issue, updating the sql query in pgloader/src/sources/mssql/sql/list-all-fkeys.sql
and recompiling pgloader solved the issue for me

@haj92 haj92 closed this as completed Oct 16, 2020
@qkdreyer
Copy link

qkdreyer commented Dec 15, 2020

57a32c7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants