-
Notifications
You must be signed in to change notification settings - Fork 20
FAQ
- How can I keep my SQL table up to date without reloading everything?
- How do I get display values for reference fields?
- What happens if a new field is added to the ServiceNow source table?
- What if I do not want certain fields to be accessible?
- Why are some records not processed?
- Why are records with empty sys_created_on not loaded?
- What if the name of a ServiceNow field is longer than the database field name limit?
- What can I do about timeout errors?
Use a Metrics file to keep track of the last time the loader ran, and specify since: last to only load records inserted or updated since the last run.
OR
Use action: sync to compare timestamps (sys_updated_on) between the source and the target.
You will need to load the reference table and use a join to include the display value in your SQL query.
The recommended architecture is to use two separate database schemas:
- A schema with the raw data from ServiceNow
- A schema with database views for end user reporting
The reporting schema should handle any necessary joins, and only include appropriate rows and columns.
SNDML will not add columns to an existing database table. However, you can use ALTER TABLE to manually add columns to a database table.
When SNDML starts processing, it compares the list of fields in the source table with the list of columns in the target table. It loads those columns where the field names match. If you have altered your database table by adding or dropping columns, then it will affect which columns are loaded by SNDML.
Use ALTER TABLE to drop the columns.
What is the cause of this warning message?
Expected x rows but only processed y rows
This is usually related to ACLs, but it could also be caused by a Before Query Business Rule.
As an example, I recently exported the table cmdb_ci_service and got this warning
Expected 1576 rows but only processed 1571 rows
It turned out that a few of the rows were is other tables such as cmdb_ci_service_business, cmdb_ci_service_technical or cmdb_ci_service_auto.
These records were included in the initial count because these tables are based on cmdb_ci_service.
However, the datapump.reader account did not have access to read these records.
The default created range is [void, start], which causes the loader to not load records inserted after the loader starts running. However, the resulting encoded query also causes records to not be loaded at all if sys_created_on is empty. The solution is to use [void, void] to override the default created range and a filter to specify the records that need to be loaded.
created: [void, void], filter: sys_created_onISEMPTY
For additional details refer to Dates and Date Ranges.
You can rename a field by editing the fieldmaps section of sqltemplates.xml as describe in Templates.
Timeouts will occur if
- The Page Size is too large
- You are running too many jobs concurrently
- You are using too many threads in a partitioned load
My recommendation is that the total number of concurrent jobs + threads should typically not exceed 4.
If a timeout occurs while trying to load an extremely large table (several million records)
and there are no other concurrent loads running and you are not using multiple threads
then make sure you have an index on sys_created_on and perform a partitioned load with smaller partitions.