In my last expose on Jerry Camel's SQL processing script for building Delta snapshots I mentioned you'd probably really want something like this for Attribute Level Change Notification (ALCN). You see when MIIS processes a typical delta row it still has to analyze each column of data to find the field that changed so while you cut out thousands of records to sift through, you still have to sift through each column. With ALCN you are actually telling MIIS "this is the row and this is the field that changed" and the processing engine will ignore data in the other columns whether it changed or not (go ahead and try, it's geeky-cool).
If there is a pecking order to speeding up DBMS based imports I would probably recommend this in order of difficulty:
- Copy any remote data sources locally first (for data sources that are Read Only) or use Linked Servers and a View (for data sources that are Read/Write) to access the remote table as the original. Importing data from a local source speeds up Imports significantly (30%-60% in ILM 2007, as much as 80% in MIIS pre SP2).
- Create a Delta Snapshot of the table. Importing only the changed rows will allow you to go from multi-hour imports to multi-minute imports depending on the frequency of data change in your source. You'll also be able to run imports more often.
- Modify your delta snapshot process to expose only the changed fields using Attribute Level Change Notification. Removing the need to process the unchanged data can improve Delta processing by a further 30%-60%.
- Build an XMA to combine all of the above. The XMA has the added benefit of distilling your data source into a simple file import which is always faster on the Import stage.
If you are really a SQL nerd there are further tricks that are beyond me depending on the data and the size of the data sets involved that involve when you should truncate tables, drop/create indexes, or modify the data in flight (like converting empty strings to nulls, etc).
So, enjoy Jerry's latest masterpiece with full Christmas spirit (this is a big deal for Mr. Camel, so don't overlook this one!) - there's no lump of coal in your stocking this year my fellow MIIS fans!
Digital Camel: Uncovering the MIIStery of Attribute Level Deltas (In Holiday Verse)