DALOAD on IBMi
DALOAD is most efficient if it can process all the EXTPNDXX input records in memory in a single segment. Once the size of the input records it must process exceeds the amount of available memory, DALOAD automatically breaks the data into segments for sequential processing. We affectionately refer to this process as chunking.
The DALOAD log file labeled QDSXX will identify if chunking is occurring during the DALOAD process. The first step in the DALOAD process is to calculate how much memory will be required to process the EXTPNDXX input records in a single segment. The calculation of total memory required includes as input the number/size of the EXTPNDXX records and the count of the number of records in the header and detail DSL data tables which is stored in STCSDT15.* Once this calculation is finished, it is compared to the available memory on the server. If there is not enough physical memory available for DALOAD to process a single segment, then it will break (chunk) the input records in EXTPNDXX into multiple segments. The initial memory calculation and the number and size of the multiple segments are both reflected in the log file(s).
If you determine DALOAD is chunking, a straightforward solution is to make available to DALOAD the amount of memory being calculated for a single segment. This is best accomplished on IBMi by creating a dedicated subsystem to which the amount of memory required by DALOAD is allocated. This, of course, assumes enough physical memory exists on the hardware. Step-by-step instructions for creating a dedicated subsystem for DALOAD can be found on CustomerNet or by contacting Silvon support.
If making more memory available to DALOAD is not an option, there is an alternative that may require a bit more work. You can load the EXTPNDXX tables(s) from your source system with just enough data to prevent chunking. This in all likelihood will require modifications to those daily processes that extract source data and populate EXTPNDXX. If multiple years are being loaded you can try loading a single year in EXTPNDXX, running DALOAD and then repeating the process for subsequent years. Another approach is to load EXTPNDXX with a single category, run DALOAD and repeat for other categories. The key is to utilize all the available memory while minimizing the number of times you need to load EXTPNDXX and run DALOAD, but the performance improvements can be worth the time spent.
*The STCSDT15 table can occasionally become corrupt if a DALOAD process ended abnormally and was not recovered through the standard Stratum recovery process. If the table is corrupt, it could adversely affect the DALOAD memory calculation. The DADSLCNT command will refresh the STCSDT15 table if you suspect that your STCSDT15 may be incorrect.
Categorized in: Silvon Customers
This post was written by Steve Morgan