Any suggestion on how to speed up large batch inserts?

"Amber" wrote in message
news:89c3ccaa-ddf4-4688-96b5-8e18cee535d3@e39g2000hsf.googlegroups.com...
> We are using 8.2.9 Windows 64 edtion, in one of our projects we need
> to recreate a few lager tables which have many millions of rows each,
> we have used concurrent Java threads to read from data source and do
> batch inserts to the target tables concurrently, to speed up the
> inserts we have used the following configuration policies:
>
> 1. Use large page size (32K) and extend size (256) on the target db,
> 2. Use large buffer pools for the tablespaces which contains the large
> tables on the target db.
> 3. Set num_cleaners to 4 to speed up moving data from bufferpool to
> disks on the target db, we have 4 4-core CPU on the server.
> 4. Set chngpgs_thresh to 80%, using this value we mean avoiding DB2 to
> moving data from bufferpool to disk too frequtenlly, because when
> doing these inserts we have not other readers on the target.
> 5. Use lage page size(32K) and extend size(256) and prefetch size
> (256) on the source db.
> 6. Use lage bufferpool on the source db.
> 7. Set num_ioservers to 4, we have 4 4-core cpu on the source db.
>
> We are not sure about the chngpgs_thresh value, any suggestions?

Instead of modifying the chngpgs_thresh, set the
DB2_USE_ALTERNATE_PAGE_CLEANING to ON (with db2set).

In the db cfg, set you logbuffsz to about 256. This requires that you
increase DBHEAP by same amount since it is part of DBHEAP memory.

On Jul 14, 11:42*am, "Mark A" wrote:
> "Amber" wrote in message
>
> news:89c3ccaa-ddf4-4688-96b5-8e18cee535d3@e39g2000hsf.googlegroups.com...
>
>
>
>
>
> > We are using 8.2.9 Windows 64 edtion, in one of our projects we need
> > to recreate a few lager tables which have *many millions of rows each,
> > we have used concurrent Java threads to read from data source and do
> > batch inserts to the target tables concurrently, to speed up the
> > inserts we have used the following configuration policies:
>
> > 1. Use large page size (32K) and extend size (256) on the target db,
> > 2. Use large buffer pools for the tablespaces which contains the large
> > tables on the target db.
> > 3. Set num_cleaners to 4 to speed up moving data from bufferpool to
> > disks on the target db, we have 4 *4-core CPU on the server.
> > 4. Set chngpgs_thresh to 80%, using this value we mean avoiding DB2 to
> > moving data from bufferpool to disk too frequtenlly, because when
> > doing these inserts we have not other readers on the target.
> > 5. Use lage page size(32K) and extend size(256) and prefetch size
> > (256) on the source db.
> > 6. Use lage bufferpool on the source db.
> > 7. Set num_ioservers to 4, we have 4 4-core cpu on the source db.
>
> > We are not sure about the chngpgs_thresh value, any suggestions?
>
> Instead of modifying the chngpgs_thresh, set the
> DB2_USE_ALTERNATE_PAGE_CLEANING to ON (with db2set).
>
> In the db cfg, set you logbuffsz to about 256. This requires that you
> increase DBHEAP by same amount since it is part of DBHEAP memory.- Hide quoted text -
>
> - Show quoted text -

This has become a performance tuning thread, but I'd use a cursor load
instead, great speed, no space usage..

On Jul 14, 5:10*am, Amber wrote:
> We are using 8.2.9 Windows 64 edtion, in one of our projects we need
> to recreate a few lager tables which have *many millions of rows each,
> we have used concurrent Java threads to read from data source and do
> batch inserts to the target tables concurrently, to speed up the
> inserts we have used the following configuration policies:
>
> 1. Use large page size (32K) and extend size (256) on the target db,
> 2. Use large buffer pools for the tablespaces which contains the large
> tables on the target db.
> 3. Set num_cleaners to 4 to speed up moving data from bufferpool to
> disks on the target db, we have 4 *4-core CPU on the server.
> 4. Set chngpgs_thresh to 80%, using this value we mean avoiding DB2 to
> moving data from bufferpool to disk too frequtenlly, because when
> doing these inserts we have not other readers on the target.
> 5. Use lage page size(32K) and extend size(256) and prefetch size
> (256) on the source db.
> 6. Use lage bufferpool on the source db.
> 7. Set num_ioservers to 4, we have 4 4-core cpu on the source db.
>
> We are not sure about the chngpgs_thresh value, any suggestions?

If you are using SMS tablespace, I highly recommend enabling multipage
file allocation. Formatting one page at a time is a big performance
hit. See http://publib.boulder.ibm.com/infoce...e/r0002054.htm.

HTH,
Norm