Re: Bulk Load - performance degrades

[EMAIL PROTECTED] (Chad) wrote in message news:<[EMAIL PROTECTED]>...

> I am writing an application which continuously creates tables of 5
> million records. The time to load one such table is about 18 minutes.
>
> If I run the app 20 times to creates 1 table each time it takes 18
> minutes to create each table. However if I run the app once and tell
> it to create 20 tables (in one application invocation) the create time
> eventually shoots up to 38 minutes per table.

Are there 20 threads of control running in parallel to create
20 tables, or a single thread of control that's creating 20
tables sequentially?

> What could be causing this ? I have tried closing tables (Dbs) as I go
> but that did not improve performance much. Note, due to sort order,
> the locality of reference is not great in this application. A batch of
> 20,000 new records could cause updates to pages spread right across
> the full width of the tree. However I would expect the create time of
> equal sized tables to be constant regardless.

You are almost certainly, for some reason, seeing more cache
misses in the second case.  You can monitor this by reviewing
the cache statistics displayed by the db_stat utility that is
included in the Berkeley DB distribution. 
     
Regards,
--keith
 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Keith Bostic                          [EMAIL PROTECTED]
Sleepycat Software Inc.        +1-781-259-3139
Lincoln, MA 01773               http://www.sleepycat.com