Re: BLOB + BTRIEVE = ?

There are a few possible limits involved.
1) In a normal file the fixed length portion of the record must fit on
a single data file page.  Indexes must reside on the fixed length
portion of the record.  Page sizes currently go up to 4K so indexes
must be in the first 4K of the record.  The index location is relaxed
somewhat with compressed files but I believe the compressed file total
record length is limitted to a 64K internal buffer.
2) The operation must fit into the 64K data buffer or 53K normal worst
case client / server limitted by network stack which I have seen
software chop to 32K and routers start dropping data at 16K. You can
use Get_Chunk* and Update_Chunk* operations to work on a data buffer
sized portion of the record at a time.
3) The total length of a record (fixed + variable) is limitted to 4GB
(32 bit pointer).  The practical limit is smaller.
4) Total single file size is limitted to 64GB with 4K pages, smaller
with smaller pages.  You can have as many files as you want.
*)  IDS does not support chunk operations.  IDS is pretty much a
legacy product and is in transition support.  
It has been superceeded by the Pervasive.SQL V8 Security release.

Hmmm.... very interesting.
He gave a mixed mode example creating the file through one API (SQL
Create Table statement) and accessing it through the transactional
API.  And in fact the way LVAR() fields are stored in the current
product has changed to allow more than one LVAR() per record which
requires some pointers to be stored in the record.  So with the
current product running that create table statement would produce a
file that did not quite match the example struct given.  If the file
is created through the Btrieve API it can still be created to match
the example or if accessed through the SQL API it would be
transparent.  Check out the current documentation at:
http://www.pervasive.com/developerZone/

It would also be possible to break the one arbitrarily large "file"
that needs to be stored into any number of arbitrary sized chunks, and
add a segment / sequence to the record.  Although in all honesty it is
probably better to just store the file as a file and eliminate the
complexity and overhead of accessing a file through a database engine.

Leonard

On Fri, 7 Nov 2003 23:42:47 +0100, "GTi" <[EMAIL PROTECTED]> wrote:

>Dave,
>Is this "maximum" limit restricted to Btrieve,
>or is this a common to all Databases ?
>
>Using files on disk is handy, but when using IDS it is not handy anymore...
>
>GTi
>
>
>"David Reed" <[EMAIL PROTECTED]> wrote in message
>news:[EMAIL PROTECTED]
>> You're correct about the maximum.
>>
>> Funny you mention pictures.
>>
>> I was faced with a similar situation earlier this year.  I ended up just
>> storing a path to an external file containing the picture.  Once I hit the
>> record I wanted, I'd just open the file at that location, which was a file
>> in the folder I was in.
>>
>> ...dave
>>
>>
>> "GTi" <[EMAIL PROTECTED]> wrote in message
>> news:[EMAIL PROTECTED]
>> > Thanks David,
>> >
>> > In your case you have a maximum length of 4000 bytes in the comment
>> field -
>> > right?
>> > And what if the comment fields is over 4000 bytes. What happens then?
>> >
>> > > comment          LVAR(4000))
>> > As I can see it, you MUST set a maximum length to something. If it is
>> 4000,
>> > 10000
>> > or 400000 it MUST be a value. Is that correct?
>> >
>> > Today I store the files on the disk, and the filename is the key value
>of
>> my
>> > record.
>> > That gives me no limits on the file size. But it should be in the
>> database.
>> > Say - For some years ago a digital camera was producing pictures of 1-2
>> MB.
>> > Today it have 5-6 MB.
>> > If I limited this to 3 MB for some years ago, I have been lost today.
>> > So if I go for this I must set a limit for the records to 100 MB - say
>200
>> > MB to be
>> > sure if I can still use it in the future (not limited to pictures).
>> > Or ?
>> >
>> >
>> >
>> > "David Reed" <[EMAIL PROTECTED]> wrote in message
>> > news:[EMAIL PROTECTED]
>> > > The following code uses the Windows SDK GlobalAlloc.  Feel free to
>> > > substitute the malloc of your choice.
>> > >
>> > > All my files were created via Novell's XQL, so I don't have code to
>tell
>> > you
>> > > how to create it.  The XQL statement was:
>> > >
>> > > create table TABLE using "TABLE .DAT" DCOMPRESS pagesize 4096
>> > >      (recordID             INT(4),
>> > >       code                 CHAR(8),
>> > >       reserved             CHAR(16),
>> > >       flags                INT(4),
>> > >       commentText          LVAR(4000))
>> > >            with index (recordID UNIQUE,
>> > >                        code UNIQUE MOD)
>> > >
>> > > Here's how I access it in the code:
>> > >
>> > > Ok, start with a structure for the fixed portion of the record:
>> > >
>> > > //
>> > > //  Table Structure
>> > > //  Record length: Variable: Fixed portion = 32
>> > > //  Defined flags: None
>> > > //
>> > > //  Record Layout:  Bytes  Description
>> > > //                  ~~~~~  ~~~~~~~~~~~
>> > > //                   1-4   Record ID
>> > > //                   5-12  Code
>> > > //                  13-28  Reserved
>> > > //                  29-32  Record Flags
>> > > //                  33-4032 Text (variable)
>> > > //
>> > > #define CODE_LENGTH           8
>> > > #define RESERVED_LENGTH      16
>> > > #define FIXED_LENGTH         32
>> > > #define VARIABLE_LENGTH    4000
>> > > #define TOTAL_LENGTH FIXED_LENGTH + VARIABLE_LENGTH
>> > >
>> > > typedef struct TABLEStruct
>> > > {
>> > >   long recordID;
>> > >   char code[CODE_LENGTH];
>> > >   char reserved[RESERVED_LENGTH];
>> > >   long flags;
>> > > } TABLEDef;
>> > >
>> > > typedef struct TABLEKey0Struct
>> > > {
>> > >   long recordID;
>> > > } TABLEKey0Def;
>> > >
>> > > typedef struct TABLEKey1Struct
>> > > {
>> > >   char code[COMMENTS_CODE_LENGTH];
>> > > } TABLEKey1Def;
>> > >
>> > > EXTERN TABLEDef     TABLE;
>> > > EXTERN TABLEKey1Def TABLEKey1;
>> > > EXTERN HANDLE hMemTableText;
>> > > EXTERN char *pTableText;
>> > >
>> > >
>> > > * * * *
>> > >
>> > > You can do the following in the main routine:
>> > >
>> > > //
>> > > //  Allocate space for the text portion of the table
>> > > //
>> > >   hMemTableText = GlobalAlloc(GPTR, TOTAL_LENGTH);
>> > >   pTableText = (char *)hMemTableText;
>> > >   if(hMemCommentText == NULL || pCommentText == NULL)
>> > >   {
>> > >     /* Error */
>> > >   }
>> > >
>> > > * * * * *
>> > >
>> > > In the routine you use to access all of the variable length file:
>> > >
>> > > char code[] = "SomeText";
>> > >
>> > > strcpy(TABLE.code, code);
>> > >
>> > > dataBufferLength = TOTAL_LENGTH;
>> > > rcode = BTRVID((BTI_WORD)GETEQUAL, positionBlock, pTableText ,
>> > > &dataBufferLength, TABLE.code, (BTI_SINT)1,
>(BTI_BUFFER_PTR)&clientID);
>> > >
>> > > if(rcode == 0)
>> > > {
>> > >   memcpy(&TABLE, pTableText, FIXED_LENGTH);
>> > >   /* Variable portion starts at pTableText[FIXED_LENGTH] */
>> > > }
>> > >
>> > > * * * * *
>> > > If you're just cycling through, set the data buffer length to
>> FIXED_LENGTH
>> > > and pass the data structure instead of pTable Text.  You'll get rcode
>==
>> > 22,
>> > > but you won't have the overhead of getting (potentially) a bunch of
>> > > additional data for records you may end up skipping.
>> > >
>> > > *******
>> > >
>> > > Bill Bach, if you're reading, is there much overhead savings in this?
>> > >
>> > >
>> > > Anyway, hope this helps.
>> > >
>> > > David Reed
>> > > Schedule Masters, Inc.
>> > >
>> > >
>> >
>> >
>>
>>
>