Error importing large files

[Scott Cave]

I have been using A5 v5 successfully to import and query large text files (1.5 - 2 GB) with no problems (it's a bit slow but it works). Recently I decided to test these operations in v6. I downloaded and installed the trial version on a new computer, copied over the database files containing the v5 database with the import operations and queries defined, and then tried to import a 1.5 GB text file.

When the import operation was near completion I received an error message stating that it could not find the $$xxxxx.mdx file where xxxxx represents characters and numbers. I've run this import operation several times and each time the characters and numbers in this filename are different.

I've not had this problem imported this same file in v5 so I'm wondering if this is a bug in v6? Or does v6 have a new limit on file sizes (the table that the file was importing into had grown to just over 2 GB when the operation stopped)?

[Lenny Forziati]

RE: Error importing large files

2GB is the maximum file size of a DBF. This is not new in A5v6, it is a limitation of the DBF format.

-Lenny

[csda1]

RE: Error importing large files

Lenny,

Is this a limitation of the DBF as you said, or is it of the OS (e.g FAT32 partitions are limited to 2GB, NTFS partitions are not).

If it truly is the DBF, why should there really be any limitation?

Since the dbf format is a fixed size record format, I don't think there is any real file size limitation in the dbf file itself (the dbf header), although the index files may have some inherent limitation in their structure.

Regards,

Ira

[Scott Cave]

RE: Error importing large files

I'm rather new to Alpha5 and when I purchased it I was told by the people who offered the sales webinar that there was no file size limitation. I would also like to know if the .dbf 2 GB restriction is dependent upon OS. Additionally, is there a way in Alpha5 to save the table in a format other than .dbf to bypass the 2 GB restriction?

[Tom Cone Jr]

RE: Error importing large files

Ira,

I've always thought that the max record count was limited to 2 GB since only 4 bytes are available in the DBF format to describe the current record count in the table. I'm no wiz with binary math but I think the largest binary number one can store in 4 bytes is 2 to the 32nd power, or 2 GB. One would multiply the max record count times the record length to get a max file size, right?

-- tom

[csda1]

RE: Error importing large files

Hi Tom,

Actually 2 to the 31st power is 2 GB, not 2 the 32nd. But maybe there is a bit used for something else. But that is exactly right (which is why I said there would be no practical limitation). So, I wouldn't think the dbf file size itself would be limited by a 2GB (or even a 4 GB) record count.

The memo field file uses 10 byte pointers. Even if this is stored as an ASCII value (as opposed to a 10 byte binary pointer) which I believe is the case, it would still allow 10 billion memo entries/changes. So I don't think that is a limitation here.

That leaves the index file storage as the last possible bastion of a physical limitation, otherwise it must be a code library limitation of the dbf engine. I suspect Alpha has tweaked the code enough, so I can't see a limitation there, except the size of an offset pointer into the physical file may not have enough precision to get you there. Seems in that case, it could be easily overcome. For the index, there might be some issues, but I don't know.

On the other hand, I don't know anyone else that is actually sharing the index and dbf structure of Alpha 5 at the same time as Alpha is operating, so that might mean the index file structure is open to actually be played with and modified (enhanced). You might lose some compatibility, but I don't think anyone is really using it either.

Regards,

Ira

[Steve Workings]

RE: Error importing large files

I can't argue and discuss the technical stuff like Ira can, but I can tell you from my experience that a .dbf file is limited to 2 GB in size. I've run into it and had to make adjustments to stay under it.

Some things in life we have to accept, and that's one of them.

- Steve

[Lenny Forziati]

RE: Error importing large files

Ira,

The max file size for FAT is 2GB, for FAT 32 it is 4GB and on NTFS it is limited only by the available disk space.

The DBF itself imposes an additional 2GB file size limit. I typically work at the Xbasic level and am not overly familiar with the DBF format's internals so I am admittedly not qualified to give you the exact reason for this limitation. I do know however that it is common among other DBF-based products including Foxpro

-Lenny

[Lenny Forziati]

RE: Error importing large files

It is not dependant on OS, though if you use a FAT filesystem you will be further restricted to 1GB per file.

The current version of Alpha Five only works natively with DBF files. However it can use other ADO data sources in a limted way.

We are working on a MySQL version and a general client/server version of Alpha Five. These versions will allow you to use MySQL or any ADO data source as a back-end and work with the data as if it were a DBF. With these versions, you woould not be limited by a DBF file size but instead by the maximum table size of the engine you chose.

-Lenny

[csda1]

RE: Error importing large files

Lenny,

Although going through this layer of ADO and the like, I suspect there is at least some additional overhead that would slow things a bit. As to whether this is significant overhead (particularly with MySQL where I know Alpha's interface been massaged for the best possible speed) or not, will have to be determined by testing etc.

Regards,

Ira