Alpha Software Mobile Development Tools:   Alpha Anywhere    |   Alpha TransForm subscribe to our YouTube Channel  Follow Us on LinkedIn  Follow Us on Twitter  Follow Us on Facebook

Announcement

Collapse

The Alpha Software Forum Participation Guidelines

The Alpha Software Forum is a free forum created for Alpha Software Developer Community to ask for help, exchange ideas, and share solutions. Alpha Software strives to create an environment where all members of the community can feel safe to participate. In order to ensure the Alpha Software Forum is a place where all feel welcome, forum participants are expected to behave as follows:
  • Be professional in your conduct
  • Be kind to others
  • Be constructive when giving feedback
  • Be open to new ideas and suggestions
  • Stay on topic


Be sure all comments and threads you post are respectful. Posts that contain any of the following content will be considered a violation of your agreement as a member of the Alpha Software Forum Community and will be moderated:
  • Spam.
  • Vulgar language.
  • Quotes from private conversations without permission, including pricing and other sales related discussions.
  • Personal attacks, insults, or subtle put-downs.
  • Harassment, bullying, threatening, mocking, shaming, or deriding anyone.
  • Sexist, racist, homophobic, transphobic, ableist, or otherwise discriminatory jokes and language.
  • Sexually explicit or violent material, links, or language.
  • Pirated, hacked, or copyright-infringing material.
  • Encouraging of others to engage in the above behaviors.


If a thread or post is found to contain any of the content outlined above, a moderator may choose to take one of the following actions:
  • Remove the Post or Thread - the content is removed from the forum.
  • Place the User in Moderation - all posts and new threads must be approved by a moderator before they are posted.
  • Temporarily Ban the User - user is banned from forum for a period of time.
  • Permanently Ban the User - user is permanently banned from the forum.


Moderators may also rename posts and threads if they are too generic or do not property reflect the content.

Moderators may move threads if they have been posted in the incorrect forum.

Threads/Posts questioning specific moderator decisions or actions (such as "why was a user banned?") are not allowed and will be removed.

The owners of Alpha Software Corporation (Forum Owner) reserve the right to remove, edit, move, or close any thread for any reason; or ban any forum member without notice, reason, or explanation.

Community members are encouraged to click the "Report Post" icon in the lower left of a given post if they feel the post is in violation of the rules. This will alert the Moderators to take a look.

Alpha Software Corporation may amend the guidelines from time to time and may also vary the procedures it sets out where appropriate in a particular case. Your agreement to comply with the guidelines will be deemed agreement to any changes to it.



Bonus TIPS for Successful Posting

Try a Search First
It is highly recommended that a Search be done on your topic before posting, as many questions have been answered in prior posts. As with any search engine, the shorter the search term, the more "hits" will be returned, but the more specific the search term is, the greater the relevance of those "hits". Searching for "table" might well return every message on the board while "tablesum" would greatly restrict the number of messages returned.

When you do post
First, make sure you are posting your question in the correct forum. For example, if you post an issue regarding Desktop applications on the Mobile & Browser Applications board , not only will your question not be seen by the appropriate audience, it may also be removed or relocated.

The more detail you provide about your problem or question, the more likely someone is to understand your request and be able to help. A sample database with a minimum of records (and its support files, zipped together) will make it much easier to diagnose issues with your application. Screen shots of error messages are especially helpful.

When explaining how to reproduce your problem, please be as detailed as possible. Describe every step, click-by-click and keypress-by-keypress. Otherwise when others try to duplicate your problem, they may do something slightly different and end up with different results.

A note about attachments
You may only attach one file to each message. Attachment file size is limited to 2MB. If you need to include several files, you may do so by zipping them into a single archive.

If you forgot to attach your files to your post, please do NOT create a new thread. Instead, reply to your original message and attach the file there.

When attaching screen shots, it is best to attach an image file (.BMP, .JPG, .GIF, .PNG, etc.) or a zip file of several images, as opposed to a Word document containing the screen shots. Because Word documents are prone to viruses, many message board users will not open your Word file, therefore limiting their ability to help you.

Similarly, if you are uploading a zipped archive, you should simply create a .ZIP file and not a self-extracting .EXE as many users will not run your EXE file.
See more
See less

A5 chokes on import of 2 million row .csv file

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • A5 chokes on import of 2 million row .csv file

    I am using Alpha Five ver 11 on a Windows Vista PC. I am using .dbf files, not SQL.

    I have a 1.9 million row .csv file that I want to import into A5.

    The file only has seven columns; not very wide, just quite tall.

    MS-Access has imported this file just fine, but A5 just hangs on it.

    I have used A5 to sucessfully Import many other .csv files so I am familiar with the process.

    Thanks for helping - this is a critical need.

  • #2
    Re: A5 chokes on import of 2 million row .csv file

    A little more information would be useful.

    What happens when A5 "chokes"?

    What method or approach are you using to do this? Do have some code you can show us?

    Are you using the same machine that works for Access to also do this in A5? If not, is there a difference in available RAM?
    -Steve
    sigpic

    Comment


    • #3
      Re: A5 chokes on import of 2 million row .csv file

      Have you tried dividing the .csv file into several files?

      Comment


      • #4
        Re: A5 chokes on import of 2 million row .csv file

        Are you reaching the max size of dbf files ??

        Comment


        • #5
          Re: A5 chokes on import of 2 million row .csv file

          Hi Daniel,

          Originally posted by JimDK View Post
          Are you reaching the max size of dbf files ??
          Specifically, there is a 2 Gigabyte limit on any single file imposed by windows and the DBF format.

          Assuming each line of the CSV is less than 1000 or so characters, the size of the original file will be under 2 gigabytes, but it's likely that is not the issue. My guess is the csv file is probably under 100 Megabytes, and that would not cause a problem for A5 either. However, if the row delimiter is just a Carriage return (ascii 13) or just a line feed (chr 10), that could be a problem. But you could easily fix that with the function GET_FROM_FILE(), SAVE_TO_FILE(), and STRTRAN() functions.

          If the structure of the DBF is over 1000 or so characters per record, you would also exceed the maximum 2 Gigabyte size for the dbf file if you tried an import that large.

          It is also possible you have another, non-ASCII character in a field. I suggest 1st trying the import with a small subset of the file, and see if that works. If that works, you should be able to isolate the choke point by trying a version running it against subsets of the entire file and find where it has issues until you locate the spot or spots that cause trouble.
          Regards,

          Ira J. Perlow
          Computer Systems Design


          CSDA A5 Products
          New - Free CSDA DiagInfo - v1.39, 30 Apr 2013
          CSDA Barcode Functions

          CSDA Code Utility
          CSDA Screen Capture


          Comment


          • #6
            Re: A5 chokes on import of 2 million row .csv file

            "chokes" = A5 hangs after the import process begins.

            Comment


            • #7
              Re: A5 chokes on import of 2 million row .csv file

              Originally posted by WaltMorgan View Post
              Have you tried dividing the .csv file into several files?
              Yes. A5 also hangs on 852,000 row subset.

              Comment


              • #8
                Re: A5 chokes on import of 2 million row .csv file

                Originally posted by csda1 View Post
                Hi Daniel,


                Specifically, there is a 2 Gigabyte limit on any single file imposed by windows and the DBF format.

                Assuming each line of the CSV is less than 1000 or so characters, the size of the original file will be under 2 gigabytes, but it's likely that is not the issue. My guess is the csv file is probably under 100 Megabytes, and that would not cause a problem for A5 either. However, if the row delimiter is just a Carriage return (ascii 13) or just a line feed (chr 10), that could be a problem. But you could easily fix that with the function GET_FROM_FILE(), SAVE_TO_FILE(), and STRTRAN() functions.

                If the structure of the DBF is over 1000 or so characters per record, you would also exceed the maximum 2 Gigabyte size for the dbf file if you tried an import that large.

                It is also possible you have another, non-ASCII character in a field. I suggest 1st trying the import with a small subset of the file, and see if that works. If that works, you should be able to isolate the choke point by trying a version running it against subsets of the entire file and find where it has issues until you locate the spot or spots that cause trouble.
                There are actually two .csv files - same structure but different row count, different data - one is 102 Mb and the other is 78 Mb. A5 hangs on the smaller one (big one too). If their were some "poison" character(s) in the data then I would have thought Access would have barfed on it too but it did not.

                Comment


                • #9
                  Re: A5 chokes on import of 2 million row .csv file

                  DB_Dan reread Ira's post. I just spent like 15 hours working on something that has worked for years and now does not. This to is an import.
                  I had bad characters for a field that it could not except. You have to look through the CSV file for abnormal data and Ira is correct break it
                  down to smaller sets and try. Good Luck I got mine working now off for some more Holiday Cheer.
                  Nicholas Wieland
                  LedgerSuite.com Corp
                  [email protected]
                  http://www.ledgersuite.com

                  Comment


                  • #10
                    Re: A5 chokes on import of 2 million row .csv file

                    Hi Daniel,

                    Originally posted by DB_Dan View Post
                    If their were some "poison" character(s) in the data then I would have thought Access would have barfed on it too but it did not.
                    Why would you ever assume that 1 program would do the same thing as another? Different people programmed them. Some people assume just because they used a MID() function in some program that it must exist in A5. The equivalent is there with SUBSTR(), but they are not the same either.

                    If you use Alpha to display the bad row of data, you may not see the character, or it may appear as a rectangle in Alpha Five. I use a binary (hex) display program to see what actual ASCII codes are there. You could use Alpha commands to read the file is as a blob and then dump the values as hex characters too to find the bad data.

                    Also, sometimes it is typed wrong as well. Imagine you were importing to a number field and had the letter B in the CSV. Imagine you had an invalid date imported into a date field. I'm not sure what Alpha does in those cases, and it may have changed from version to version.

                    A basic Database rule is that foreign data (ones you did not generate) should always be cleaned up before importing into your data as you have no idea of the data quality. This can be bad data, un-normalized values, wrong case values, different format (e.g. dates), etc.
                    Regards,

                    Ira J. Perlow
                    Computer Systems Design


                    CSDA A5 Products
                    New - Free CSDA DiagInfo - v1.39, 30 Apr 2013
                    CSDA Barcode Functions

                    CSDA Code Utility
                    CSDA Screen Capture


                    Comment


                    • #11
                      Re: A5 chokes on import of 2 million row .csv file

                      Thanks everyone for the help. If I could go 'back in the day' to my command line UNIX (anyone remember SCO?) there would be no problem examining or cleaning up the data. Looks like I am finally going to have to learn to live in/with Windows.

                      Comment


                      • #12
                        Re: A5 chokes on import of 2 million row .csv file

                        Dan use should be able to open the data in notepad and review it it is text.
                        Nicholas Wieland
                        LedgerSuite.com Corp
                        [email protected]
                        http://www.ledgersuite.com

                        Comment


                        • #13
                          Re: A5 chokes on import of 2 million row .csv file

                          Originally posted by DB_Dan View Post
                          Thanks everyone for the help. If I could go 'back in the day' to my command line UNIX (anyone remember SCO?) there would be no problem examining or cleaning up the data. Looks like I am finally going to have to learn to live in/with Windows.
                          Sure! it's still on the market

                          Comment


                          • #14
                            Re: A5 chokes on import of 2 million row .csv file

                            Hi Daniel,
                            This may be way out there, but...having successfully imported the file into Access can you now import it from Access into your DBF using File/Import Data and then stepping through the DAO import Genie. It'd be interesting to see what happens at least.

                            -John

                            Comment


                            • #15
                              Re: A5 chokes on import of 2 million row .csv file

                              As Ira has mentioned, bad characters in the file can cause problems for Alpha's import, sometimes when I import files with Danish characters in (,,), I have to import to Access first, and them import from the Access table to Alpha's dbf files, it seems like Access import is more tolerant than Alpha.

                              Comment

                              Working...
                              X