Alpha Software Mobile Development Tools:   Alpha Anywhere    |   Alpha TransForm subscribe to our YouTube Channel  Follow Us on LinkedIn  Follow Us on Twitter  Follow Us on Facebook

Announcement

Collapse

The Alpha Software Forum Participation Guidelines

The Alpha Software Forum is a free forum created for Alpha Software Developer Community to ask for help, exchange ideas, and share solutions. Alpha Software strives to create an environment where all members of the community can feel safe to participate. In order to ensure the Alpha Software Forum is a place where all feel welcome, forum participants are expected to behave as follows:
  • Be professional in your conduct
  • Be kind to others
  • Be constructive when giving feedback
  • Be open to new ideas and suggestions
  • Stay on topic


Be sure all comments and threads you post are respectful. Posts that contain any of the following content will be considered a violation of your agreement as a member of the Alpha Software Forum Community and will be moderated:
  • Spam.
  • Vulgar language.
  • Quotes from private conversations without permission, including pricing and other sales related discussions.
  • Personal attacks, insults, or subtle put-downs.
  • Harassment, bullying, threatening, mocking, shaming, or deriding anyone.
  • Sexist, racist, homophobic, transphobic, ableist, or otherwise discriminatory jokes and language.
  • Sexually explicit or violent material, links, or language.
  • Pirated, hacked, or copyright-infringing material.
  • Encouraging of others to engage in the above behaviors.


If a thread or post is found to contain any of the content outlined above, a moderator may choose to take one of the following actions:
  • Remove the Post or Thread - the content is removed from the forum.
  • Place the User in Moderation - all posts and new threads must be approved by a moderator before they are posted.
  • Temporarily Ban the User - user is banned from forum for a period of time.
  • Permanently Ban the User - user is permanently banned from the forum.


Moderators may also rename posts and threads if they are too generic or do not property reflect the content.

Moderators may move threads if they have been posted in the incorrect forum.

Threads/Posts questioning specific moderator decisions or actions (such as "why was a user banned?") are not allowed and will be removed.

The owners of Alpha Software Corporation (Forum Owner) reserve the right to remove, edit, move, or close any thread for any reason; or ban any forum member without notice, reason, or explanation.

Community members are encouraged to click the "Report Post" icon in the lower left of a given post if they feel the post is in violation of the rules. This will alert the Moderators to take a look.

Alpha Software Corporation may amend the guidelines from time to time and may also vary the procedures it sets out where appropriate in a particular case. Your agreement to comply with the guidelines will be deemed agreement to any changes to it.



Bonus TIPS for Successful Posting

Try a Search First
It is highly recommended that a Search be done on your topic before posting, as many questions have been answered in prior posts. As with any search engine, the shorter the search term, the more "hits" will be returned, but the more specific the search term is, the greater the relevance of those "hits". Searching for "table" might well return every message on the board while "tablesum" would greatly restrict the number of messages returned.

When you do post
First, make sure you are posting your question in the correct forum. For example, if you post an issue regarding Desktop applications on the Mobile & Browser Applications board , not only will your question not be seen by the appropriate audience, it may also be removed or relocated.

The more detail you provide about your problem or question, the more likely someone is to understand your request and be able to help. A sample database with a minimum of records (and its support files, zipped together) will make it much easier to diagnose issues with your application. Screen shots of error messages are especially helpful.

When explaining how to reproduce your problem, please be as detailed as possible. Describe every step, click-by-click and keypress-by-keypress. Otherwise when others try to duplicate your problem, they may do something slightly different and end up with different results.

A note about attachments
You may only attach one file to each message. Attachment file size is limited to 2MB. If you need to include several files, you may do so by zipping them into a single archive.

If you forgot to attach your files to your post, please do NOT create a new thread. Instead, reply to your original message and attach the file there.

When attaching screen shots, it is best to attach an image file (.BMP, .JPG, .GIF, .PNG, etc.) or a zip file of several images, as opposed to a Word document containing the screen shots. Because Word documents are prone to viruses, many message board users will not open your Word file, therefore limiting their ability to help you.

Similarly, if you are uploading a zipped archive, you should simply create a .ZIP file and not a self-extracting .EXE as many users will not run your EXE file.
See more
See less

Splitting a Large Text File

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Splitting a Large Text File

    I have a very large text file that I would like to split into several parts prior to importing. I know the size of the file in bytes. I also now the size in bytes of each "row" in the file. I would like to be able to create at least four files from the original source file. I have tried using the .readb() function but I am getting an error that says that I need to contact Alpha Software and report the error (I have never seen A5 or A4 error message that asked me to call Alpha Software before!). So my first assumption is that I am using the wrong function or using it incorrectly. The size of the source text file is 2.3 Gigabytes.

    Thanks in advance for anyone's help,

    CPGood

  • #2
    Re: Splitting a Large Text File

    So my first assumption is that I am using the wrong function or using it incorrectly.
    Could be. Either post your code here or move over to the left a bit. I'm having trouble seeing your screen over your shoulder.
    There can be only one.

    Comment


    • #3
      Re: Splitting a Large Text File

      Originally posted by Stan Mathews View Post
      I'm having trouble seeing your screen over your shoulder.
      Did you lose your 'special' eyeglasses again???
      Al Buchholz
      Bookwood Systems, LTD
      Weekly QReportBuilder Webinars Thursday 1 pm CST

      Occam's Razor - KISS
      Normalize till it hurts - De-normalize till it works.
      Advice offered and questions asked in the spirit of learning how to fish is better than someone giving you a fish.
      When we triage a problem it is much easier to read sample systems than to read a mind.
      "Make it as simple as possible, but not simpler."
      Albert Einstein

      http://www.iadn.com/images/media/iadn_member.png

      Comment


      • #4
        Re: Splitting a Large Text File

        Originally posted by Al Buchholz View Post
        Did you lose your 'special' eyeglasses again???
        Not lost, they're at the cleaners.

        This works for me on a 600 meg file in just under 3 minutes.

        Code:
        st = now()
        fil = file.open("P:\executive\hdb\temp\trakstor.txt",FILE_RW_EXCLUSIVE)
        lns = 0
        while .not. fil.eof()
        	fil.read_line()
        	lns = lns+1
        end while
        fst = ceiling(lns/4)
        sec = fst
        thd = sec
        fth = lns-fst-sec-thd
        fil.seek(0)
        out_fil = file.create("P:\executive\hdb\temp\output_1.txt",FILE_RW_EXCLUSIVE)
        for qxd = 1 to fst
        	fil_lin = fil.read_line()
        	out_fil.write_line(fil_lin)
        next qxd
        out_fil.flush()
        out_fil.close()
        
        out_fil = file.create("P:\executive\hdb\temp\output_2.txt",FILE_RW_EXCLUSIVE)
        for qxd = 1 to sec
        	fil_lin = fil.read_line()
        	out_fil.write_line(fil_lin)
        next qxd
        out_fil.flush()
        out_fil.close()
        	
        out_fil = file.create("P:\executive\hdb\temp\output_3.txt",FILE_RW_EXCLUSIVE)
        for qxd = 1 to thd
        	fil_lin = fil.read_line()
        	out_fil.write_line(fil_lin)
        next qxd
        out_fil.flush()
        out_fil.close()	
        
        out_fil = file.create("P:\executive\hdb\temp\output_4.txt",FILE_RW_EXCLUSIVE)
        for qxd = 1 to fth
        	fil_lin = fil.read_line()
        	out_fil.write_line(fil_lin)
        next qxd
        out_fil.flush()
        out_fil.close()
        fil.close()
        nd = now()
        ui_msg_box("Finished","Took "+alltrim(str(nd-st))+ " seconds.")
        There can be only one.

        Comment


        • #5
          Re: Splitting a Large Text File

          If you know how many rows.. and the size of each row.. it's safe to assume that this text file is representative of delimited records from another application, either that.. or your psychic abilities far exceeds any known to man!.

          I put my money on the former assumption and if my assumption is correct, I'd say import it as records or as an array. Array would be the fastest way to do that.

          Comment


          • #6
            Re: Splitting a Large Text File

            Sorry I'm late adding to this thread.

            Any time you are dealing with reading/writing lots of data, eliminating disk seeks can give you INCREDIBLE performance improvements ranging from a low 2x to as much as 40x or so.

            So for cases like the above where you are dealing with a lot of data, there are some speed principles to always consider.

            1) Always process data files in huge blocks. If you can read all the input into memory first, that is great because it eliminates lots of seeks. And then rather than writing output to several files at the same time, process them sequentially one file at a time (unless you can write them to separate physical devices).

            2) If you can't fit it all into memory, select a separate hard disk for temporary output. That will again eliminate time-robbing seeks that occur back and forth if both the reads and the writes occur on the same physical drive. (Note that just using a separate partition does not help; it helps ONLY when they are separate physical devices.) You can see incredible improvements by attaching a $50 USB drive and using that to help physically separate the data. And even with the cost of copying the data from the temp device, it's still worth it.

            3) Make your processing loops as small as possible. A5 is interpreted code and is therefore not NEARLY as fast as truly compiled code (or assembly language).

            4) Redesign your process in order to reduce disk seeks.

            5) If there's an A5 function or method to do what you want, choose it rather than coding it yourself (unless your approach is proven to be faster).

            6) For those of you with the background and determination, if the process is compute-intensive, consider writing in C/C++/C# -- or assembly language, if you dare. Time or profile your code to see what's taking the most time, and that can help indicate what you should work on.

            You can usually see a 2x to 5x improvement, and I've seen as much as a 40x (where a two-hour report process was reduced to three minutes through a two-day redesign process - that was not using A5, however, but the principles work with any language).

            Food for thought -- and for more speed!

            Regards,

            Eric
            There are three kinds of people in the world: those who can count, and those who can't.

            Comment


            • #7
              Re: Splitting a Large Text File

              I'm going back a bit, but I'm pretty sure that there is a DOS command to split and reassemble files.
              Has DOS not gots some features still in the "Run" Command Line
              See our Hybrid Option here;
              https://hybridapps.example-software.com/


              There are 3 things you can do in any situation;
              Live with it, Change it or Walk away from it.

              Comment


              • #8
                Re: Splitting a Large Text File

                Ted,

                Many years back, I was in the business of creating DOS utilities, and I don't remember such a command. There were some third-party utilities that would do this, and it's not too hard to create one. It's like backing up files; you just have to have a fail-proof way of knowing how to put the pieces back together (which is what I did in my old DOS backup program, Back-It).

                Eric
                There are three kinds of people in the world: those who can count, and those who can't.

                Comment


                • #9
                  Re: Splitting a Large Text File

                  Maybe I'm thinking of something in the old BASIC commands (mid 70's).
                  I'lll look at my books whan I get home next.
                  See our Hybrid Option here;
                  https://hybridapps.example-software.com/


                  There are 3 things you can do in any situation;
                  Live with it, Change it or Walk away from it.

                  Comment

                  Working...
                  X