Get a Jumpstart with our Sample Apps
Results 1 to 9 of 9

Thread: Splitting a Large Text File

  1. #1
    Member
    Real Name
    CP Good
    Join Date
    Sep 2009
    Posts
    241

    Default Splitting a Large Text File

    I have a very large text file that I would like to split into several parts prior to importing. I know the size of the file in bytes. I also now the size in bytes of each "row" in the file. I would like to be able to create at least four files from the original source file. I have tried using the .readb() function but I am getting an error that says that I need to contact Alpha Software and report the error (I have never seen A5 or A4 error message that asked me to call Alpha Software before!). So my first assumption is that I am using the wrong function or using it incorrectly. The size of the source text file is 2.3 Gigabytes.

    Thanks in advance for anyone's help,

    CPGood

  2. #2
    "Certified" Alphaholic Stan Mathews's Avatar
    Real Name
    Stan Mathews
    Join Date
    Apr 2000
    Location
    Bowling Green, KY
    Posts
    24,698

    Default Re: Splitting a Large Text File

    So my first assumption is that I am using the wrong function or using it incorrectly.
    Could be. Either post your code here or move over to the left a bit. I'm having trouble seeing your screen over your shoulder.
    There can be only one.

  3. #3
    Moderator
    Real Name
    Alan Buchholz
    Join Date
    Oct 2000
    Location
    Delavan, Wisconsin
    Posts
    9,383

    Default Re: Splitting a Large Text File

    Quote Originally Posted by Stan Mathews View Post
    I'm having trouble seeing your screen over your shoulder.
    Did you lose your 'special' eyeglasses again???
    Al Buchholz
    Bookwood Systems, LTD
    Weekly QReportBuilder Webinars Thursday 1 pm CST

    Occam's Razor - KISS
    Normalize till it hurts - De-normalize till it works.
    Advice offered and questions asked in the spirit of learning how to fish is better than someone giving you a fish.
    When we triage a problem it is much easier to read sample systems than to read a mind.

  4. #4
    "Certified" Alphaholic Stan Mathews's Avatar
    Real Name
    Stan Mathews
    Join Date
    Apr 2000
    Location
    Bowling Green, KY
    Posts
    24,698

    Default Re: Splitting a Large Text File

    Quote Originally Posted by Al Buchholz View Post
    Did you lose your 'special' eyeglasses again???
    Not lost, they're at the cleaners.

    This works for me on a 600 meg file in just under 3 minutes.

    Code:
    st = now()
    fil = file.open("P:\executive\hdb\temp\trakstor.txt",FILE_RW_EXCLUSIVE)
    lns = 0
    while .not. fil.eof()
    	fil.read_line()
    	lns = lns+1
    end while
    fst = ceiling(lns/4)
    sec = fst
    thd = sec
    fth = lns-fst-sec-thd
    fil.seek(0)
    out_fil = file.create("P:\executive\hdb\temp\output_1.txt",FILE_RW_EXCLUSIVE)
    for qxd = 1 to fst
    	fil_lin = fil.read_line()
    	out_fil.write_line(fil_lin)
    next qxd
    out_fil.flush()
    out_fil.close()
    
    out_fil = file.create("P:\executive\hdb\temp\output_2.txt",FILE_RW_EXCLUSIVE)
    for qxd = 1 to sec
    	fil_lin = fil.read_line()
    	out_fil.write_line(fil_lin)
    next qxd
    out_fil.flush()
    out_fil.close()
    	
    out_fil = file.create("P:\executive\hdb\temp\output_3.txt",FILE_RW_EXCLUSIVE)
    for qxd = 1 to thd
    	fil_lin = fil.read_line()
    	out_fil.write_line(fil_lin)
    next qxd
    out_fil.flush()
    out_fil.close()	
    
    out_fil = file.create("P:\executive\hdb\temp\output_4.txt",FILE_RW_EXCLUSIVE)
    for qxd = 1 to fth
    	fil_lin = fil.read_line()
    	out_fil.write_line(fil_lin)
    next qxd
    out_fil.flush()
    out_fil.close()
    fil.close()
    nd = now()
    ui_msg_box("Finished","Took "+alltrim(str(nd-st))+ " seconds.")
    There can be only one.

  5. #5
    "Certified" Alphaholic G Gabriel's Avatar
    Real Name
    G. Gabriel
    Join Date
    Oct 2004
    Posts
    7,182

    Default Re: Splitting a Large Text File

    If you know how many rows.. and the size of each row.. it's safe to assume that this text file is representative of delimited records from another application, either that.. or your psychic abilities far exceeds any known to man!.

    I put my money on the former assumption and if my assumption is correct, I'd say import it as records or as an array. Array would be the fastest way to do that.

  6. #6
    Member
    Real Name
    Eric Ruff
    Join Date
    Aug 2007
    Location
    Pleasant Grove, UT
    Posts
    104

    Default Re: Splitting a Large Text File

    Sorry I'm late adding to this thread.

    Any time you are dealing with reading/writing lots of data, eliminating disk seeks can give you INCREDIBLE performance improvements ranging from a low 2x to as much as 40x or so.

    So for cases like the above where you are dealing with a lot of data, there are some speed principles to always consider.

    1) Always process data files in huge blocks. If you can read all the input into memory first, that is great because it eliminates lots of seeks. And then rather than writing output to several files at the same time, process them sequentially one file at a time (unless you can write them to separate physical devices).

    2) If you can't fit it all into memory, select a separate hard disk for temporary output. That will again eliminate time-robbing seeks that occur back and forth if both the reads and the writes occur on the same physical drive. (Note that just using a separate partition does not help; it helps ONLY when they are separate physical devices.) You can see incredible improvements by attaching a $50 USB drive and using that to help physically separate the data. And even with the cost of copying the data from the temp device, it's still worth it.

    3) Make your processing loops as small as possible. A5 is interpreted code and is therefore not NEARLY as fast as truly compiled code (or assembly language).

    4) Redesign your process in order to reduce disk seeks.

    5) If there's an A5 function or method to do what you want, choose it rather than coding it yourself (unless your approach is proven to be faster).

    6) For those of you with the background and determination, if the process is compute-intensive, consider writing in C/C++/C# -- or assembly language, if you dare. Time or profile your code to see what's taking the most time, and that can help indicate what you should work on.

    You can usually see a 2x to 5x improvement, and I've seen as much as a 40x (where a two-hour report process was reduced to three minutes through a two-day redesign process - that was not using A5, however, but the principles work with any language).

    Food for thought -- and for more speed!

    Regards,

    Eric
    There are three kinds of people in the world: those who can count, and those who can't.

  7. #7
    "Certified" Alphaholic Ted Giles's Avatar
    Real Name
    Ted Giles
    Join Date
    Aug 2000
    Location
    In the Wolds, Louth, Lincolnshire, UK
    Posts
    3,812

    Default Re: Splitting a Large Text File

    I'm going back a bit, but I'm pretty sure that there is a DOS command to split and reassemble files.
    Has DOS not gots some features still in the "Run" Command Line

  8. #8
    Member
    Real Name
    Eric Ruff
    Join Date
    Aug 2007
    Location
    Pleasant Grove, UT
    Posts
    104

    Default Re: Splitting a Large Text File

    Ted,

    Many years back, I was in the business of creating DOS utilities, and I don't remember such a command. There were some third-party utilities that would do this, and it's not too hard to create one. It's like backing up files; you just have to have a fail-proof way of knowing how to put the pieces back together (which is what I did in my old DOS backup program, Back-It).

    Eric
    There are three kinds of people in the world: those who can count, and those who can't.

  9. #9
    "Certified" Alphaholic Ted Giles's Avatar
    Real Name
    Ted Giles
    Join Date
    Aug 2000
    Location
    In the Wolds, Louth, Lincolnshire, UK
    Posts
    3,812

    Default Re: Splitting a Large Text File

    Maybe I'm thinking of something in the old BASIC commands (mid 70's).
    I'lll look at my books whan I get home next.

Similar Threads

  1. Profile window text TOO LARGE
    By Wanda Tucker in forum Web Application Server v6
    Replies: 22
    Last Post: 08-04-2005, 10:14 AM
  2. Large Open Text Field
    By Rich Hartnett in forum Web Application Server v6
    Replies: 3
    Last Post: 07-06-2005, 06:54 AM
  3. How Large a File does A5 Support?
    By Dana Shields in forum Alpha Five Version 5
    Replies: 2
    Last Post: 08-11-2003, 08:35 AM
  4. Large ALM file
    By richkerr in forum Alpha Five Version 5
    Replies: 6
    Last Post: 03-30-2003, 04:43 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •