Another attempt at building a csv line parser for Alpha5

[brett s]

If you need more programmatic control over the importation of CSV files than the Alpha5 standard functions provide, here's my latest attempt: CsvLineParse2(), a user UDF for use with Xbasic.

Input to this function is a line of csv field names or field data to parse. Output are the field names or data as a CRLF separated list, which is a format easily handled in Alpha5 Xbasic.

Example:

�"She sells, sea shells", "on the sea, shore",12345�

Becomes:

�She sells, sea shells
on the sea, shore
12345�

The csv conversion is done according to the RFC 4180 standard. More information here.

Particulary, the parsing must take account of commas within quoted fields and quotes within quoted fields. In the latter case quotes are escaped with a second quote, as in "". Example: �"""Hello"", she said."� is parsed as: �"Hello", she said.�

Commas can be field separators, so the code has to differentiate between comma field separators and commas that are not field separators. Also between quotes that are field delimiters and quotes that are just part of the field contents.

Not particularly efficient at 1.3 seconds per thousand records processed (on my PC with a Windows Experience Index of 5.6), this code does however work. I am not however yet claiming it is entirely bug free!

Preferably I would like to see the processing speed under 0.5 seconds per thousand records processed. But that will have to be for another day.

I have tested and so far been unable to find on the Internet any other code, including regular expressions, that is both suitable for use with Alpha5 and that meets the RFC 4180 standard. But I am still looking.

Attached is a zip file containing the Xbasic function CsvLineParse2(), a test Xbasic script and a test file of CSV data.

Any suggestions or improvements, just let me know.

[Keith Hubert]

Re: Another attempt at building a csv line parser for Alpha5

Hi Brett,

I have not looked at you sample yet but have a look at this function and see what it does to the speed of your Parse.

tbl.batch_begin()

your process

tbl.batch_end()

Let me know how this works for you.

[brett s]

Re: Another attempt at building a csv line parser for Alpha5

Hi Keith. Yes, good suggestion. I use batch_begin and end a lot. They make big improvements in the speed of processing of tables when both reading and updating a table at the same time. Unfortunately here though I am reading and not updating a csv text file and not a table. So there is no corresponding improvement in throughput to be expected. I tend to use <file>.read_line(). There is probably a much more efficient way to process a text file but I haven't stumbled across it yet.