alltrim in function help

[Pat Bremkamp]

I created a function to clean up lists, as follows:

 
function list_cleanup as c(vList as C)
 vList=comma_to_crlf(vList)
 *for_each(X,alltrim(X),vList)
 vList=remove_blank_lines(vList)
 if vList=""
  goto Endit
 end if
 vList=crlf_to_comma(vList)
 vList=sortsubstr(vList,",")
 vList=*remove_duplicates(vList,",")
 endit:
 list_cleanup=vlist
end function

This works fine in all the cases I've tested, except that alltrimming the individual elements doesn't seem to work. That is

list_cleanup("abc, def,abc")
yields
"abc, def"
It doesn't strip the leading blank off the second element.

Anyone see what I'm doing wrong?

Pat

[Al Buchholz]

Re: alltrim in fuction help

Pat

Replace your *for_each line with this. That'll tell it to replace vlist with the processed list.

vlist=*for_each(X2,alltrim(X2),vList)

help reference:
Character Result = *FOR_EACH( Tag as A, Each as A, List as C )



It was nice to see you at the conference this year. Hope it was as helpful to you as it was to me.
regards.

[Pat Bremkamp]

Re: alltrim in function help

Thanks, Al. Only thing is, my head hurts where I slapped it and yelled "OF COURSE!"

Yes, it was great to meet you at the conference. I learned a lot, and really feel like I made some new friends.

Pat

[Al Buchholz]

Re: alltrim in function help

Pat

I hear you. It was coincidence that I was working on a list processing function at the same time. I'm not going to mention if I did the same thing.....

Reminds me of the old commercial - 'Wow, I could've had a V8!'

[Stan Mathews]

Re: alltrim in function help

Pat

I hear you. It was coincidence that I was working on a list processing function at the same time. I'm not going to mention if I did the same thing.....

Reminds me of the old commercial - 'Wow, I could've had a V8!'

Just for fun....

FUNCTION list_cleanup as c(vList as C)
	list_cleanup=stritran(*remove_duplicates(sortsubstr(remove_blank_lines(*for_each(X,alltrim(X),comma_to_crlf(vList))),crlf())),crlf(),",")
END FUNCTION

At least I think it is equivalent.

[Al Buchholz]

Re: alltrim in function help

Stan

Interesting. I'm wondering of there is a performance difference between those two approaches.

There is a readability issue for me with single line, but....

Maybe Pat will try both of them and let us know....

[MoGrace]

Re: alltrim in function help

Stan,
Would you mind breaking down that function for us? I would like to be able to understand how you came up with it and how it would be used.

[Al Buchholz]

Re: alltrim in function help

Robin

It's the same logic as Pat had in the first post of the thread, just put together on one line.

The only variations are the check for an empty list after the de-blank and the assignment of the list in the *for_each().

It's a nice example of functions within a function. But I find it hard to sight read. I prefer the listing that Pat had for read ability, but if there is a speed improvement, then I'd add the documentation to it so it made sense.

As far as where it should be used.... it may be as simple as taking user input that allows duplicates to find if there are duplicates. It may be an custom import of a file where duplicates are not wanted.

It also reminds me of the request for a linear listing of child records that was asked recently. The function that was put together resulted in a comma seperated list. This would allow a de-duping and de-blanking of that list (if needed).

Yes I'm rambling... it's late here..

Robin - you should have come to the conference, or at least shown up in Vegas. So close is LA, yet so far away... Isn't that a short drive?

Pat is right - meeting people is one of the best parts of the conference. It's something that we can't emulate with all of the technology that we have.

[csda1]

Re: alltrim in function help

Hi Al,

It's a nice example of functions within a function. But I find it hard to sight read.

Yes it is hard to read, but I can tell you without even testing, that the 1 line version is significantly faster. Each assignment line in Alpha 5 has a fairly high overhead, compared to expression evaluation (although it shouldn't really IMHO - this could change in future versions), something discussed in my talk on code speed 2 years ago at the Alpha 5 Conference.

Better yet would be to use WORD_EVALUATE() instead of *for_each(). This is almost identical, except the expression is in quotes and you can specify a delimiter, saving extra processing steps and running it even faster.

Of course, if you had a timing utility, I don't know, maybe the CSDA Code Utility for Alpha 5 (Shameless plug;) ), you could get accurate timing measurements. I'll try to run some tests later today to give you real number differences.

[JerryBrightbill]

Re: alltrim in function help

*for_each() will run much faster than word_evaluate() as word_evaluate() is written in xbasic. Both do essentially the same thing, but *for_each() effectively replaces word_evaluate().

Stan's one line expression is slightly faster, but more difficult to construct and maintain. *for_each() runs slightly faster when run on a crlf() delimited list than it does on a comma delimited.

Here are some example times on a 3060 line list built of very long elements on each line (each line over 100 characters). The list was created with filefind.get_recurse().

(all times in seconds)

Original function: 3.952277
Stan's expression: 3.724890
Word_evaluate: 5.002599

These were repeated with *for_each() run on the crlf() delimited list and the result converted to comma delimited.

vList=comma_to_crlf(vList)
vlist = *for_each(X,alltrim(X),vList)
vList=remove_blank_lines(vList)

vList=sortsubstr(vList,crlf())
vList=*remove_duplicates(vList,crlf())
vList=crlf_to_comma(vList)

Time to run above: 3.760223
Time when converted to single expression: 3.762531

If the elements are much shorter, and the list shorter, the times are much faster. In that case, there is virtually no difference between the multi line code and the single expression, or between running *for_each() on crlf() delimited or comma delimited

Example: 411 lines with the longest line under 50 characters

Original function: .018460
Stan's expression: .018699
Word_evaluate: .166426

*for_each() on crlf() delimited

Time to run multi line: .019079
Time when converted to single expression: .018281

Incidentally, the biggest speed hit comes from converting from comma to crlf() and vice versa and sorting the list. If the original list is crlf() and the output is crlf(), the speed drops to

.012187

[csda1]

Re: alltrim in function help

Hi Jerry,

*for_each() will run much faster than word_evaluate() as word_evaluate() is written in xbasic. Both do essentially the same thing, but *for_each() effectively replaces word_evaluate().

After testing speed of word_evaluate, this becomes obvious.

Stan's one line expression is slightly faster, but more difficult to construct and maintain.

That's why we have comment lines, to explain difficult code. I would seldom trade speed for clarity (but you can show the expanded case in the comments)

*for_each() runs slightly faster when run on a crlf() delimited list than it does on a comma delimited.

Is there a new version of *for_each() that allows specifying the delimiter? I'm not aware of it, although, it would be a nice feature.

After testing, I found most of the time is spent in the *for_each() and sortsubstr() functions (about a half each). The assignments are such a small part of the overall processing, it ends up in the noise unless you have small lists, so I'm not surprised Stan's version did not seem any faster

The fastest expanded code was;

'vList=comma_to_crlf(vList)    ' A little slower than stritran equivalent
vList=stritran(vList,",",chr(13)+chr(10))
vlist = *for_each(X,alltrim(X),vList)
vList=remove_blank_lines(vList)

vList=sortsubstr(vList,crlf())
vList=*remove_duplicates(vList,crlf())
'list_cleanup=crlf_to_comma(vList)    ' Much slower (factor of about 2) than stritran equivalent
list_cleanup=stritran(vList,",",chr(13)+chr(10))

Note the significant difference between

STRITRAN() and CRLF_TO_COMMA()

versus

STRITRAN() and COMMA_TO_CRLF()

And CHR(13)+CHR(10) is faster than CRLF() by a small amount.