Regular Expressions

[Graham Wickens]

Hi all,

been trying to extract dates from text fields, the date is not neccessarily in the same place or same format in each record. Below is my trial-and-error attempt.
I want to replace each dot with a forward-slash and ensure that day and month are always two digits and the year is 4 digits.

testData1 = "Engine cut on approach and undershot practice forced landing Lords Bridge Cambs. 13.11.42 DBF"
SearchExpr1 ="([0-9]{2})[.]([0-9]{2})[.]([0-9]{2})"
testsplit1 = regex_split(testData1 ,SearchExpr1)
testresult1 = crlf_to_comma(testsplit1)
trace.writeln("1) "+Testresult1+" = "+padl(word(testresult1,1,","),2,"0")+"/"+padl(word(testResult1,2,","),2,"0")+"/19"+word(testresult1,3,","))

testData2 = "Engine cut on approach and undershot practice forced landing Lords Bridge Cambs. 1.11.42 DBF"
SearchExpr2 = "([0-9])[.]([0-9]{2})[.]([0-9]{2})"
testsplit2 = regex_split(testData2 ,SearchExpr2)
testresult2 = crlf_to_comma(testsplit2)
trace.writeln("2) "+Testresult2 + " = "+padl(word(testresult2,1,","),2,"0")+"/"+padl(word(testResult1,2,","),2,"0")+"/19"+word(testresult2,3,","))

testData3 = "Engine cut on approach and undershot practice forced landing Lords Bridge Cambs. 1.1.42 DBF"
SearchExpr3 = "([0-9])[.]([0-9])[.]([0-9]{2})"
testsplit3 = regex_split(testData3 ,SearchExpr3)
testresult3 = crlf_to_comma(testsplit3)
trace.writeln("3) "+Testresult3 +" = "+padl(word(testresult3,1,","),2,"0")+"/"+padl(word(testResult3,2,","),2,"0")+"/19"+word(testresult3,3,","))

testData4 = "Engine cut on approach and undershot practice forced landing Lords Bridge Cambs. 13.1.42 DBF"
SearchExpr4 = "([0-9]{2})[.]([0-9])[.]([0-9]{2})"
testsplit4 = regex_split(testData4 ,SearchExpr4)
testresult4 = crlf_to_comma(testsplit4)
trace.writeln("4) "+Testresult4 + " = "+padl(word(testresult4,1,","),2,"0")+"/"+padl(word(testResult4,2,","),2,"0")+"/19"+word(testresult4,3,","))

testData5 = "Engine cut on approach and undershot practice forced landing Lords Bridge Cambs. 1.11.42 DBF"
SearchExpr5 = "([0-9])[.]([0-9]{2})[.]([0-9]{2})"
testsplit5 = regex_split(testData5 ,SearchExpr5)
testresult5 = crlf_to_comma(testsplit5)
trace.writeln("1) "+Testresult5+" = "+padl(word(testresult5,1,","),2,"0")+"/"+padl(word(testResult5,2,","),2,"0")+"/19"+word(testresult5,3,","))

testData6 = "Engine cut on approach and undershot practice forced landing Lords Bridge Cambs. .42 DBF"
SearchExpr6 = "[.]([0-9]{2})"
testsplit6 = regex_split(testData6,SearchExpr6)
testresult6 = crlf_to_comma(testsplit6)
trace.writeln("6) "+Testresult6+" = "+"19"+word(testresult6,1,","))

testData7 = "Engine cut on approach and undershot practice forced landing Lords Bridge Cambs. 1.42 DBF"
SearchExpr7 = "([0-9])[.]([0-9]{2})"
testsplit7 = regex_split(testData7 ,SearchExpr7)
testresult7 = crlf_to_comma(testsplit7)
trace.writeln("7) "+Testresult7 +" = "+padl(word(testresult7,1,","),2,"0")+"/19"+word(testresult7,2,","))

testData8 = "Engine cut on approach and undershot practice forced landing Lords Bridge Cambs. 11.42 DBF"
SearchExpr8 = "([0-9]{2})[.]([0-9]{2})"
testsplit8 = regex_split(testData8 ,SearchExpr8)
testresult8 = crlf_to_comma(testsplit8)
trace.writeln("8) "+Testresult8+" = "+padl(word(testresult8,1,","),2,"0")+"/19"+word(testresult8,2,","))

I can do it for each of the 8 formats present in the text field , but how do I combine them into one solution. Going around in cricles so need a helping hand.

is there a more simpler way of doing this using regular expressions, where I won't have to know which format the date is in before transformation . the result will be put into another field in the record. ?


thx

[madtowng]

Hi Graham,
I'm going to swing at the low-hanging fruit first, using testData8 as the example.
I can do more, but I would need to know what basic format the dates are in (m/d/y or ?)

Gregg

testData8 = strtran(testData8,".","/")
testData8 = strtran(testData8,"/ ",". ")

[Graham Wickens]

came up with one solution, although still a bit long..

FUNCTION NewDateFormat AS C (text_data AS C )

Srch_Expr1 = "([0-9]{1})[.]([0-9]{1})[.]([0-9]{2})"  [COLOR=#2980b9]'Search for date format 1.1.42[/COLOR]
Repl_Expr1 = "0\1\/0\2\/19\3"                        [COLOR=#2980b9]'convert to date format 01/01/1942[/COLOR]

Srch_Expr2 = "([0-9]{2})[.]([0-9]{1})[.]([0-9]{2})"  [COLOR=#2980b9]'Search for date format 11.1.42[/COLOR]
Repl_Expr2 = "\1\/0\2\/19\3"                         [COLOR=#2980b9]'convert to date format 11/01/1942[/COLOR]

Srch_Expr3 = "([0-9]{1})[.]([0-9]{2})[.]([0-9]{2})"  [COLOR=#2980b9]'Search for date format 1.11.42[/COLOR]
Repl_Expr3 = "0\1\/\2\/19\3"                         [COLOR=#2980b9]'convert to date format 01/11/1942[/COLOR]

Srch_Expr4 = "([0-9]{2})[.]([0-9]{2})[.]([0-9]{2})"  [COLOR=#2980b9]'Search for date format 11.11.42[/COLOR]
Repl_Expr4 = "\1\/\2\/19\3"[COLOR=#2980b9]                          'convert to date format 11.11.1942[/COLOR]

Srch_Expr5 = "([0-9]{1})[.]([0-9]{2})"   [COLOR=#2980b9]'Search for date format 1.42[/COLOR]
Repl_Expr5 = "0\1\/19\2"                 [COLOR=#2980b9]'convert to date format 01/1942[/COLOR]

Srch_Expr6 = "([0-9]{2})[.]([0-9]{2})"   [COLOR=#2980b9]'Search for date format 11.42[/COLOR]
Repl_Expr6 = "\1\/19\2"                  [COLOR=#2980b9]'convert to date format 11/1942[/COLOR]

Srch_Expr7 = "[.]([0-9]{2})"  [COLOR=#2980b9]'Search for date format .42[/COLOR]
Repl_Expr7 = "19\1"           [COLOR=#2980b9]'convert to date format 1942[/COLOR]

for x = 1 to w_count(Text_Data," ")
Sample_Text = word(Text_Data,x," ")

select

case regex_match(Sample_Text,Srch_Expr1)
        NewDateFormat = regex_merge(Sample_Text,Srch_Expr1,Repl_Expr1)
        goto Finish

case regex_match(Sample_Text,Srch_Expr2)
        NewDateFormat = regex_merge(Sample_Text,Srch_Expr2,Repl_Expr2)
        goto Finish

case regex_match(Sample_Text,Srch_Expr3)
        NewDateFormat = regex_merge(Sample_Text,Srch_Expr3,Repl_Expr3)
        goto Finish

case regex_match(Sample_Text,Srch_Expr4)
        NewDateFormat = regex_merge(Sample_Text,Srch_Expr4,Repl_Expr4)
        goto Finish

case regex_match(Sample_Text,Srch_Expr5)
        NewDateFormat = regex_merge(Sample_Text,Srch_Expr5,Repl_Expr5)
        goto Finish

case regex_match(Sample_Text,Srch_Expr6)
        NewDateFormat = regex_merge(Sample_Text,Srch_Expr6,Repl_Expr6)
        goto Finish

case regex_match(Sample_Text,Srch_Expr7)
        NewDateFormat = regex_merge(Sample_Text,Srch_Expr7,Repl_Expr7)
        goto Finish

end select
next x
Finish:
END FUNCTION

[madtowng]

I could only do basic testing, but I think this shortens the code.

FUNCTION NewDateFormat AS C (text_data AS C )
dim fixit[0] as p

fixit[].Srch_Expr = "([0-9]{1})[.]([0-9]{1})[.]([0-9]{2})" ' find date format 1.1.42
fixit[..].Repl_Expr = "0\1\/0\2\/19\3" 'convert to date format 01/01/1942

fixit[].Srch_Expr = "([0-9]{2})[.]([0-9]{1})[.]([0-9]{2})" ' find date format 11.1.42
fixit[..].Repl_Expr = "\1\/0\2\/19\3" 'convert to date format 11/01/1942

fixit[].Srch_Expr = "([0-9]{1})[.]([0-9]{2})[.]([0-9]{2})" ' find date format 1.11.42
fixit[..].Repl_Expr = "0\1\/\2\/19\3" 'convert to date format 01/11/1942

fixit[].Srch_Expr = "([0-9]{2})[.]([0-9]{2})[.]([0-9]{2})" 'find date format 11.11.42
fixit[..].Repl_Expr = "\1\/\2\/19\3" 'convert to date format 11.11.1942

fixit[].Srch_Expr = "([0-9]{1})[.]([0-9]{2})" ' find date format 1.42
fixit[..].Repl_Expr = "0\1\/19\2" 'convert to date format 01/1942

fixit[].Srch_Expr = "([0-9]{2})[.]([0-9]{2})" 'find date format 11.42
fixit[..].Repl_Expr = "\1\/19\2" 'convert to date format 11/1942

fixit[].Srch_Expr = "[.]([0-9]{2})" 'find date format .42
fixit[..].Repl_Expr = "19\1" 'convert to date format 1942

for x = 1 to w_count(Text_Data," ")
Sample_Text = word(Text_Data,x," ")

for each option in fixit
select
case regex_match(Sample_Text,option.Srch_Expr)
NewDateFormat = regex_merge(Sample_Text,option.Srch_Expr,option.Repl_Expr)
end select

next 'each option
next x

END FUNCTION

[Graham Wickens]

Thanks Gregg, got there in the end.

[madtowng]

Glad to hear. Now for the kicker, if you really want to shorten you can bury the Srch_Expr and Repl_Expr statements in a sql table,
then populate the array with a relatively simple query, although you might have to make accommodations for and \s.

[armasoft]

I'm a bit late to the party on this one, but have coded a single line solution that matches all the test options...

?if(val(word(testdata1,-4))>0,padl(word(testdata1,-4),2,"0")+"/","")+if(val(word(testdata1,-3))>0,padl(word(testdata1,-3),2,"0")+"/","")+"19"+word(testdata1,-2)

This works if the date value(s) is always followed by a single word and not preceded by a number when day of month is not specified.

[Graham Wickens]

Glad to hear. Now for the kicker, if you really want to shorten you can bury the Srch_Expr and Repl_Expr statements in a sql table,
then populate the array with a relatively simple query, although you might have to make accommodations for and \s.

gulp! how do I do that !!!!!!!!

[Graham Wickens]

I'm a bit late to the party on this one, but have coded a single line solution that matches all the test options...

?if(val(word(testdata1,-4))>0,padl(word(testdata1,-4),2,"0")+"/","")+if(val(word(testdata1,-3))>0,padl(word(testdata1,-3),2,"0")+"/","")+"19"+word(testdata1,-2)

This works if the date value(s) is always followed by a single word and not preceded by a number when day of month is not specified.

Interesting, but unfortunately wont work in my scenario as the date can be anywhere in the field, beginning middle or end!

[armasoft]

Is it possible there could be other numeric data in the record?
If so, will the date always be the last numeric value in the record?

[Ted Giles]

Suggest you get rid of the text and work with the numbers only.
Graham said it could be anywhere.

[madtowng]

Hi Graham,

The code for this assumes you have a MySQL table named regexpressions setup as shown at http://www.sqlfiddle.com/#!9/717182c/1 , and a named::connection of AAlab.
You can change those but obviously will need to adjust the code if you do.
Let me know if (or how) this works for you.

Gregg

FUNCTION NewDateFormat AS C (text_data AS C )
dim myCN as sql::connection
dim sqlCode as c
dim itsOpen as l
dim itWorked as l
dim fixit[0] as p

sqlCode = <<%txt%
select * from regexpressions
%txt%

if .not. myCN.isopen then
itsOpen = myCN.open("::name::AAlab")
end if
if .not. itsOpen then
NewDateFormat = "Could not Open Table"
exit function
end if

itWorked = myCN.ToPropertyArray(sqlCode, fixit)

for x = 1 to w_count(Text_Data," ")
Sample_Text = word(Text_Data,x," ")

for each option in fixit
select
case regex_match(Sample_Text,option.Srch_Expr)
NewDateFormat = regex_merge(Sample_Text,option.Srch_Expr,option.Repl_Expr)
exit function
end select

next 'each option
next x

END FUNCTION

[armasoft]

Suggest you get rid of the text and work with the numbers only.
Graham said it could be anywhere.

I think the solution wioll be quite easy if there's no other numerics, but not going to spend time on it until I know

[Ted Giles]

Agree. My approach would be to strip the chars and count the numbers left.
If there were more than 8, you would have to find a pattern.
Nn.nn.nnnn would be fine (10.01.1942) so would the other mixes.
Nnn would not (500) - as in 500 feet.

We'll see.