解析不同的日期格式：正则表达式答案

【问题标题】：Parse Different Date formats: Regex解析不同的日期格式：正则表达式
【发布时间】：2018-09-28 00:26:41
【问题描述】：

用细节重新发布这个问题（因为最后一个被标记了）。

我正在从档案卡中解析混乱的 (tessearct-ocr) 以获取至少 50% 的信息 (date1)。根据下面的数据示例，数据行包含不同形式的日期。

Raw_Text
1   "15957-8 . 3n v g - vw, 1 ekresta . bowker, william e tley n0 .qu v- l. c. 
    s. peteris, forestville, n. y. .mafae date1 june 17,1942 by davis, c. j6 
    l. g. b. jonnis, buffalo, n. y. ngsted decl 17, 1949.3y 7 davis, c. j. 
    date3 by j date4 - by date5 by 6 -.5/, 7/19/l date6 17 jul 1916 salamanca. 
    hf date7 31 dec 1986 buffalo, new york "
2   ".1o2o83n5ddn.. -i ekresta i bowles, albert edwin i made date1 june 9p1909 
    by parker, elm. date2 dec . 18 w date3 . by dep osed by date5 by date7mqm 
    9 ivvld wm 4144, mac, .75 076 eaqlwli "
3   "i naime bowles, charles edward made date1 may 31. 1892 by mclaren, wneoi 
    date2 may 18. 1895 by mclaren, w.e. date3 . i by date4 may 10. 1908 by 
    bip. of chicago. date5 by date7 "
4   "101 557 am l i ekrestaibowles, donald manson ..46 ohio trlnlty cathedral, 
    cleveland, ohio made date1 6/19/76 by burt, ji. h. grace , cleveland, ohio 
   date2 11 jun 77 by bp j h burt date3 . 1 .. by date4 by date5 bv m cuyahoga 
   heights, ohio date6 4/29/27 date7 240000 "
5   "227354 101 575 m68, frederick augustus st. paujjs cathedral, buffalo, 
   n.y. made date1 6/15/63 by scaife. l.i... st. thomas. modia, bath, n.y. 
   date2 1/11/611 by scaife. l.eo date3 by date4 by date5 by bradford, n.y. i 
   . 130m 6/1/18 date7 17 jun 1996 foratvme new york z4uc-xl "
6   "1 95812d ll. il ekresta bowles, harry oscar lmade date14 july 17, 190433, 
    lepnard, w.a. date2 july 25 , 1905 by leonard, w.a. i date3 by date4 by 
   date5 by g- m. /(,,/mr date7 jay /z/,. /357i l /mwi yk/maj. "
7   "5025 ,.. 2.57631 il . - . .. .1 i ekresta bowles , jedwiah hibbafd made 
    deac0n 8., i5-0i1862i13y potter, iih. date2 10. 280 1864 1 biy stevens, w. 
    b. date3 by date4 7 .30 l 1875 by date5 by date7 "
8   "30.611126 ekhq il ekresta bowles, ralph hart made date1 12. 210 i1883 by 
    iwiiiliams, i36 date2 7.. 1. 1885 by williams , j. date3 by i date4 by 
    date5 by g .97) l/am 9- date7 10. 4. 1900 (78) if x/ma 3.4, 154.47.11.73. 
    4,... mya-ix "
9   "2.25678 . 1o14593 ekresta bowles, robert brigham, jr. st. matthew s 
    cathedra1,da11quexas made date1 6/18/65 by mason, c. a. 57 mmzws camp 
    dr7///9s tams date2 12 21 cs by 14.45.42 c a date3 i by date4 by date5 , 
    by houston, texas date6 4/11/30 date7 12 dec 2000 dallas texas 2400-xi "
10  "101 619 34hq woe ekresta bowlin1 howard bruce cathedral modia of saint 
    peter 61 st. paul, washin ton, dc made date1 13 jun 92 bybp r h haines 
   (wdc st. alban1s modia, annandale, vir inia . pdumd 16 jan 93 by r h halnes 
    (wdc) date3 by atas by date4 v by date5 by date6 31 aug 1946 e st. louis. 
   il date7 2400-i "
11  "w k8 8km tm boiling jack dnnmwm q- f grace ch , made dat j 11201). salem 
    mares. stverrett. f. ,w a x st. johms modia. memphis, tenh. date1 apr. 25. 
    1955 - bv barth, t.in.. date3 4 by date4 by date5 by date7 wq iw r 1 w .n 
    . 4.1- 1 date6z1l7i1c. "

我通过两步过程解析 date1， - 1. 解析名称“date1”和“by”之间的文本 - 2.使用日期解析器提取实际日期

import re
import dateutil.parser as dparser
for lines in Raw_Text:
    lines = lines.lower() #make lower case
    lines = lines.strip() #remove leading and ending spaces
    lines = " ".join(lines.split()) #remove duplicated spaces



    # Step 1
    #Extract data between "date1" and "by"
    deacondt = re.findall(r'date1(.*?)by',lines)

    deacondt = ''.join(deacondt)  #Convert list to a string


    # Step 2
    # use dateutil to parse dates in extracted data

    try:
        deacondt1 = dparser.parse(deacondt)
    except:
        deacondt1 = 'NA'

    print deacondt1

第 1 步的输出是，

[' june 17,1942 ']
[' june 9p1909 ']
[' may 31. 1892 ']
[' 6/19/76 ']
[' 6/15/63 ']
['4 july 17, 190433, lepnard, w.a. date2 july 25 , 1905 ']
[]
[' 12. 210 i1883 ']
[' 6/18/65 ']
[' 13 jun 92 ']
[]

而第 2 步返回以下输出

2018-06-17 00:00:00
1909-06-17 21:00:00
1892-05-31 00:00:00
1976-06-19 00:00:00
2063-06-15 00:00:00
NA
NA
NA
2065-06-18 00:00:00
1992-06-13 00:00:00
NA

第 2 步未能提供所有日期。有没有比“dateutil.parser”更好的 Python 2.7 日期解析器？

【问题讨论】：

标签： python regex python-2.7 date date-parsing

【解决方案1】：

您的许多日期都有不同的格式：这会让事情变得困难。

您可以使用datetime 库来解析日期。由于您的数据有多种格式，因此您将需要几种不同的格式字符串。

datetime有两个有用的函数：datetime.strptime（字符串解析时间，返回datetime.datetime）和datetime.strftime（字符串FROM时间，返回str）

如果你有足够的格式字符串，下面是一个如何解析的例子：

import datetime

for lines in Raw_Text:

    ## Do the regex stuff above.
    ## Keep each returned result as a separate string.
    regex_results = get_your_regex_results()


    # Step 2
    # use dateutil to parse dates in extracted data

    date_formats = [ ## You will need several formats to try.
        '%m/%d/%Y',
    ] 

    for datestring in regex_results:

        for fmt in date_formats:
            try:
                date_str = date_str.strip()
                deacondt1 = datetime.datetime.strptime(date_str, fmt)
                print(deacondt1)
                break
            except ValueError:
                continue

https://docs.python.org/2/library/datetime.html#strftime-strptime-behavior

【讨论】：

【解决方案2】：

你可以试试这个，

deacondt1 = dparser.parse(deacondt, dayfirst=False, fuzzy=True)

fuzzy - 允许包含 un-dateformat 字样的字符串，例如“Today is January 1, 2047 at 8:21:00AM”。
dayfirst=False 表示 month-first date-format 输入字符串喜欢你的。

但dateutil-parser 不足以提取您想要的输出，因此需要将更接近日期格式的字符串传递给解析器。

Regex 提取关于date1的字符串

(?s)date1\d?((?:(?!by|date2|date3).)*)

Demo,,, 其中不仅'by' 'date2' 和 'date3' 被视为separator 和date10~date19 被视为@987654338 @。

然后，对提取的字符串进行操作（删除前导和尾随空格等），以获取 date-util 解析器的可接受输入。

regx= re.compile(r'(?s)date1\d?((?:(?!by|date2|date3).)*)')
raw_date= [re.sub(r'(?i)(?<=\s)[a-z]?(\d{4}|\d{2})\d*', r'\1', re.sub(r'\s+|,|(?<=\d)[^\d\s\/](?=\d)',' ', re.sub(r'^\s+|\s+$|\n+','', m))) for m in regx.findall(Raw_Text)]

for deacondt in raw_date: 
    try:
        deacondt1 = dparser.parse(deacondt, dayfirst=False, fuzzy=True)
    except:
        deacondt1 = 'NA'

print(deacondt +"\n"+ str(deacondt1))

输出

june 17 1942
1942-06-17 00:00:00
june 9 1909
1909-06-09 00:00:00
may 31. 1892
1892-05-31 00:00:00
6/19/76
1976-06-19 00:00:00
6/15/63
2063-06-15 00:00:00
july 17  1904  lepnard  w.a.
1904-07-17 00:00:00
12. 21 1883
1883-12-21 00:00:00
6/18/65
2065-06-18 00:00:00
13 jun 92
1992-06-13 00:00:00
apr. 25. 1955 - bv barth  t.in..
1955-04-25 00:00:00

【讨论】：

【解决方案3】：

没有解析模块可以为您可能遇到的每个 OCR 曲线提供完整的解决方案。
您必须建立一些评估/纠正框架来发现和修复您可以修复的内容。

我建议以下工作流程：

尝试解析日期序列。
将未解析的序列保存到特殊文件中
编辑文件，添加一些正则表达式替换规则，将序列重写为可挽救的形式。
应用文件中的规则并尝试再次解析
从 2 开始重复，直到处理完所有内容。

下面是一些示例代码：

parser.py

import re
import csv
import glob, os
from datetime import datetime
import dateutil.parser as dparser

def load_patterns():
    ''' load patterns from existing pat_*.csv 
        return a dict of the form { sequence: [sequence,pattern,replace] }
        sequence is an example of the string that should be handled by this pattern
        pattern and replace have the same meaning as for re.sub
    '''
    patterns = {}
    for pattern_file in glob.glob("pat_*.csv"):
        with open(pattern_file, 'r') as fh:
            reader = csv.DictReader(fh, delimiter=',', quotechar='"', skipinitialspace=True)
            reader.fieldnames=[f.strip() for f in reader.fieldnames]
            for row in reader:
                # skipping empty patterns if there was non-empty one for this sequence
                if row['sequence'] in patterns and  not row['pattern']:
                    continue
                patterns[row['sequence']]=(row['pattern'],row['replace'])
    return patterns

def save_nonmatched(patterns, nonmatched):
    ''' saves a new pattern file with the empty pattern field
        supposed to be edited manually afterwards
    '''
    items_to_save = [ key for key in nonmatched if key not in patterns ]
    if not items_to_save:
        return

    new_file=datetime.now().strftime('pat_%Y%m%d_%H%M%S.csv')
    with open(new_file, 'w', newline='') as fh:
        writer = csv.DictWriter(fh, fieldnames=['sequence', 'pattern', 'replace'], quoting=csv.QUOTE_ALL)
        writer.writeheader()
        for key in items_to_save:
            writer.writerow({'sequence':key, 'pattern':'', 'replace':''})

def sub_with_patterns(s, patterns):
    ''' try to match each pattern in patterns iterable
        return expanded string if match succeeded
    '''
    debug=1
    for sequence, (pattern, replace) in patterns.items():
        if not pattern:
            continue
        match=re.search(pattern,s,re.X)
        if match:
            return match.expand(replace)
    return None


nomatch={}
patterns = load_patterns()
Raw_Text = re.sub(r'\s+', ' ' ,open('in.txt','r').read().lower()).strip()

for dt in re.findall(r'date1(.*?)by', Raw_Text, re.S):
    corrected = sub_with_patterns(dt, patterns)
    try:
        parsed = dparser.parse(corrected or dt)
        print ("input:%s parsed:%s" % (dt,parsed))
    except:
        nomatch[dt]=1
        print ("input:%s ** not parsed" % (dt))            

save_nonmatched(patterns, nomatch)

现在，如果在您的输入上尝试脚本，我们会得到第一个更正 csv：

"sequence","pattern","replace"
"4 july 17, 190433, lepnard, w.a. date2 july 25 , 1905 ","",""
" 12. 210 i1883 ","",""
" apr. 25. 1955 - bv barth, t.in.. date3 4 ","",""

和输出：

input: june 17,1942  parsed:2018-06-17 00:00:00
...
input:4 july 17, 190433, lepnard, w.a. date2 july 25 , 1905  ** not parsed
...

我们像下面这样编辑文件：

"sequence","pattern","replace"                                                    
"4 july 17, 190433, lepnard, w.a. date2 july 25 , 1905 ","^
     \s*(?P<day>\d+)
     \s+(?P<month>(?:jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)[a-z]*)
     \s+(?P<year>\d{2})
    ","\g<day> \g<month> 19\g<year>"
" 12. 210 i1883 ","",""
" apr. 25. 1955 - bv barth, t.in.. date3 4 ","",""

然后再次运行解析器：

input: june 17,1942  parsed:2018-06-17 00:00:00
...
input:4 july 17, 190433, lepnard, w.a. date2 july 25 , 1905  parsed:1917-07-04 00:00:00
...

当然，这远远不能解决您将遇到的所有 OCR 解析问题，但这可能是一个好的开始。

【讨论】：