TL;DR
乍一看几乎不可能，除非您可以访问一些包含非常复杂组件的 API。
长期
乍一看，您似乎是在要求神奇地解决自然语言问题。但是，让我们将其分解并将其范围限定为可以构建的点。
首先，要识别国家和城市，您需要枚举它们的数据，所以让我们试试：https://www.google.com/search?q=list+of+countries+and+cities+in+the+world+json
在搜索结果的顶部，我们找到了指向 world-cities.json 文件的https://datahub.io/core/world-cities。现在我们将它们加载到一组国家和城市中。
import requests
import json
cities_url = "https://pkgstore.datahub.io/core/world-cities/world-cities_json/data/5b3dd46ad10990bca47b04b4739a02ba/world-cities_json.json"
cities_json = json.loads(requests.get(cities_url).content.decode('utf8'))
countries = set([city['country'] for city in cities_json])
cities = set([city['name'] for city in cities_json])
现在给定数据，让我们尝试构建 组件 ONE：
让我们把它们放在一起。
import requests
import json
from flashtext import KeywordProcessor
cities_url = "https://pkgstore.datahub.io/core/world-cities/world-cities_json/data/5b3dd46ad10990bca47b04b4739a02ba/world-cities_json.json"
cities_json = json.loads(requests.get(cities_url).content.decode('utf8'))
countries = set([city['country'] for city in cities_json])
cities = set([city['name'] for city in cities_json])
keyword_processor = KeywordProcessor(case_sensitive=False)
keyword_processor.add_keywords_from_list(sorted(countries))
keyword_processor.add_keywords_from_list(sorted(cities))
texts = ['new york to venice, italy for usd271',
'return flights from brussels to bangkok with etihad from €407',
'from los angeles to guadalajara, mexico for usd191',
'fly to australia new zealand from paris from €422 return including 2 checked bags']
keyword_processor.extract_keywords(texts[0])
[出]：
['York', 'Venice', 'Italy']
嘿，怎么了？！
做尽职调查，第一个预感是“纽约”不在数据中，
>>> "New York" in cities
False
什么？！ #$%^&* 为了理智，我们检查这些：
>>> len(countries)
244
>>> len(cities)
21940
是的，你不能只信任一个数据源，所以让我们尝试获取所有数据源。
从https://www.google.com/search?q=list+of+countries+and+cities+in+the+world+json，你可以找到另一个链接https://github.com/dr5hn/countries-states-cities-database 让我们把这个...
import requests
import json
cities_url = "https://pkgstore.datahub.io/core/world-cities/world-cities_json/data/5b3dd46ad10990bca47b04b4739a02ba/world-cities_json.json"
cities1_json = json.loads(requests.get(cities_url).content.decode('utf8'))
countries1 = set([city['country'] for city in cities1_json])
cities1 = set([city['name'] for city in cities1_json])
dr5hn_cities_url = "https://raw.githubusercontent.com/dr5hn/countries-states-cities-database/master/cities.json"
dr5hn_countries_url = "https://raw.githubusercontent.com/dr5hn/countries-states-cities-database/master/countries.json"
cities2_json = json.loads(requests.get(dr5hn_cities_url).content.decode('utf8'))
countries2_json = json.loads(requests.get(dr5hn_countries_url).content.decode('utf8'))
countries2 = set([c['name'] for c in countries2_json])
cities2 = set([c['name'] for c in cities2_json])
countries = countries2.union(countries1)
cities = cities2.union(cities1)
现在我们神经质了，我们要做理智检查。
>>> len(countries)
282
>>> len(cities)
127793
哇，这比以前多了很多城市。
让我们再次尝试flashtext 代码。
from flashtext import KeywordProcessor
keyword_processor = KeywordProcessor(case_sensitive=False)
keyword_processor.add_keywords_from_list(sorted(countries))
keyword_processor.add_keywords_from_list(sorted(cities))
texts = ['new york to venice, italy for usd271',
'return flights from brussels to bangkok with etihad from €407',
'from los angeles to guadalajara, mexico for usd191',
'fly to australia new zealand from paris from €422 return including 2 checked bags']
keyword_processor.extract_keywords(texts[0])
[出]：
['York', 'Venice', 'Italy']
说真的？！没有纽约？！ $%^&*
好的，为了更全面的检查，让我们在城市列表中查找“约克”。
>>> [c for c in cities if 'york' in c.lower()]
['Yorklyn',
'West York',
'West New York',
'Yorktown Heights',
'East Riding of Yorkshire',
'Yorke Peninsula',
'Yorke Hill',
'Yorktown',
'Jefferson Valley-Yorktown',
'New York Mills',
'City of York',
'Yorkville',
'Yorkton',
'New York County',
'East York',
'East New York',
'York Castle',
'York County',
'Yorketown',
'New York City',
'York Beach',
'Yorkshire',
'North Yorkshire',
'Yorkeys Knob',
'York',
'York Town',
'York Harbor',
'North York']
尤里卡！这是因为它叫“纽约市”而不是“纽约”！
你：这是什么恶作剧？！
语言学家：欢迎来到自然语言处理的世界，自然语言是一种社会结构，受社区和方言变体的影响。
你：废话，告诉我如何解决这个问题。
NLP 实践者（一个真正的可以处理嘈杂的用户生成文本的人）：您只需添加到列表中。但在此之前，请根据您已有的列表检查您的 metric。
对于样本“测试集”中的每个文本，您应该提供一些真实标签以确保您可以“衡量您的指标”。
from itertools import zip_longest
from flashtext import KeywordProcessor
keyword_processor = KeywordProcessor(case_sensitive=False)
keyword_processor.add_keywords_from_list(sorted(countries))
keyword_processor.add_keywords_from_list(sorted(cities))
texts_labels = [('new york to venice, italy for usd271', ('New York', 'Venice', 'Italy')),
('return flights from brussels to bangkok with etihad from €407', ('Brussels', 'Bangkok')),
('from los angeles to guadalajara, mexico for usd191', ('Los Angeles', 'Guadalajara')),
('fly to australia new zealand from paris from €422 return including 2 checked bags', ('Australia', 'New Zealand', 'Paris'))]
# No. of correctly extracted terms.
true_positives = 0
false_positives = 0
total_truth = 0
for text, label in texts_labels:
extracted = keyword_processor.extract_keywords(text)
# We're making some assumptions here that the order of
# extracted and the truth must be the same.
true_positives += sum(1 for e, l in zip_longest(extracted, label) if e == l)
false_positives += sum(1 for e, l in zip_longest(extracted, label) if e != l)
total_truth += len(label)
# Just visualization candies.
print(text)
print(extracted)
print(label)
print()
实际上，它看起来并没有那么糟糕。我们得到了 90% 的准确率：
>>> true_positives / total_truth
0.9
但我 %^&*(-ing 想要 100% 提取！！
好吧，好吧，看看上述方法所犯的“唯一”错误，只是“纽约”不在城市列表中。
你：我们为什么不把“纽约”加到城市列表中，即
keyword_processor.add_keyword('New York')
print(texts[0])
print(keyword_processor.extract_keywords(texts[0]))
[出]：
['New York', 'Venice', 'Italy']
你：看，我做到了！！！现在我应该喝啤酒。
语言学家：'I live in Marawi' 怎么样？
>>> keyword_processor.extract_keywords('I live in Marawi')
[]
NLP 实践者（插话）：'I live in Jeju' 怎么样？
>>> keyword_processor.extract_keywords('I live in Jeju')
[]
Raymond Hettinger 的粉丝（远方）：“一定有更好的方法！”
是的，如果我们只是尝试一些愚蠢的事情，比如在我们的keyword_processor 中添加以“City”结尾的城市关键字，那会怎样？
for c in cities:
if 'city' in c.lower() and c.endswith('City') and c[:-5] not in cities:
if c[:-5].strip():
keyword_processor.add_keyword(c[:-5])
print(c[:-5])
有效！
现在让我们重试我们的回归测试示例：
from itertools import zip_longest
from flashtext import KeywordProcessor
keyword_processor = KeywordProcessor(case_sensitive=False)
keyword_processor.add_keywords_from_list(sorted(countries))
keyword_processor.add_keywords_from_list(sorted(cities))
for c in cities:
if 'city' in c.lower() and c.endswith('City') and c[:-5] not in cities:
if c[:-5].strip():
keyword_processor.add_keyword(c[:-5])
texts_labels = [('new york to venice, italy for usd271', ('New York', 'Venice', 'Italy')),
('return flights from brussels to bangkok with etihad from €407', ('Brussels', 'Bangkok')),
('from los angeles to guadalajara, mexico for usd191', ('Los Angeles', 'Guadalajara')),
('fly to australia new zealand from paris from €422 return including 2 checked bags', ('Australia', 'New Zealand', 'Paris')),
('I live in Florida', ('Florida')),
('I live in Marawi', ('Marawi')),
('I live in jeju', ('Jeju'))]
# No. of correctly extracted terms.
true_positives = 0
false_positives = 0
total_truth = 0
for text, label in texts_labels:
extracted = keyword_processor.extract_keywords(text)
# We're making some assumptions here that the order of
# extracted and the truth must be the same.
true_positives += sum(1 for e, l in zip_longest(extracted, label) if e == l)
false_positives += sum(1 for e, l in zip_longest(extracted, label) if e != l)
total_truth += len(label)
# Just visualization candies.
print(text)
print(extracted)
print(label)
print()
[出]：
new york to venice, italy for usd271
['New York', 'Venice', 'Italy']
('New York', 'Venice', 'Italy')
return flights from brussels to bangkok with etihad from €407
['Brussels', 'Bangkok']
('Brussels', 'Bangkok')
from los angeles to guadalajara, mexico for usd191
['Los Angeles', 'Guadalajara', 'Mexico']
('Los Angeles', 'Guadalajara')
fly to australia new zealand from paris from €422 return including 2 checked bags
['Australia', 'New Zealand', 'Paris']
('Australia', 'New Zealand', 'Paris')
I live in Florida
['Florida']
Florida
I live in Marawi
['Marawi']
Marawi
I live in jeju
['Jeju']
Jeju
100% 是的，NLP-bunga !!!
但是说真的，这只是问题的一小部分。如果你有这样的句子会发生什么：
>>> keyword_processor.extract_keywords('Adam flew to Bangkok from Singapore and then to China')
['Adam', 'Bangkok', 'Singapore', 'China']
为什么Adam 被提取为城市？！
然后你做一些更神经质的检查：
>>> 'Adam' in cities
Adam
恭喜你跳进了另一个NLP多义词的兔子洞，同一个词有不同的含义，在这种情况下，Adam很可能在句子中指代一个人，但也巧合的是一个城市的名字（根据您从中提取的数据）。
我明白你在那里做了什么......即使我们忽略了这个多义的废话，你仍然没有给我想要的输出：
[输入]：
['new york to venice, italy for usd271',
'return flights from brussels to bangkok with etihad from €407',
'from los angeles to guadalajara, mexico for usd191',
'fly to australia new zealand from paris from €422 return including 2 checked bags'
]
[出]：
Origin: New York, USA; Destination: Venice, Italy
Origin: Brussels, BEL; Destination: Bangkok, Thailand
Origin: Los Angeles, USA; Destination: Guadalajara, Mexico
Origin: Paris, France; Destination: Australia / New Zealand (this is a complicated case given two countries)
语言学家：即使假设城市前面的介词（例如from，to）给你“起源”/“目的地”标签，你将如何处理“多段”航班的情况，例如
>>> keyword_processor.extract_keywords('Adam flew to Bangkok from Singapore and then to China')
这句话的期望输出是什么：
> Adam flew to Bangkok from Singapore and then to China
也许是这样的？规格是什么？您的输入文本如何（非）结构化？
> Origin: Singapore
> Departure: Bangkok
> Departure: China
让我们尝试构建组件 TWO 来检测介词。
让我们假设您的假设并尝试对相同的flashtext 方法进行一些黑客攻击。
如果我们将to 和from 添加到列表中会怎样？
from itertools import zip_longest
from flashtext import KeywordProcessor
keyword_processor = KeywordProcessor(case_sensitive=False)
keyword_processor.add_keywords_from_list(sorted(countries))
keyword_processor.add_keywords_from_list(sorted(cities))
for c in cities:
if 'city' in c.lower() and c.endswith('City') and c[:-5] not in cities:
if c[:-5].strip():
keyword_processor.add_keyword(c[:-5])
keyword_processor.add_keyword('to')
keyword_processor.add_keyword('from')
texts = ['new york to venice, italy for usd271',
'return flights from brussels to bangkok with etihad from €407',
'from los angeles to guadalajara, mexico for usd191',
'fly to australia new zealand from paris from €422 return including 2 checked bags']
for text in texts:
extracted = keyword_processor.extract_keywords(text)
print(text)
print(extracted)
print()
[出]：
new york to venice, italy for usd271
['New York', 'to', 'Venice', 'Italy']
return flights from brussels to bangkok with etihad from €407
['from', 'Brussels', 'to', 'Bangkok', 'from']
from los angeles to guadalajara, mexico for usd191
['from', 'Los Angeles', 'to', 'Guadalajara', 'Mexico']
fly to australia new zealand from paris from €422 return including 2 checked bags
['to', 'Australia', 'New Zealand', 'from', 'Paris', 'from']
嘿，使用 to/from 的规则非常糟糕，
- 如果“发件人”指的是票价怎么办？
- 如果国家/城市前面没有“to/from”怎么办？
好的，让我们处理上面的输出，看看我们对问题 1 做了什么。也许检查 from 后面的词是否是 city，如果不是，删除 to/from？ p>
from itertools import zip_longest
from flashtext import KeywordProcessor
keyword_processor = KeywordProcessor(case_sensitive=False)
keyword_processor.add_keywords_from_list(sorted(countries))
keyword_processor.add_keywords_from_list(sorted(cities))
for c in cities:
if 'city' in c.lower() and c.endswith('City') and c[:-5] not in cities:
if c[:-5].strip():
keyword_processor.add_keyword(c[:-5])
keyword_processor.add_keyword('to')
keyword_processor.add_keyword('from')
texts = ['new york to venice, italy for usd271',
'return flights from brussels to bangkok with etihad from €407',
'from los angeles to guadalajara, mexico for usd191',
'fly to australia new zealand from paris from €422 return including 2 checked bags']
for text in texts:
extracted = keyword_processor.extract_keywords(text)
print(text)
new_extracted = []
extracted_next = extracted[1:]
for e_i, e_iplus1 in zip_longest(extracted, extracted_next):
if e_i == 'from' and e_iplus1 not in cities and e_iplus1 not in countries:
print(e_i, e_iplus1)
continue
elif e_i == 'from' and e_iplus1 == None: # last word in the list.
continue
else:
new_extracted.append(e_i)
print(new_extracted)
print()
这似乎可以解决问题并删除不在城市/国家之前的from。
[出]：
new york to venice, italy for usd271
['New York', 'to', 'Venice', 'Italy']
return flights from brussels to bangkok with etihad from €407
from None
['from', 'Brussels', 'to', 'Bangkok']
from los angeles to guadalajara, mexico for usd191
['from', 'Los Angeles', 'to', 'Guadalajara', 'Mexico']
fly to australia new zealand from paris from €422 return including 2 checked bags
from None
['to', 'Australia', 'New Zealand', 'from', 'Paris']
但“来自纽约”的问题仍然没有解决！！
语言学家：仔细想想，是否应该通过做出明智的决定使歧义词变得明显来解决歧义？如果是这样，知情决定中的“信息”是什么？是否应该先按照一定的模板来检测信息，然后再填写歧义？
你：我对你失去耐心了……你把我带得兜兜转转，我从新闻和谷歌中不断听到的能理解人类语言的人工智能在哪里？还有 Facebook 等等？！
你：你给我的东西是基于规则的，人工智能在哪里？
NLP 实践者：你不是想要 100% 吗？在没有任何可用于“训练 AI”的预设数据集的情况下，编写“业务逻辑”或基于规则的系统将是真正实现“100%”的唯一方法。
你：你说的训练人工智能是什么意思？为什么我不能只使用谷歌、Facebook、亚马逊或微软，甚至 IBM 的人工智能？
NLP 实践者：让我给你介绍一下
欢迎来到计算语言学和 NLP 的世界！
简而言之
是的，没有真正现成的神奇解决方案，如果您想使用“AI”或机器学习算法，您很可能需要更多的训练数据，例如上面示例中显示的 texts_labels 对。