【发布时间】：2016-10-21 05:46:54
【问题描述】：
我有一个大约 100,000 行的表格。该表在 Excel 文件中，下面是它的快照：
+------------+-----------+-----+-----+-----------------------------------------------------------+
| First Name | Last Name | Sex | Age | Address |
+------------+-----------+-----+-----+-----------------------------------------------------------+
| Parm | Jit | m | 23 | palm court scoeity, RD. golf course, delhi |
| Param | jit | m | 24 | palm cort society, road golf course, delhi |
| Pram | Jet | m | 28 | palm court socityt Road golf course, Delhi |
| Prm | jit | m | 31 | society palm court, Rod. Golf coure, delhi |
| Param | Jeet | m | 33 | palm court scoety, delhi |
| varun | nagraj | m | 36 | Thame Square, auckland-AZ-2014 |
| Janet | kumar | m | 40 | Thame Square, auckland-AZ-2014 |
| varun | kumar | m | 42 | Thame Square, auckland-AZ-2014 |
| Jatin | Kakkar | m | 45 | Noida, near shipra mall, sectr 57, Noida, U.P. |
| Jatin | Kakar | m | 56 | Noida, near shipra mall, sectr 57, Noida, Uttar pardesh |
| Jatin | Kakkr | m | 57 | Noida, Flat no- 23, near shipra mall, sectr 57, Noida, UP |
| Janet | Yellen | F | 23 | 11 CORONADO POINTELAGUNA NIGUELCA92677 |
| Janet | Yellen | F | 24 | 11 CORONADO POINTELAGUNA NIGUELCA |
| Janet | Yellen | F | 25 | 11 CORONADO POINTELAGUNA 92677-0000 |
| Jant | Yelen | F | 26 | 11 CORONADO POINTELAGUNA NIGUELCA0000 |
| Janet | Yellen | F | 26 | 11 CORONADO POINTELAGUNA NIGUELC |
| Abigail | Johnson | F | 24 | PRESERVE DRIVE NE, 11BELMONTMI4930 |
| andrew | symonds | m | 24 | Fame Stret, brisbane, hn 181 |
| Angel | Ahrendts | F | 26 | WYNGATE MANOR CTALEXANDRIAVA |
| Safra | Catz | F | 26 | 31155 ZOAR SCHOOL ROADLOCUST GROVEVA22508-0000 |
| Park | Geun-hye | F | 30 | CATHOLIC CHURCH RDBEACH LAKEPA |
| Sheryl | Sandberg | F | 24 | 80164 SULTANA AVEINDIOCA92201-0000 |
| Sheryl | Sandberg | F | 24 | SULTANA AVEINDIOC |
| Safra | Catz | F | 26 | OAR SCHOOL ROADLOCUST GROVEV |
| Park | Geun-hye | F | 30 | 308 CATHOLIC CHURCH RDBEACH LAKEPA18405-0000 |
| andrw | simnds | m | 24 | Fame Stret, 181 HOUSE NO |
| prashat | vats | m | 35 | Al thei, al nzar, dubai12 |
| prasant | vats | m | 37 | Al, al nazar, dubai23 |
| andrw | simonds | m | 34 | Fame brisbane, 181 H.N. |
| vats | prashant | m | 30 | Al thei, al nazar, dubai |
| vast | prshant | m | 30 | al nazar, dubai, street adamifullah |
| prashant | vats | m | 37 | Al thei, al nazar, dubai |
| ram | vats | m | 29 | Al thei, nazar, dubai |
| Kiss | hanes | m | 45 | Sydney, andrew str. 223 |
+------------+-----------+-----+-----+-----------------------------------------------------------+
我正在尝试找出此数据中的行相似性，例如，第 1 行与第 2 行非常相似。我尝试过聚类算法（即BIRCH、DBSCAN、K means、Spectral 和Markov Clustering)，但它们都需要大约半小时才能在 100,000 行上运行，然后在 python 中出现内存错误（因为我在我的 python 平台上获取所有数据，所以我的机器有一个16gb)。
我应该使用一些更好的算法来解决这个问题，还是需要将我的数据移动到像spark 这样的平台，然后再处理它？如果是前者，你能帮我用一些不需要太多时间的算法吗？请不要将此视为理论问题，因为我期待找到一种解决大数据实际问题的方法。
【问题讨论】：
-
Excel，只有 100000 条记录——这不是大数据。它是兆字节，而不是艾字节......
标签： python pattern-matching cluster-analysis fuzzy-comparison record-linkage