【发布时间】：2016-06-21 15:53:09
【问题描述】：
我正在使用名为 commonsearch 的系统。特别是在这篇文章中，我们将讨论它的后端部分，witch 是用 python 编写的。
后端系统将 warc 文件和索引的内容流式传输到 2 个弹性搜索集群 - 1) 文本弹性搜索集群 2) 文档弹性搜索集群。
在添加我的更改之前，索引的平均速度约为每个索引 0.02。
经过我的更改后，它约为 1.00（在 aws 上为 0.4）。
那么我做了什么。
我为每个使用html2text 的warc 主体剥离了html，但这并不需要太多时间（可能是+0.02），但它确实使性能更加尖锐，内容更多，需要更多时间来剥离html.
我为每个索引值添加了 2 个textblob 文本分类器（naiveBays）检查，它的训练被序列化（pickle）并在循环之前加载。
第一次训练包含 33'000 个测试数据，第二次包含数百个（i 将被添加到第二个更多）。
性能分析
每个 10 个示例。
在我的更改之前：
Indexing http://2sao.vn/p1004c1007n20110413113841718/mau-vay-du-tiec-cho-quy-co-hoan-hao.vnn [64/1817]
--- 0.0224668979645 seconds ---
Indexing http://2sidesoftheocean.blogspot.com/2012/04/my-first-family-in-1940-us-census_02.html
--- 0.0367019176483 seconds ---
Indexing http://3.pulsitemeter.com/exbii/exbii-photos-aunties-bath-.html
--- 0.00342702865601 seconds ---
Indexing http://303cycling.com/Meredith-Miller-USGP-Cyclocross-Video-Specialized-bikes
--- 0.0187289714813 seconds ---
Indexing http://303magazine.com/2012/10/undead-mans-party-casselmans-hosts-zombie-crawl-aftermath-featuring-celldweller/
--- 0.0460560321808 seconds ---
Indexing http://38-avg.blogspot.com/2008/05/birdheart.html
--- 0.0178949832916 seconds ---
Indexing http://3docean.net/item/motorola-droid-razr-low-poly-/3712487?sso
--- 0.0468878746033 seconds ---
Indexing http://4.bp.blogspot.com/_hZs38tqNXns/StdbQyR_zGI/AAAAAAAAEyw/VvNCalngDbY/s1600-h/Vanderwood
--- 0.00142908096313 seconds ---
Indexing http://411mania.com/sports/young-firpo-the-best-light-heavywieght-to-never-win-a-title/
--- 0.0295450687408 seconds ---
添加html2text后：
Indexing http://17hmr.net/index.php?action=profile;area=showposts;u=994
--- 0.0240960121155 seconds ---
Indexing http://17hmr.net/index.php?board=1.3060;sort=last_post
--- 0.0262401103973 seconds ---
Indexing http://17hmr.net/index.php?topic=12827.msg177073
--- 0.0259499549866 seconds ---
Indexing http://17hmr.net/index.php?topic=6751.45
--- 0.0249440670013 seconds ---
Indexing http://1889.ca/2012/11/interview-with-horror-author-mike-kearby/
--- 0.0152020454407 seconds ---
Indexing http://1980s.fm/modules.php?name=Forums&file=profile&mode=viewprofile&u=94
--- 0.151058912277 seconds ---
Indexing http://1n73r.net/category/microsoft/windows-microsoft/xp/
--- 0.0693669319153 seconds ---
Indexing http://2013missworld.com/
--- 0.0448951721191 seconds ---
Indexing http://24demayito.blogspot.com/
--- 0.111493110657 seconds ---
Indexing http://24kadra.com/2009/03/04/serial-bratany/
--- 0.145864963531 seconds ---
添加html2text和一个分类器（小的）后：
Indexing http://102theriver.iheart.com/articles
--- 0.333050012589 seconds ---
Indexing http://1035kissfm.iheart.com/articles/trending-104650/reading-rainbow-campaign-nets-1-million-12410738
--- 0.334407091141 seconds ---
Indexing http://1037theq.iheart.com/articles/trending-465498/tiesto-celebrates-a-town-called-paradise-12478486/
--- 0.34556388855 seconds ---
Indexing http://1065ctq.iheart.com/articles/national-news-104668/new-electronic-license-plates-could-be-11383289/
--- 0.330471038818 seconds ---
Indexing http://10kbullets.com/reviews/neon-nights/
--- 0.328196048737 seconds ---
Indexing http://12160.info/group/gunsandtactics/forum/topic/show?id=2649739%3ATopic%3A1105218&xg_source=msg
--- 0.353976011276 seconds ---
Indexing http://12under12under2012.blogspot.com/2012/04/aprils-forsta-vinnare-blev.html
--- 0.363568067551 seconds ---
Indexing http://1350kman.com/settlement-reached-in-salina-contamination-cleanup/
--- 0.367321968079 seconds ---
Indexing http://14ers.com/php14ers/loginviaforum.php?prgm=peakstatus_main
--- 0.309129953384 seconds ---
Indexing http://16sarkisozleri.blogspot.com/2012/12/nasip-degilmis-demet-akaln-ftozcan-deniz.html
--- 0.361335992813 seconds ---
添加html2text和一个分类器（大的）后：
Indexing http://10000birds.com/white-crested-laughingthrush.htm
--- 2.16983008385 seconds ---
Indexing http://1012lounge.com/
--- 1.48357391357 seconds ---
Indexing http://1015store.com/dresses-by-colors/coral-dresses.html
--- 1.85999703407 seconds ---
Indexing http://1019ampradio.cbslocal.com/tag/happy-holidays/
--- 1.24361300468 seconds ---
Indexing http://102theriver.iheart.com/articles
--- 1.25308895111 seconds ---
Indexing http://1035kissfm.iheart.com/articles/trending-104650/reading-rainbow-campaign-nets-1-million-12410738
--- 1.19226098061 seconds ---
Indexing http://1037theq.iheart.com/articles/trending-465498/tiesto-celebrates-a-town-called-paradise-12478486/
--- 1.14514183998 seconds ---
Indexing http://1065ctq.iheart.com/articles/national-news-104668/new-electronic-license-plates-could-be-11383289/
--- 1.09987902641 seconds ---
Indexing http://10kbullets.com/reviews/neon-nights/
--- 1.07253599167 seconds ---
Indexing http://12160.info/group/gunsandtactics/forum/topic/show?id=2649739%3ATopic%3A1105218&xg_source=msg
--- 1.1537129879 seconds ---
添加 html2text 和两个分类器后：
Indexing http://12under12under2012.blogspot.com/2012/04/aprils-forsta-vinnare-blev.html
--- 1.43961000443 seconds ---
Indexing http://1350kman.com/settlement-reached-in-salina-contamination-cleanup/
--- 1.37341785431 seconds ---
Indexing http://14ers.com/php14ers/loginviaforum.php?prgm=peakstatus_main
--- 1.26939201355 seconds ---
Indexing http://16sarkisozleri.blogspot.com/2012/12/nasip-degilmis-demet-akaln-ftozcan-deniz.html
--- 1.36402606964 seconds ---
Indexing http://17hmr.net/index.php?action=profile;area=showposts;u=994
--- 1.23323822021 seconds ---
Indexing http://17hmr.net/index.php?board=1.3060;sort=last_post
--- 1.22554993629 seconds ---
Indexing http://17hmr.net/index.php?topic=12827.msg177073
--- 1.23036003113 seconds ---
Indexing http://17hmr.net/index.php?topic=6751.45
--- 1.20131611824 seconds ---
Indexing http://1889.ca/2012/11/interview-with-horror-author-mike-kearby/
--- 1.1732749939 seconds ---
Indexing http://1980s.fm/modules.php?name=Forums&file=profile&mode=viewprofile&u=94
--- 1.36015105247 seconds ---
Indexing http://1n73r.net/category/microsoft/windows-microsoft/xp/
--- 1.2988049984 seconds ---
很少提及
这个项目也部署在 aws 上。当我在aws上运行它时，它显示每个索引0.4（我自己的1.3）。
问题
我怎样才能提高所有这些的性能？ 我应该让我的分类器的训练更轻松，但更精确吗？ 为什么aws和我的电脑差别这么大？ 你需要代码来理解吗？如果需要我可以添加。
欢迎所有想法！
【问题讨论】：
标签： python performance amazon-web-services filter rank