在linux上一行代码不用写实现自动采集+hadoop分词

将下面的shell脚本保存成到xxx.sh，然后执行即可在linux上一行代码不用写实现自动采集+hadoop分词

cd /opt/hadoop

mkdir spider
wget -O spider/test.html "http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html"
hadoop fs -mkdir /spider
hadoop fs -put spider/test.html /spider

hadoop jar share/hadoop/mapreduce/wordcount.jar wordcount.wordcount /spider/test.html /fenci2

执行结果如下：

在linux上一行代码不用写实现自动采集+hadoop分词

2021-07-04
2022-02-23
2021-06-10
2022-01-21
2022-02-26
2021-09-18
2022-12-23