Hbase on Yarn-编译部署slider

1、编译Slider，

1
2
[[email protected] apache-slider-0.91.0-incubating]# pwd
/usr/local/src/apache-slider-0.91.0-incubating

因采用CDH版本hadoop(2.6.0-cdh5.9.0),hbase(1.2.0-cdh5.9.0)，修改该目录下pom文件

1
2
3
<hadoop.version>2.6.0-cdh5.9.0</hadoop.version>
<hbase.version>1.2.0-cdh5.9.0</hbase.version>
<accumulo.version>1.7.0</accumulo.version>

注释掉slider-core和slider-funtest中对hadoop-minicluster包依赖

1
2
3
4
5
6
7
<!--
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minicluster</artifactId>
<scope>test</scope>
</dependency>
-->

跳过maven测试，执行编译命令

1
[[email protected] apache-slider-0.91.0-incubating]#  mvn clean package -Dmaven.test.skip=true -DskipTests


[INFO] Slider ............................................. SUCCESS [  0.762 s]

[INFO] Command Logger Sample Application .................. SUCCESS [  0.971 s]

[INFO] Slider Command Logger Slider App Package ........... SUCCESS [  0.766 s]

[INFO] Slider Core ........................................ SUCCESS [ 10.766 s]

[INFO] Slider Agent ....................................... SUCCESS [  1.245 s]

[INFO] Apache Slider Binary Distribution .................. SUCCESS [ 21.933 s]

[INFO] Slider Functional Tests ............................ SUCCESS [  5.285 s]

[INFO] Slider App Packages ................................ SUCCESS [  0.012 s]

[INFO] Slider Accumulo App Package ........................ SUCCESS [  5.578 s]

[INFO] Slider HBase App Package ........................... SUCCESS [  0.017 s]

[INFO] Slider Kafka App Package ........................... SUCCESS [  1.046 s]

[INFO] Slider Storm App Package ........................... SUCCESS [  0.011 s]

[INFO] Slider Tomcat App Package Testing WAR file ......... SUCCESS [  0.886 s]

[INFO] Slider Tomcat App Package .......................... SUCCESS [  1.666 s]

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 51.578 s

[INFO] Finished at: 2016-12-01T12:00:14+08:00

[INFO] Final Memory: 89M/2916M

[INFO] ------------------------------------------------------------------------


2、下载hbase-1.2.0-cdh5.9.0.tar.gz，存放目录/usr/local/src

wget http://archive.cloudera.com/cdh5/cdh/5/hbase-1.2.0-cdh5.9.0.tar.gz


3、生成slider-hbase-app-package-1.2.0-cdh5.9.0.zip

1
mvn clean package -P hbase-app-package -D pkg.version=1.2.0-cdh5.9.0 -Dpkg.name=hbase-1.2.0-cdh5.9.0.tar.gz -D pkg.src=/usr/local/src


-Dhbase.version=<hbase version> Hbase版本

-Dpkg.version=<app package version> 应用程序版本

-Dpkg.name=<file name of app tarball> app tar包文件名

-Dpkg.src=<folder location where the pkg is available> app tar包文件所在文件夹



4、将slider编译成功的包部署，配置环境变量


成功编译的包位于SLIDER_SRC_HOME/slider-assembly/target/slider-0.91.0-incubating-all.tar.gz

解压至/usr/local/slider

并配置环境变量export SLIDER_HOME=/usr/local/slider


5、创建package

1
slider package --install --name HBASE --package slider-0.91.0-incubating/slider-hbase-app-package-1.2.0-cdh5.9.0.zip

验证：

1
2
[hadoop@nn1 slider]$ hadoop fs -ls /user/hadoop/.slider/package/HBASE/slider-hbase-app-package-1.2.0-cdh5.9.0.zip
-rw-r--r--   3 hadoop hadoop  241347310 2016-12-01 13:43 /user/hadoop/.slider/package/HBASE/slider-hbase-app-package-1.2.0-cdh5.9.0.zip

6、提取slider-hbase-app-package-1.2.0-cdh5.9.0.zip中appConfig-default.json和resource-default.json

unzip slider-hbase-app-package-1.2.0-cdh5.9.0.zip  appConfig-default.json -d /usr/local/slieder/

unzip slider-hbase-app-package-1.2.0-cdh5.9.0.zip  resource-default.json -d /usr/local/slider/


7、修改appConfig-default.json

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
{
"schema": "http://example.org/specification/v2.0.0",
"metadata": {
},
"global": {
"application.def": ".slider/package/HBASE/slider-hbase-app-package-1.2.0-cdh5.9.0.zip",
"create.default.zookeeper.node": "true",
"java_home": "/usr/local/jdk",
"system_configs": "core-site",
"site.global.app_user": "hadoop",
"site.global.app_version": "1.2.0-cdh5.9.0",
"site.global.app_root": "${AGENT_WORK_ROOT}/app/install/hbase-1.2.0-cdh5.9.0",
"site.global.metric_collector_host": "${NN_HOST}",
"site.global.metric_collector_port": "6188",
"site.global.metric_collector_lib": "",
"site.global.hbase_instance_name": "instancename",
"site.global.hbase_root_password": "secret",
"site.global.user_group": "hadoop",
"site.global.monitor_protocol": "http",
"site.global.hbase_thrift_port": "${HBASE_THRIFT.ALLOCATED_PORT}",
"site.global.hbase_thrift2_port": "${HBASE_THRIFT2.ALLOCATED_PORT}",
"site.global.hbase_rest_port": "${HBASE_REST.ALLOCATED_PORT}",
"site.hbase-env.hbase_master_heapsize": "1024m",
"site.hbase-env.hbase_regionserver_heapsize": "1024m",
"site.hbase-site.hbase.rootdir": "${DEFAULT_DATA_DIR}/data",
"site.hbase-site.hbase.superuser": "${USER_NAME}",
"site.hbase-site.hbase.tmp.dir": "${AGENT_WORK_ROOT}/work/app/tmp",
"site.hbase-site.hbase.local.dir": "${hbase.tmp.dir}/local",
"site.hbase-site.hbase.zookeeper.quorum": "${ZK_HOST}",
"site.hbase-site.zookeeper.znode.parent": "${DEFAULT_ZK_PATH}",
"site.hbase-site.hbase.regionserver.info.port": "0",
"site.hbase-site.hbase.bulkload.staging.dir": "/user/${USER_NAME}/hbase-staging",
"site.hbase-site.hbase.coprocessor.region.classes": "org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint",
"site.hbase-site.hbase.master.info.port": "${HBASE_MASTER.ALLOCATED_PORT}",
"site.hbase-site.hbase.regionserver.port": "0",
"site.hbase-site.hbase.master.port": "0"
},
"components": {
"slider-appmaster": {
"jvm.heapsize": "1024M"
}
}
}

8、修改resource-default.json

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
{
"schema": "http://example.org/specification/v2.0.0",
"metadata": {
},
"global": {
"yarn.log.include.patterns": "",
"yarn.log.exclude.patterns": "",
"yarn.component.instances": "1",
"yarn.vcores": "1"
},
"components": {
"slider-appmaster": {
"yarn.memory": "10240"
},
"HBASE_MASTER": {
"yarn.role.priority": "1",
"yarn.component.instances": "1",
"yarn.placement.escalate.seconds": "10",
"yarn.memory": "15000"
},
"HBASE_REGIONSERVER": {
"yarn.role.priority": "2",
"yarn.component.instances": "1",
"yarn.memory": "15000",
"yarn.container.failure.threshold": "15",
"yarn.placement.escalate.seconds": "60"
},
"HBASE_REST": {
"yarn.role.priority": "3",
"yarn.component.instances": "1",
"yarn.component.placement.policy": "1",
"yarn.memory": "5560"
},
"HBASE_THRIFT": {
"yarn.role.priority": "4",
"yarn.component.instances": "1",
"yarn.component.placement.policy": "1",
"yarn.memory": "5560"
},
"HBASE_THRIFT2": {
"yarn.role.priority": "5",
"yarn.component.instances": "1",
"yarn.component.placement.policy": "1",
"yarn.memory": "5560"
}
}
}

9、启动hbase应用on yarn

[[email protected] slider]$ slider create hbase1 --template ./appConfig-default.json --resources ./resources-default.json

Hbase on Yarn-编译部署slider



     本文转自巧克力黒 51CTO博客，原文链接：http://blog.51cto.com/10120275/1879569，如需转载请自行联系原作者








相关文章：

猜你喜欢
相关资源
相似解决方案
热门标签
Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode