基于Flink SQL构建流式应用

基于 Kafka, MySQL, Elasticsearch, Kibana，使用 Flink SQL 构建电商用户行为的实时分析应用。Flink SQL 可轻松连接各种外部系统，原生支持事件时间和乱序数据处理、维表关联，有丰富的内置函数等。

1)购买腾讯云服务器，安装Java 13.0.2，安装Docker

//ubuntu 18.04 安装 docker
https://www.cnblogs.com/ws17345067708/p/10455460.html

2)下载 docker-compose.yml 文件。Docker Compose 包含的容器有：

 1 version: '2.1'
 2 services:
 3   datagen:
 4     image: jark/datagen:0.1
 5     command: "java -classpath /opt/datagen/flink-sql-demo.jar myflink.SourceGenerator --input /opt/datagen/user_behavior.log --output kafka kafka:9094 --speedup 1000"
 6     depends_on:
 7       - kafka
 8     environment:
 9       ZOOKEEPER_CONNECT: zookeeper
10       KAFKA_BOOTSTRAP: kafka
11   mysql:
12     image: jark/mysql-example:0.1
13     ports:
14       - "3306:3306"
15     environment:
16       - MYSQL_ROOT_PASSWORD=123456
17   zookeeper:
18     image: wurstmeister/zookeeper:3.4.6
19     ports:
20       - "2181:2181"
21   kafka:
22     image: wurstmeister/kafka:2.12-2.2.1
23     ports:
24       - "9092:9092"
25       - "9094:9094"
26     depends_on:
27       - zookeeper
28     environment:
29       - KAFKA_ADVERTISED_LISTENERS=INSIDE://:9094,OUTSIDE://localhost:9092
30       - KAFKA_LISTENERS=INSIDE://:9094,OUTSIDE://:9092
31       - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
32       - KAFKA_INTER_BROKER_LISTENER_NAME=INSIDE
33       - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
34       - KAFKA_CREATE_TOPICS="user_behavior:1:1"
35     volumes:
36       - /var/run/docker.sock:/var/run/docker.sock
37   elasticsearch:
38     image: docker.elastic.co/elasticsearch/elasticsearch:7.6.0
39     environment:
40       - cluster.name=docker-cluster
41       - bootstrap.memory_lock=true
42       - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
43       - discovery.type=single-node
44     ports:
45       - "9200:9200"
46       - "9300:9300"
47     ulimits:
48       memlock:
49         soft: -1
50         hard: -1
51       nofile:
52         soft: 65536
53         hard: 65536
54   kibana:
55     image: docker.elastic.co/kibana/kibana:7.6.0
56     ports:
57       - "5601:5601"

View Code