Hive 静态分区加载所有记录

Question

我正在尝试根据员工记录的部门执行 hive 静态分区，当我执行加载命令将数据插入分区表时，它会根据我在加载数据命令中指定的分区值更改所有部门值.

我在 hdfs 中的数据如下所示：

1,adi,Admin,50000,A
2,Gokul,Admin,50000,B
3,Janet,Sales,60000,A
4,Hari,Admin,50000,C
5,Sanker,Admin,50000,C
6,Margaret,Tech,12000,A
7,Nirmal,Tech,12000,B
8,jinju,Engineer,45000,B
9,Nancy,Admin,50000,A
10,Andrew,Manager,40000,A
11,Arun,Manager,40000,B
12,Harish,Sales,60000,B
13,Robert,Manager,40000,A
14,Laura,Engineer,45000,A
15,Anju,Ceo,100000,B
16,Aarathi,Manager,40000,B
17,Parvathy,Engineer,45000,B
18,Gopika,Admin,50000,B
19,Steven,Engineer,45000,A
20,Michael,Ceo,100000,A

我的分区表如下图：

create table employee(
id string,
name string,
role string,
salary string)
partitioned by (dept string)
row format delimited
fields terminated by ','
lines terminated by '\n'
stored as textfile;

load command
load data inpath '/user/adithyan/employee.txt' overwrite into table employee partition (dept='A');

我已经给出了上面的输入数据，执行后这个命令记录已经从 hdfs 插入到 hive，但它加载了由 'A' 更改的部门的所有记录

输出：

employee.id employee.name   employee.degree employee.salary employee.dept
1   adi Admin   50000   A
2   Gokul   Admin   50000   A
3   Janet   Sales   60000   A
4   Hari    Admin   50000   A
5   Sanker  Admin   50000   A
6   Margaret    Tech    12000   A
7   Nirmal  Tech    12000   A
8   jinju   Engineer    45000   A
9   Nancy   Admin   50000   A
10  Andrew  Manager 40000   A
11  Arun    Manager 40000   A
12  Harish  Sales   60000   A
13  Robert  Manager 40000   A
14  Laura   Engineer    45000   A
15  Anju    Ceo 100000  A
16  Aarathi Manager 40000   A
17  Parvathy    Engineer    45000   A
18  Gopika  Admin   50000   A
19  Steven  Engineer    45000   A
20  Michael Ceo 100000  A

所有部门都已更改为错误的 A。 有人可以帮助我如何将分区数据插入我的表中吗？

Answer 1

加载数据似乎不适合这项工作。

加载操作目前是纯复制/移动操作 数据文件到 Hive 表对应的位置。

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Loadingfilesintotables

但是，您可以使用外部表来实现您的目标。

create external table employee_ext
(
    id      string
   ,name    string
   ,role    string
   ,salary  string
   ,dept    string
)
row format delimited
fields terminated by ','
lines terminated by '\n'
stored as textfile
location '/user/adithyan/'
;

---

insert into employee partition (dept) select * from employee_ext;

---

select * from employee
;

---

+-------------+---------------+---------------+-----------------+---------------+
| employee.id | employee.name | employee.role | employee.salary | employee.dept |
+-------------+---------------+---------------+-----------------+---------------+
|           1 | adi           | Admin         |           50000 | A             |
|           3 | Janet         | Sales         |           60000 | A             |
|           6 | Margaret      | Tech          |           12000 | A             |
|           9 | Nancy         | Admin         |           50000 | A             |
|          10 | Andrew        | Manager       |           40000 | A             |
|          13 | Robert        | Manager       |           40000 | A             |
|          14 | Laura         | Engineer      |           45000 | A             |
|          19 | Steven        | Engineer      |           45000 | A             |
|          20 | Michael       | Ceo           |          100000 | A             |
|           2 | Gokul         | Admin         |           50000 | B             |
|           7 | Nirmal        | Tech          |           12000 | B             |
|           8 | jinju         | Engineer      |           45000 | B             |
|          11 | Arun          | Manager       |           40000 | B             |
|          12 | Harish        | Sales         |           60000 | B             |
|          15 | Anju          | Ceo           |          100000 | B             |
|          16 | Aarathi       | Manager       |           40000 | B             |
|          17 | Parvathy      | Engineer      |           45000 | B             |
|          18 | Gopika        | Admin         |           50000 | B             |
|           4 | Hari          | Admin         |           50000 | C             |
|           5 | Sanker        | Admin         |           50000 | C             |
+-------------+---------------+---------------+-----------------+---------------+