通过 php 将 csv 文件上传到 MySQL 数据库

【问题标题】：upload a csv file to MySQL database via php通过 php 将 csv 文件上传到 MySQL 数据库
【发布时间】：2015-06-30 18:03:53
【问题描述】：

我正在制作一个网页，我的非技术客户可以上传一个包含 25,000 多行的 csv 文件。我找到了一种使用 fgetcsv() 的方法，并进行如下查询：

INSERT INTO `usbcData` (first, last, usbc, year) values
("Joshua", "Doom", "1324-1324", "2015");
INSERT INTO `usbcData` (first, last, usbc, year) values
("Turd", "Fergason", "1324-1325", "2014");
INSERT INTO `usbcData` (first, last, usbc, year) values
("Bruce", "Wayne", "1324-1326", "2013");
<Insert 20,000 more INSERTS here.>

问题是这需要的时间太长了。 20-30分钟太长了。所以现在我正试图让 MySQL 的 LOAD DATA INFILE 工作。当我包含来自 /home/joshua/file.csv 的文件（我在我的笔记本电脑上运行 GNU/Linux）时，我可以使它与 phpmyadmin 一起工作，但不能与我的 php 代码一起工作。我已经尝试使用带有和不带有 local 关键字、pdo 扩展、mysqli 扩展和系统调用的查询，但没有一个工作。

我按照本指南：Error 1148 MySQL The used command is not allowed with this MySQL version 设置 my.cnf 以允许 local 关键字工作，以及配置我的 pdo 连接以允许使用 local 关键字。

[mysqld]
local-infile 

[mysql]
local-infile

这里是 php 脚本，用户上传的 csv 文件被发送到这里。它包含几种尝试使用 get MySQL 的 LOAD DATA INFILE 工作的方法：

<?php
ini_set('display_errors', 'On');
error_reporting(E_ALL | E_STRICT);

echo "The file name is ".$_FILES['userfile']['name']."<br>";
echo "The file type is ".$_FILES['userfile']['type']."<br>";
echo "The size of the file is ".$_FILES['userfile']['size']." bytes <br>";
echo "The file is stored on the server as ".$_FILES['userfile']['tmp_name']."<br>";

$file_name = $_FILES['userfile']['name'];
$file_size = $_FILES['userfile']['size'];
$file_tmp = $_FILES['userfile']['tmp_name'];
$file_type = $_FILES['userfile']['type'];

$user = "root";
$pass = "passord";

try {
  $dbh = new PDO('mysql:host=localhost;dbname=usbc', $user, $pass,
               array(PDO::MYSQL_ATTR_LOCAL_INFILE => true,
                     PDO::ATTR_PERSISTENT => true)
);
} catch (Exception $e) {
  die("Unable to connect: " . $e->getMessage());
}


$dbh->query("TRUNCATE TABLE `usbcData`");

if (chmod($file_tmp, 0777)) {
    echo "chmod worked on $file_tmp <br>";
} else {
    echo "chmod didn't work on $file_tmp <br>";
}
$sql = 'load data infile "'.$file_tmp.'" into table `usbcData` ';
$sql .= 'fields terminated by "," ';
$sql .= 'lines terminated by "\n" ';
$sql .= '(first, last, usbc, year) ';
$dbh->query($sql);
//$result = $dbh->query($sql);
echo $sql."<br>";

$sql = 'load data infile "'.$file_name.'" into table `usbcData` ';
$sql .= 'fields terminated by "," ';
$sql .= 'lines terminated by "\n" ';
$sql .= '(first, last, usbc, year) ';
$dbh->query($sql);
//$result = $dbh->query($sql);
echo $sql."<br>";

$sql = 'load data local infile "'.$file_tmp.'" into table `usbcData` ';
$sql .= 'fields terminated by "," ';
$sql .= 'lines terminated by "\n" ';
$sql .= '(first, last, usbc, year) ';
$dbh->query($sql);
//$result = $dbh->query($sql);
echo $sql."<br>";

$sql = 'load data local infile "'.$file_name.'" into table `usbcData` ';
$sql .= 'fields terminated by "," ';
$sql .= 'lines terminated by "\n" ';
$sql .= '(first, last, usbc, year) ';
$dbh->query($sql);
//$result = $dbh->query($sql);
echo $sql."<br>";

$dbh = null;

$mysqli = new mysqli("localhost", "root", "password", "usbc");
if ($mysqli->connect_errno) {
    echo "Failed to connect to MySQL: (" . $mysqli->connect_errno . ") " . $mysqli->connect_error;
}

$sql = 'load data infile "'.$file_tmp.'" into table `usbcData` ';
$sql .= 'fields terminated by "," ';
$sql .= 'lines terminated by "\n" ';
$sql .= '(first, last, usbc, year) ';
$mysqli->query($sql);
//$result = $dbh->query($sql);
echo $sql."<br>";

$sql = 'load data infile "'.$file_name.'" into table `usbcData` ';
$sql .= 'fields terminated by "," ';
$sql .= 'lines terminated by "\n" ';
$sql .= '(first, last, usbc, year) ';
$mysqli->query($sql);
//$result = $dbh->query($sql);
echo $sql."<br>";

$sql = 'load data local infile "'.$file_tmp.'" into table `usbcData` ';
$sql .= 'fields terminated by "," ';
$sql .= 'lines terminated by "\n" ';
$sql .= '(first, last, usbc, year) ';
$mysqli->query($sql);
//$result = $dbh->query($sql);
echo $sql."<br>";

$sql = 'load data local infile "'.$file_name.'" into table `usbcData` ';
$sql .= 'fields terminated by "," ';
$sql .= 'lines terminated by "\n" ';
$sql .= '(first, last, usbc, year) ';
$mysqli->query($sql);
//$result = $dbh->query($sql);
echo $sql."<br>";




$dbUser = "root";
$dbHost = "localhost";
$dbPass = "password";
$dbName = "usbc";

$sql = "LOAD DATA INFILE '$file_tmp' INTO TABLE `usbcData` FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n';";
echo $sql."<br>";
system("mysql -u $dbUser -h $dbHost --password=$dbPass --local_infile=1 -e \"$sql\" $dbName");

$sql = "LOAD DATA INFILE '$file_name' INTO TABLE `usbcData` FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n';";
echo $sql."<br>";
system("mysql -u $dbUser -h $dbHost --password=$dbPass --local_infile=1 -e \"$sql\" $dbName");

$sql = "LOAD DATA local INFILE '$file_tmp' INTO TABLE `usbcData` FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n';";
echo $sql."<br>";
system("mysql -u $dbUser -h $dbHost --password=$dbPass --local_infile=1 -e \"$sql\" $dbName");

$sql = "LOAD DATA local INFILE '$file_name' INTO TABLE `usbcData` FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n';";
echo $sql."<br>";
system("mysql -u $dbUser -h $dbHost --password=$dbPass --local_infile=1 -e \"$sql\" $dbName");

?>

输出如下：

The file name is MonthlyCSVDataFileforIndianaYouth.csv
The file type is text/csv
The size of the file is 759494 bytes 
The file is stored on the server as /tmp/phpT9cyBK
/tmp/phpT9cyBK is writable
chmod worked on /tmp/phpT9cyBK 
load data infile "/tmp/phpT9cyBK" into table `usbcData` fields terminated by "," lines terminated by "\n" (first, last, usbc, year) 
load data infile "MonthlyCSVDataFileforIndianaYouth.csv" into table  `usbcData` fields terminated by "," lines terminated by "\n" (first, last,  usbc, year) 
load data local infile "/tmp/phpT9cyBK" into table `usbcData` fields terminated by "," lines terminated by "\n" (first, last, usbc, year) 
load data local infile "MonthlyCSVDataFileforIndianaYouth.csv" into table `usbcData` fields terminated by "," lines terminated by "\n" (first, last, usbc, year) 
load data infile "/tmp/phpT9cyBK" into table `usbcData` fields terminated by "," lines terminated by "\n" (first, last, usbc, year) 
load data infile "MonthlyCSVDataFileforIndianaYouth.csv" into table `usbcData` fields terminated by "," lines terminated by "\n" (first, last, usbc, year) 
load data local infile "/tmp/phpT9cyBK" into table `usbcData` fields terminated by "," lines terminated by "\n" (first, last, usbc, year) 
load data local infile "MonthlyCSVDataFileforIndianaYouth.csv" into table `usbcData` fields terminated by "," lines terminated by "\n" (first, last, usbc, year) 
LOAD DATA INFILE '/tmp/phpT9cyBK' INTO TABLE `usbcData` FIELDS TERMINATED BY ',' LINES TERMINATED BY ' ';
LOAD DATA INFILE 'MonthlyCSVDataFileforIndianaYouth.csv' INTO TABLE `usbcData` FIELDS TERMINATED BY ',' LINES TERMINATED BY ' ';
LOAD DATA local INFILE '/tmp/phpT9cyBK' INTO TABLE `usbcData` FIELDS TERMINATED BY ',' LINES TERMINATED BY ' ';
LOAD DATA local INFILE 'MonthlyCSVDataFileforIndianaYouth.csv' INTO TABLE `usbcData` FIELDS TERMINATED BY ',' LINES TERMINATED BY ' ';

任何帮助都会很好。 :)

【问题讨论】：

  • 你改完.ini后重启mysql了吗？
  • 如果 MySQL 在localhost 上运行，则不需要LOCAL 选项，因为本地机器与服务器相同。仅当您尝试将文件加载到远程 MySQL 服务器时才需要它。
  • 我确实在更改 .ini 后重新启动了 mysql 数据库。

标签： php mysql csv pdo mysqli


【解决方案1】：

LOCAL 如果您尝试通过脚本语言（例如 PHP）发送文件，则不需要关键字。尝试在没有LOCAL 的情况下运行您的查询。

如果这不起作用，请尝试将文件复制到某处并使用LOAD DATA INFILE 中的新路径，最后删除该文件。

【讨论】：

  • 删除 local 关键字似乎没有帮助。我的一些原始查询没有本地关键字。但是感谢您让我知道当您使用 php 上传文件时不需要本地。我也将文件复制到/tmp/data.csv，并尝试对其进行查询，但也失败了:(
  • 尝试复制到其他文件夹，例如在您的项目根目录中。
【解决方案2】：

终于搞定了！！！我接受了 Robo 的建议，并尝试使用

将文件复制到我的工作目录中 
$file_copy = getcwd()."/uploaded-file.csv";
copy ($file_tmp, $file_copy);

起初这不起作用，因为 Apache/php 在用户“http”下运行，它没有写入我的工作目录的权限。 所以我将用户 http 添加到 joshua 组，因为我在 /home/joshua 下的所有目录都有 joshua 组。 

#add user http to group joshua
sudo gpasswd -a http joshua

然后我注销并重新登录，只是为了确保权限已更新。 然后我尝试使用这个 MySQL 查询来上传正确的文件：

$sql  = "load data CONCURRENT infile \"$file_copy\" ";
$sql .= "into table `usbcData` ";
$sql .= "fields terminated by \",\" ";
$sql .= "lines terminated by \"\n\" ";
$sql .= "(first, last, usbc, year) ";
echo $sql."<br>";
$dbh->query($sql);

但是当我运行 php 脚本时，表 usbcData 表没有更新。我尝试在 phpmyadmin 中运行 sql 查询，它给了我一个权限错误。显然我的 MySQL 实现是在用户“mysql”下运行的，它也没有正确的权限来访问具有组 joshua 的文件。所以我将mysql添加到了joshua组

#add user mysql to group joshua
sudo gpasswd -a mysql joshua

我退出并重新登录，它成功了！呜呼！感谢上帝的stackoverflow，否则我不会有工作。附： Robo 如果我有足够的代表，我会投票赞成你的答案。

【讨论】：

    猜你喜欢
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2017-09-04
    • 2015-10-12
    • 2014-08-29
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多
    热门标签
    Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode