如何将 json 文件从 https 网页导入 mariadb 或 mysql 数据库？

Question

我正在尝试将文件导入 mariadb (mysql) 数据库。此位置的 Web 上的概念证明文件为 .json 格式。 https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_week.geojson

我知道如何在 db2 for i 上执行此操作。

select * from JSON_TABLE(
               SYSTOOLS.HTTPGETCLOB('https://earthquake.usgs.gov' ||
                '/earthquakes/feed/v1.0/summary/all_week.geojson',null),
                '$.features[*]'
               COLUMNS( MILLISEC BIGINT PATH '$.properties.time',
                        MAG DOUBLE PATH '$.properties.mag', 
                        PLACE VARCHAR(100) PATH '$.properties.place'
                      )) AS X;

这会从网上读取 a 并列出 3 个字段。用插入子句包围它会将它放入数据库文件中。

我想使用 mariadb 在我的家庭服务器上做同样的事情。最终，脚本将在托管服务器上无人看管地运行。

地震数据的 .json 片段如下所示：

… "features":[
{"type":"Feature",
    "properties":{
        "mag":1.1,
        "place":"58 km WNW of Anchor Point, Alaska",
        "time":1640472257402,
        "updated":1640472615410,
        "tz":null,
        "url":"https://earthquake.usgs.gov/earthquakes/eventpage/ak021gi3al5x",
        "detail":"https://earthquake.usgs.gov/earthquakes/feed/v1.0/detail/ak021gi3al5x.geojson",
        "felt":null,
        "cdi":null,
        "mmi":null,
        "alert":null,
        "status":"automatic",
        "tsunami":0,
        "sig":19,
        "net":"ak",
        "code":"021gi3al5x",
        "ids":",ak021gi3al5x,",
        "sources":",ak,",
        "types":",origin,",
        "nst":null,
        "dmin":null,
        "rms":0.79,
        "gap":null,
        "magType":"ml",
        "type":"earthquake",
        "title":"M 1.1 - 58 km WNW of Anchor Point, Alaska"},
        "geometry":{
            "type":"Point",
            "coordinates":[-152.8406,59.9119,89.7]
        },
        "id":"ak021gi3al5x"
    }, ...

Answer 1

只是一个想法，但你可以只写一个 BASH 脚本，例如

#!/bin/sh


cd "$(dirname "$0")"
export PATH=/bin:/usr/bin:/usr/local/bin
TODAY=`date +"%d%b%Y-%H%M"`

MYSQL_HOST='127.0.0.1'
MYSQL_PORT='3306'
MYSQL_USER='root'
MYSQL_PASSWORD='root'


url=https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_week.geojson
DATA=$(curl ${url} 2>/dev/null)
printf '%s' "$DATA" | awk '{print $0}'
exit

...
...
...

然后使用https://webinstall.dev/jq/、Python 和系统上的任何其他工具来提取所需的数据，然后更新数据库。

我更熟悉 PHP，它也可以工作。您可以稍微整理一下，但似乎确实有效。

earthquake.php

<?php
$database = false;
try {

    $options = array(PDO::ATTR_DEFAULT_FETCH_MODE => PDO::FETCH_OBJ, PDO::ATTR_ERRMODE => PDO::ERRMODE_WARNING, PDO::ATTR_EMULATE_PREPARES => true );
    $conn = new PDO('mysql:host=127.0.0.1;dbname=test;port=3306;charset=utf8','root','root', $options);


} catch (PDOException $e) {

    // Echo custom message. Echo error code gives you some info.
    echo '[{"error":"Database connection can not be estabilished. Please try again later.  Error code: ' . $e->getCode() . '"}]';
    exit;
}

$url = "https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_week.geojson";
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
$result = curl_exec($ch);
$features = json_decode($result)->features;
foreach ($features as $feature) {
    echo 'Mag:  '.$feature->properties->mag.', Place:  '.$feature->properties->place.', '.gmdate("Y-m-d H:i:s", $feature->properties->time/1000).PHP_EOL;
    $query = 'INSERT INTO features (mag, place, time) VALUES (?, ?, ?)';
    $params = [$feature->properties->mag, $feature->properties->place, gmdate("Y-m-d H:i:s", $feature->properties->time/1000)];
    $stmt = $conn->prepare($query) or die ('["status":{"error":"Prepare Statement Failure","query":"' .$query . '"}]');
    $stmt->execute($params) or die('[{"error":"' . $stmt->errorInfo()[2]  . '","query":"' .$query . '","params":' .json_encode($params) . '}]');

}

?>

创建一个名为 features 的本地数据库，其中包含 mag、place 和 time 列。如果您的系统上有 php，只需从 CLI 运行它，php 地震.php

例如Insert into mysql from Bash script

我使用了一点 Laravel，并且可能实际上会构建我自己的模型并使用 Eloquent 和一个小 UI 来处理它，但使用脚本似乎是一种选择。