php - 在数据库中存储发布的 JSON 对象

Question

我正在使用 Java 读取一个大型 json 文件并发布我的本地主机的每一行，然后获取 JSON 并将其作为对象读取，然后使用 MySQL 将对象的部分存储在该数据库中。

这是一个非常缓慢的过程。

如何优化它？

<?php
error_reporting(0);
@ini_set('display_errors', 0);

$json = $_POST['data'];

if(!empty($json)){

    $obj = json_decode($json);

    $user_id =  $obj->interaction->author->id;
    $user_link =  $obj->interaction->author->link;
    $name =  $obj->interaction->author->name;
    $user_name =  $obj->interaction->author->username;
    $user_gender =  $obj->demographic->gender;
    $user_language =  $obj->twitter->lang;
    $user_image =  $obj->interaction->author->avatar;
    $user_klout =  $obj->klout->score;
    $user_confidence =  $obj->language->confidence;
    $user_desc =  $obj->twitter->user->description;
    $user_timezone =  $obj->twitter->user->time_zone;
    $user_tweet_count = $obj->twitter->user->statuses_count;
    $user_followers_count = $obj->twitter->user->followers_count;
    $user_friends_count = $obj->twitter->user->friends_count;
    $user_location = $obj->twitter->user->location;
    $user_created_at = $obj->twitter->user->created_at;

    $tweet_id =  $obj->twitter->id;
    $tweet_text =  $obj->interaction->content;
    $tweet_link =  $obj->interaction->link;

    $tweet_created_at =  $obj->interaction->created_at;

    $tweet_location =  $obj->twitter->user->location;

    //$tweet_geo_lat =  $obj->interaction->geo->latitude;
    //$tweet_geo_long =  $obj->interaction->geo->longitude;

    $con = mysqli_connect("localhost","root","", "cohort");
    // Check connection
    if (mysqli_connect_errno()) {
      echo "Failed to connect to MySQL: " . mysqli_connect_error();
    }

    $sql = "INSERT INTO tweeters (user_id, screen_name, name, profile_image_url, location, url,
            description, created_at, followers_count,
            friends_count, statuses_count, time_zone,
            last_update, klout, confidence, gender
    )
    VALUES ('$user_id', '$user_name','$name',
    '$user_image', '$user_location', '$user_link',
    '$user_desc', '$user_created_at', '$user_followers_count',
    '$user_friends_count', '$user_tweet_count', '$user_timezone',
    '', '$user_klout', '$user_confidence', '$user_gender' )";

    if (!mysqli_query($con,$sql)) {
      //die('Error: ' . mysqli_error($con));
    }

    $sql = "INSERT INTO search_tweets (tweet_id, tweet_text, created_at_date,
        created_at_time, location, geo_lat,
        geo_long, user_id, is_rt)
        VALUES ('$tweet_id', '$tweet_text','',
                '$tweet_created_at', '$tweet_location', '',
                '', '$user_id', '')";

    if (!mysqli_query($con,$sql)) {
      //die('Error: ' . mysqli_error($con));
    }

    mysqli_close($con);
    echo json_encode(array("id" => $user_id ));
}

?>

Java：

    String inputfile = "D:\\Datasift\\Tweets.json"; //  Source File Name.  
        double nol = 200000; //  No. of lines to be split and saved in each output file.  
        File file = new File(inputfile);  
        Scanner scanner = new Scanner(file);  
        int count = 0;  
        System.out.println("Storing file in stack"); 
        int may = 0, june = 0, just_june=0, july = 0;
        BufferedReader br = null;

        BufferedReader  in = new BufferedReader (new InputStreamReader (new ReverseLineInputStream(file)));


        while(true) {
            String line = in.readLine();
            if (line == null) {
                break;
            }
            //System.out.println("X:" + line);
            // Send POST output.
            URL url1;
        URLConnection   urlConn;
        DataOutputStream    printout;
        DataInputStream     input;
                // URL of CGI-Bin script.
        url1 = new URL ("http://localhost/json/");
                // URL connection channel.
        urlConn = url1.openConnection();
                // Let the run-time system (RTS) know that we want input.
        urlConn.setDoInput (true);
                // Let the RTS know that we want to do output.
        urlConn.setDoOutput (true);
                // No caching, we want the real thing.
        urlConn.setUseCaches (false);
                // Specify the content type.
        urlConn.setRequestProperty
        ("Content-Type", "application/x-www-form-urlencoded");
            printout = new DataOutputStream (urlConn.getOutputStream ());
            String content =
                "data=" + URLEncoder.encode (line);
            printout.writeBytes (content);
            printout.flush ();
            printout.close ();
            // Get response data.
            input = new DataInputStream (urlConn.getInputStream ());
            String str;
            while (null != ((str = input.readLine()))){
                //System.out.println (str);                   
            }

            input.close ();
        }
        System.out.println("Lines in the file: " + count);

Answer 1

我不想以任何方式拖钓，但你为什么不直接用 PHP 读取文件？

Answer 2

如果您多次重复此过程，一种方法是使用多个数据集。

因此，与其执行 INSERT INTO table (field,field) VALUES (value,value) 并循环多次，不如执行 $ins = "INSERT INTO table (field, field) VALUES";然后在你的 foreach 循环中（或每次调用代码）建立一个数组 $ins_values[] = "(escaped_value,escaped_value)";然后运行查询 $ins.implode(',',$ins_values)。

Mysql 在这种情况下会运行得更快，但要注意 Mysql 对 max_allowed_pa​​cket 设置了数据限制，这可能需要根据情况进行调整。

希望对您有所帮助，我已正确理解您的问题。