这甚至是JSON吗？

Question

我尝试从http://www.msmarco.org/dataset.aspx 下载数据集。很高兴看到数据集是 JSON 格式。

现在，如果你打开 dev 数据集，你可以看到它的组成元素是这样的：

{"passages": [{"is_selected": 1, "url": "http://www.indeed.com/cmp/Walgreens/salaries", "passage_text": "The average Walgreens salary ranges from approximately $15,000 per year for Customer Service Associate / Cashier to $179,900 per year for District Manager. Average Walgreens hourly pay ranges from approximately $7.35 per hour for Laboratory Technician to $68.90 per hour for Pharmacy Manager. Salary information comes from 7,810 data points collected directly from employees, users, and jobs on Indeed."}, {"is_selected": 0, "url": "http://www.answers.com/Q/What_is_the_average_gross_sales_volume_of_a_single_Walgreen's_Store", "passage_text": "The average revenue in 2011 of a Starbuck Store was $1,078,000, up  from $1,011,000 in 2010.    The average ticket (total purchase) at domestic Starbuck stores in  No … vember 2007 was reported at $6.36.    In 2008, the average ticket was flat (0.0% change)."}, {"is_selected": 0, "url": "http://news.walgreens.com/fact-sheets/frequently-asked-questions.htm", "passage_text": "In fiscal 2014, Walgreens opened a total of 184 new locations and acquired 84 locations, for a net decrease of 273 after relocations and closings. How big are your stores? The average size for a typical Walgreens is about 14,500 square feet and the sales floor averages about 11,000 square feet. How do we select locations for new stores? There are several factors that Walgreens takes into account, such as major intersections, traffic patterns, demographics and locations near hospitals."}, {"is_selected": 0, "url": "http://www.babson.edu/executive-education/thought-leadership/retailing/Documents/walgreens-strategic-evolution.pdf", "passage_text": "th store in 1984, reaching $4 billion in sales in 1987, and $5 billion two years later. Walgreens ended the 1980s with 1,484 stores, $5.3 billion in revenues and $154 million in profits. However, profit margins remained just below 3 percent of sales, and returns on assets of less than 10 percent."}, {"is_selected": 0, "url": "http://www.trefis.com/stock/wag/articles/199532/key-trends-driving-walgreens-business/2013-08-07", "passage_text": "The number of Walgreen stores has risen from 5,000 in 2005 to more than 8,000 at present. The average square footage per store stood at approximately 10,200 and we forecast the figure to remain constant over our review period. Walgreen earned $303 as average front-end revenue per store square foot in 2012."}, {"is_selected": 0, "url": "http://www.walgreens.com/storelocator/find.jsp?requestType=locator", "passage_text": "Your Walgreens Store. Select a store from the search results to make it Your Walgreens Store and save time getting what you need. Your Walgreens Store will be the default location for picking up prescriptions, photos, in store orders and finding deals in the Weekly Ad."}], "query_id": 9652, "answers": ["Approximately $15,000 per year."], "query_type": "numeric", "query": "walgreens store sales average"}
{"passages": [{"is_selected": 0, "url": "http://www.breakintobartending.com/how-much-do-bartenders-make/", "passage_text": "A bartender’s income is comprised mostly of tips– 55% to be exact. In some states, employers aren’t even required to pay their bartenders the minimum wage and can pay as low as $2.13 per hour, and they depend on their tips almost entirely. Bartending can be a lot of things. For some it is exciting, for others exhausting. At times there is a lot of fun to be had, at others it is rather dull. But for the most part, bartending is almost always rewarding in the financial sense, as long as you stick with it."}, {"is_selected": 1, "url": "http://www.breakintobartending.com/how-much-do-bartenders-make/", "passage_text": "According to the Bureau of Labor Statistics, the average hourly wage for a bartender is $10.36, and the average yearly take-home is $21,550. Bartending can be a lot of things. For some it is exciting, for others exhausting. At times there is a lot of fun to be had, at others it is rather dull. But for the most part, bartending is almost always rewarding in the financial sense, as long as you stick with it."}, {"is_selected": 0, "url": "http://careerswiki.com/how-much-do-bartenders-make/", "passage_text": "About 551,100 individuals are employed as bartenders, with half of this number working part-time. The average annual salary for bartenders is $19,050 or an equivalent of $9.16 per hour, including tips. No formal training is needed for one to get a job as a bartender as all it takes are good customer service skills and a comprehensive knowledge about beverages and recipes. "}, {"is_selected": 0, "url": "http://www.answers.com/Q/How_much_do_bartenders_in_Las_Vegas_make", "passage_text": "Confidence votes 11. Bartenders in Vegas can make up to $7-$15 wage plus $10-$50 in tips per hour. But that totally depends on personality, gender and outlet. A hot young female can make $1000 a day tending bar in a strip club, but that is very exceptional. A bartender makes as much as a bartender wants to make. I wouldn't say there is a cap on how much a bartender make in one year. If the service is good and the coversation is i … nteresting a bartender can make alot of money. Location is also a big factor."}, {"is_selected": 0, "url": "https://answers.yahoo.com/question/index?qid=20100102233548AAZEutT", "passage_text": "Best Answer: An average bartender makes about...2 to 3 dollars an hour. but all the money is made off tips depending on the popularity of the bar. I used to make $700 to $1000 a night. but that is in Atlanta. If the bar is busy and you are a good bartender you will make quite a bit. I dont know how much, because I live in a town with a population of 2000 so there is not alot going on around here. Im sure the bartenders make a hundred to two hundred a night total (on a good night)...just depends."}, {"is_selected": 0, "url": "https://answers.yahoo.com/question/index?qid=20100102233548AAZEutT", "passage_text": "Report Abuse. no way to tell you how much bartenders make. in wages anything from minimum to $15 an hour. tips, anywhere from $20 to $300 or more a night. depends on a lot of things. those top dollar jobs only come after a lot of experience. If the bar is busy and you are a good bartender you will make quite a bit. I dont know how much, because I live in a town with a population of 2000 so there is not alot going on around here. Im sure the bartenders make a hundred to two hundred a night total (on a good night)...just depends."}, {"is_selected": 0, "url": "http://work.chron.com/much-bartender-make-annually-7503.html", "passage_text": "Pay by Employment Setting. Bartenders who worked in hotels and hotel restaurants generally reported the highest incomes in 2011, an average of $26,180 a year. Bartenders employed in full-service restaurants tended to earn somewhat less, averaging about $22,130 a year. Bartenders employed by bars earned an average of $20,230 per year, and bartenders who worked for civic and social organizations earned an average of $18,970 a year. The median earnings of bartenders during this period were $9.06 an hour and $18,850 a year. Eighty percent of bartenders in the U.S. reported annual incomes of between $16,170 and $31,860."}, {"is_selected": 0, "url": "http://www.celebritynetworth.com/articles/how-much-does/how-much-does-bartender-make/", "passage_text": "Tips make up half or more of bartender's salaries. If a bartender earned $6.00 an hour, their tips generally average out to $12.00 to $18.00 an hour as additional income. A bartender in an average bar will typically earn $15.00 $30.00 an hour between their wages and tips. According to bartending.org, bartenders in a high volume resort or establishment can earn $50,000 to $75,000 per year between hourly wages and tips. Indeed.com 2010 results show bartenders in restaurants at median salary rates can make a good salary per year: Bartender $73,000."}], "query_id": 9653, "answers": ["$21,550 per year", "The average hourly wage for a bartender is $10.36 and the average yearly take-home is $21,550."], "query_type": "numeric", "query": "how much do bartenders make"}

现在这些块重复没有任何封闭的{}和没有逗号分隔它们。显然这不可能是 JSON。

我的问题是：

1. 这真的是数据集声称的 json 吗？

2. 任何库（最好在 python 中）解析它以提取所有单独的字段。

我在 python 中尝试了ijson，但这看起来真的不像真正的 json。

Answer 1

此文件由每行一个 JSON 文档组成：这是JSON Lines。

整个文件不是有效的 JSON 文档。您应该逐行阅读并将每一行输入 JSON 解析器：

#!/usr/bin/env python3
from sys import argv
import json

with open(argv[1], "r", encoding="UTF-8") as f:
    lines = [ json.loads(line) for line in f ]

注意：另一种类似的格式是 RFC7464 中记录的 JSON-seq。