【问题标题】:Extracting structured data from web page using JSOUP and Coldfusion使用 JSOUP 和 Coldfusion 从网页中提取结构化数据
【发布时间】:2022-05-06 13:13:19
【问题描述】:

我需要使用 Coldfusion 使用 JSOUP(或任何其他有效方法)从网站中提取食谱的结构化数据。

数据结构如下:https://developers.google.com/search/docs/advanced/structured-data/recipe

我需要从页面中获取 JSON 并将其解析为可用的变量。

我尝试了许多不同的选择,但都没有成功。我不知道 JSOUP,并感谢您的帮助。

数据如下所示:

<script type=\"application/ld+json\">
{
  \"@context\": \"https://schema.org/\",
  \"@type\": \"Recipe\",
  \"name\": \"Party Coffee Cake\",
  \"image\": [
    \"https://example.com/photos/1x1/photo.jpg\",
    \"https://example.com/photos/4x3/photo.jpg\",
    \"https://example.com/photos/16x9/photo.jpg\"
  ],
  \"author\": {
    \"@type\": \"Person\",
    \"name\": \"Mary Stone\"
  },
  \"datePublished\": \"2018-03-10\",
  \"description\": \"This coffee cake is awesome and perfect for parties.\",
  \"prepTime\": \"PT20M\",
  \"cookTime\": \"PT30M\",
  \"totalTime\": \"PT50M\",
  \"keywords\": \"cake for a party, coffee\",
  \"recipeYield\": \"10\",
  \"recipeCategory\": \"Dessert\",
  \"recipeCuisine\": \"American\",
  \"nutrition\": {
    \"@type\": \"NutritionInformation\",
    \"calories\": \"270 calories\"
  },
  \"recipeIngredient\": [
    \"2 cups of flour\",
    \"3/4 cup white sugar\",
    \"2 teaspoons baking powder\",
    \"1/2 teaspoon salt\",
    \"1/2 cup butter\",
    \"2 eggs\",
    \"3/4 cup milk\"
    ],
  \"recipeInstructions\": [
    {
      \"@type\": \"HowToStep\",
      \"name\": \"Preheat\",
      \"text\": \"Preheat the oven to 350 degrees F. Grease and flour a 9x9 inch pan.\",
      \"url\": \"https://example.com/party-coffee-cake#step1\",
      \"image\": \"https://example.com/photos/party-coffee-cake/step1.jpg\"
    },
    {
      \"@type\": \"HowToStep\",
      \"name\": \"Mix dry ingredients\",
      \"text\": \"In a large bowl, combine flour, sugar, baking powder, and salt.\",
      \"url\": \"https://example.com/party-coffee-cake#step2\",
      \"image\": \"https://example.com/photos/party-coffee-cake/step2.jpg\"
    },
    {
      \"@type\": \"HowToStep\",
      \"name\": \"Add wet ingredients\",
      \"text\": \"Mix in the butter, eggs, and milk.\",
      \"url\": \"https://example.com/party-coffee-cake#step3\",
      \"image\": \"https://example.com/photos/party-coffee-cake/step3.jpg\"
    },
    {
      \"@type\": \"HowToStep\",
      \"name\": \"Spread into pan\",
      \"text\": \"Spread into the prepared pan.\",
      \"url\": \"https://example.com/party-coffee-cake#step4\",
      \"image\": \"https://example.com/photos/party-coffee-cake/step4.jpg\"
    },
    {
      \"@type\": \"HowToStep\",
      \"name\": \"Bake\",
      \"text\": \"Bake for 30 to 35 minutes, or until firm.\",
      \"url\": \"https://example.com/party-coffee-cake#step5\",
      \"image\": \"https://example.com/photos/party-coffee-cake/step5.jpg\"
    },
    {
      \"@type\": \"HowToStep\",
      \"name\": \"Enjoy\",
      \"text\": \"Allow to cool and enjoy.\",
      \"url\": \"https://example.com/party-coffee-cake#step6\",
      \"image\": \"https://example.com/photos/party-coffee-cake/step6.jpg\"
    }
  ],
  \"aggregateRating\": {
    \"@type\": \"AggregateRating\",
    \"ratingValue\": \"5\",
    \"ratingCount\": \"18\"
  },
  \"video\": {
    \"@type\": \"VideoObject\",
    \"name\": \"How to make a Party Coffee Cake\",
    \"description\": \"This is how you make a Party Coffee Cake.\",
    \"thumbnailUrl\": [
      \"https://example.com/photos/1x1/photo.jpg\",
      \"https://example.com/photos/4x3/photo.jpg\",
      \"https://example.com/photos/16x9/photo.jpg\"
     ],
    \"contentUrl\": \"http://www.example.com/video123.mp4\",
    \"embedUrl\": \"http://www.example.com/videoplayer?video=123\",
    \"uploadDate\": \"2018-02-05T08:00:00+08:00\",
    \"duration\": \"PT1M33S\",
    \"interactionStatistic\": {
      \"@type\": \"InteractionCounter\",
      \"interactionType\": { \"@type\": \"WatchAction\" },
      \"userInteractionCount\": 2347
    },
    \"expires\": \"2019-02-05T08:00:00+08:00\"
  }
}
</script>

我尝试了以下方法:

<cfset source = \"https://www.allrecipes.com/recipe/216319/homemade-sweet-italian-sausage-mild-or-hot/\">

<cfhttp method=\"get\" url=\"#source#\" result=\"theresult\" useragent=\"Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/533.7 (KHTML, like Gecko) Chrome/5.0.391.0 Safari/533.7\"> 
<cfhttpparam type=\"header\" name=\"Accept-Encoding\" value=\"gzip,deflate,sdch\" >
<cfhttpparam type=\"header\" name=\"Proxy-Connection\" value=\"keep-alive\" >
<cfhttpparam type=\"header\" name=\"Accept\" value=\"application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5\">
<cfhttpparam type=\"header\" name=\"Accept-Language\" value=\"en-US,en;q=0.8\">
<cfhttpparam type=\"header\" name=\"Accept-Charset\" value=\"ISO-8859-1,utf-8;q=0.7,*;q=0.3\">
<cfhttpparam type=\"cookie\" name=\"some-cookie\" value=\"1\">
</cfhttp>

上面有他,我得到了网页。

然后我尝试提取 JSON:

<cfscript>
// Create the jsoup object
Jsoup = createObject(\"java\", \"org.jsoup.Jsoup\");
// HTML string
html = \"#theresult.filecontent#\";
// Parse the string
document = Jsoup.parse(html);
// Extract content
title = document.title();
tags = document.select(\"script[type=application/ld+json]\"); 
</cfscript>
<cfdump var=\"#tags#\">
<cfloop index=\"e\" array=\"#tags#\">
<cfoutput>
    #e.attr(\"content\")#<br>
</cfoutput>
</cfloop>

但我什么也没得到。

  • 你试过什么??首先提供一些代码示例。
  • 用我的尝试更新了我的问题。
  • 您为什么不使用正则表达式的任何原因?您是否尝试过输出 tags.toString()?
  • 从结果中提取 JSON 的最佳方法是隔离开始和结束 &lt;script&gt; 标记之间的文本,并从该结果中使用 deserializeJSON() 将其分配给 CF 对象。这使得从结果结构中访问要使用的所有变量值变得非常容易。

标签: json web-scraping coldfusion jsoup


【解决方案1】:
<script type="application/ld+json">[
    {
      "@context": "http://schema.org",
      "@type": "BreadcrumbList",
       ...
    }
</script>

&lt;script&gt; 标记没有名为 "content" 的属性(只有一个名为 "type" 的属性)。要检索标记内容(或其内部 html),请使用 Element.html() 方法。然后将返回的内容反序列化为json:

<cfscript>
   Jsoup = createObject("java", "org.jsoup.Jsoup");
   document = Jsoup.parse( theResult.fileContent );
   tag = document.select("script[type=application/ld+json]").first(); 
   if (isJSON(tag.html())) {
       contents = deserializeJSON( tag.html() );
       writeDump(contents);
   }
</cfscript>

【讨论】:

  • 非常感谢。你能帮我输出“内容”吗?我试过这个: <cfloop array="#contents#" index="prop"> <cfoutput> #prop.recipeIngredient#<br> </cfoutput> </cfloop> 我得到一个错误,“recipeIngredient is undefined in支柱”
猜你喜欢
  • 2013-11-15
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 2023-03-18
  • 2015-07-08
  • 2019-09-16
  • 1970-01-01
相关资源
最近更新 更多