Jsoup 在某些链接上返回 Status 400 错误，但在同一页面上没有其他链接

Question

很奇怪，我得到了前 12 个左右链接所需的结果，然后它每次都在同一个链接 (AAA 090) 上崩溃。我不确定链接之间有什么区别。

    import org.jsoup.*;
    import org.jsoup.helper.*;
    import org.jsoup.nodes.*;
    import org.jsoup.select.*;
    import org.w3c.dom.Document;

    import java.io.*; // Only needed if scraping a local File.

    public class Scraper {


        public Scraper() {

            org.jsoup.nodes.Document page = null;
            org.jsoup.nodes.Document prefix = null;
            org.jsoup.nodes.Document course = null;
            org.jsoup.nodes.Document cls = null;
            try {
                page = Jsoup.connect("https://erpdnssb.cccs.edu/PRODCCCS/ccns_pub_controller.p_command_processor?pi_course_status=A&pi_term_code=201920&pi_search_type=SB_PREFIX").get();

            // get the page title
            String title = page.title();
            System.out.println("title: " + title);

            // get all links in page
            Elements links = page.select("a[href]");
            for (Element link : links) {
                if(!link.text().equalsIgnoreCase("HELP")&&!link.text().equalsIgnoreCase("Exit"))
                { 
                    prefix = Jsoup.connect("https://erpdnssb.cccs.edu/PRODCCCS/" + link.attr("href")).get();
                    Elements links2 = prefix.select("a[href]");
                    for (Element link2 : links2) {
                        if(link2.text().matches("[A-Z]{3}"))
                        {
                            course = Jsoup.connect("https://erpdnssb.cccs.edu/PRODCCCS/" + link2.attr("href")).get();
                            Elements links3 = course.select("a[href]");
                            for (Element link3 : links3) {
                                if(link3.text().matches("[A-Z]{3} [0-9]{3}"))
                                {
                                    cls = Jsoup.connect("https://erpdnssb.cccs.edu/PRODCCCS/" + link3.attr("href")).get();
                                    Elements links4 = cls.getAllElements();


                                    //Elements courseNum = link4.select("body > div.pagebodydiv > table:nth-child(4) > tbody > tr:nth-child(1)");
                                    System.out.println("\nhref = " + link3.attr("href") + "\n" +  cls.text() + "\n");
                                    System.out.println("link: " + cls.tagName());
                                    System.out.println("Course Number: " + link3.data().toString());

                                }
                            }
                        }
                    }
                }
            }
        }catch (IOException ioe) {
            ioe.printStackTrace();
        }
    }

    public static void main (String args[]) {

        new Scraper();
    }

}

org.jsoup.HttpStatusException：获取 URL 的 HTTP 错误。状态=400，>URL=https://erpdnssb.cccs.edu/PRODCCCS/ccns_pub_controller.p_command_processor?>pi_search_type=SB_COURSE&_subj_code=AAA&pi_crse_numb=090&pi_archive_date=&pi_co>urse_status=A&pi_term_code=201920 at >org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:776 )at >org .jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:722) 在 >org.jsoup.helper.HttpConnection.execute(HttpConnection.java:306) 在 >org.jsoup.helper.HttpConnection.get(HttpConnection.java :295) 在 Scraper.(Scraper.java:42) 在 Scraper.main(Scraper.java:64)"

Answer 1

我认为您正在遵循“如果可以复杂，为什么要简单？”的路径。我只是看看你试图抓取的那个网站。该网站的结构和组织良好。所以不需要连接字符串，使用正则表达式并在构造函数中进行所有的抓取。我建议采取以下步骤：

1. 获取第一页并选择带有从 A 到 Z 的前缀链接的表
2. 对于来自 1. 的每个链接，获取前缀的页面并选择带有课程前缀链接的表。例如前缀 A 获取从 AAA 到 AVT 的链接
3. 对于来自 2. 的每个课程前缀链接，获取课程前缀的页面并选择包含课程链接的表格。例如，AAA 获取从 AAA 010 到 AAA 999 的链接

您的代码可能类似于：

import java.io.IOException;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

public class JsoupTest {
    final static String homePage = "https://erpdnssb.cccs.edu/PRODCCCS/ccns_pub_controller.p_command_processor?"
                      + "pi_course_status=A&pi_term_code=201920&pi_search_type=SB_PREFIX";

    public static void main(String[] args) {
        try {
            Document page = Jsoup.connect(homePage).get();
            Elements prefixLinks = page.select(".pagebodydiv > table:nth-child(6) tbody tr td a");

            for(Element prefix : prefixLinks){
                Document prefixPage = Jsoup.connect(prefix.absUrl("href")).get();
                Elements coursePrefixLinks = prefixPage.select(".pagebodydiv > table:nth-child(6) tbody tr td a");

                for(Element coursePrefix : coursePrefixLinks){
                    Document coursePrefixPage = Jsoup.connect(coursePrefix.absUrl("href")).get();
                    Elements courseLinks = coursePrefixPage.select(".pagebodydiv > table:nth-child(6) tbody tr td a");                    

                    for(Element course : courseLinks){
                        Document coursePage = Jsoup.connect(course.absUrl("href")).get();
                        try{
                            String courseNr = coursePage.select("tr:contains(Course:)").first().text();  
                            String courseTitle = coursePage.select("tr:contains(Title:)").first().text(); 
                            System.out.println(courseNr +"\n" + courseTitle + "\n********************");
                        }
                        catch(NullPointerException np){
                            System.out.println("Broken link, Page not Found");
                        }
                    }
                }
            }

        } catch (IOException ex) {
            ex.printStackTrace();
        }
    }
}