基于 boost::asio 的慢速 http 客户端 - (分块传输)

Question

我正在使用以下代码（取自 boost 教程）从服务器获取 json 字符串。

问题是它需要一些时间来执行，即超过 2 秒才能完成，并且客户端和服务器都在本地主机上。如果我删除程序的最后两行，即此时：

while (boost::asio::read(socket, response, boost::asio::transfer_at_least(1), 错误))

程序执行得非常快。可能是什么问题？

        boost::asio::streambuf response;
        boost::asio::read_until(socket, response, "\r\n");

        std::istream response_stream(&response);
        std::string http_version;
        response_stream >> http_version;
        unsigned int status_code;
        response_stream >> status_code;
        std::string status_message;
        std::getline(response_stream, status_message);
        if (!response_stream || http_version.substr(0, 5) != "HTTP/")
        {
          std::cout << "Invalid response\n";
          return 1;
        }
        if (status_code != 200)
        {
          std::cout << "Response returned with status code " << status_code << "\n";
          return 1;
        }


        boost::asio::read_until(socket, response, "\r\n\r\n");

        // Process the response headers.
        std::string header;
        while (std::getline(response_stream, header) && header != "\r");


        if (response.size() > 0)
          std::cout << &response;

        // Read until EOF, writing data to output as we go.
        boost::system::error_code error;
        while (boost::asio::read(socket, response,
              boost::asio::transfer_at_least(1), error))
          std::cout << &response;
        if (error != boost::asio::error::eof)
          throw boost::system::system_error(error);

一个 tcpdump 显示来自服务器的一些数据

HTTP/1.1 200 OK
Connection: close
Content-Length: 42
Server: C/1.1
Date: Thu, 24 Nov 2016 07:47:27 GMT

{"Out":[1],"In":[1,2,3,4,5,6]}

Answer 1

从 cmets 的讨论中了解到，主要问题在于读取分块数据。对于 HTTP 块编码数据，大小在块数据开始之前以十六进制为前缀。因此，必须读取该块的内容长度大小。

      asio::streambuf response;
      // Get till all the headers
      asio::read_until(socket, response, "\r\n\r\n");

      // Check that response is OK. 
      std::istream response_stream(&response);
      std::string http_version;
      response_stream >> http_version;
      std::cout << "Version : " << http_version << std::endl;

      unsigned int status_code;
      response_stream >> status_code;

      std::string status_message;
      std::getline(response_stream, status_message);

      if (!response_stream || http_version.substr(0, 5) != "HTTP/") {
        std::cerr << "invalid response";
        return -1; 
      }

      if (status_code != 200) {
        std::cerr << "response did not returned 200 but " << status_code;
        return -1; 
      }

      //read the headers.
      std::string header;
      while (std::getline(response_stream, header) && header != "\r") {
        std::cout << "H: " << header << std::endl;
      }

      bool chunk_size_present = false;
      std::string chunk_siz_str;

      // Ignore the remaining additional '\r\n' after the header
      std::getline(response_stream, header);

      // Read the Chunk size
      asio::read_until(socket, response, "\r\n");
      std::getline(response_stream, chunk_siz_str);
      std::cout << "CS : " << chunk_siz_str << std::endl;
      size_t chunk_size = (int)strtol(chunk_siz_str.c_str(), nullptr, 16);


      // Now how many bytes yet to read from the socket ?
      // response might have some additional data still with it
      // after the last `read_until`
      auto chunk_bytes_to_read = chunk_size - response.size();

      std::cout << "Chunk Length = " << chunk_size << std::endl;
      std::cout << "Additional bytes to read: " << response_stream.gcount() << std::endl;

      std::error_code error;
      size_t n = asio::read(socket, response, asio::transfer_exactly(chunk_bytes_to_read), error);

      if (error) {
        return -1; //throw boost::system::system_error(error);
      }

      std::ostringstream ostringstream_content;
      ostringstream_content << &response;

      auto str_response = ostringstream_content.str();
      std::cout << str_response << std::endl;

有点难以理解的是asio::read_until 保证它读取数据达到提供的模式，但它也可以将更多数据读取到缓冲区中。

Answer 2

HTTP 中唯一的“EOF”是 TCP 连接关闭时。在这种情况下，您很幸运，服务器在关闭连接前仅 2 秒后就超时了 - 否则，您的应用程序会停留更长时间。

您需要使用Content-Length 值来了解要读取多少数据，而不是寻找EOF 条件。

谷歌“HTTP pipelining”了解为什么服务器没有在你期望的时候关闭 TCP 连接。