没有使用 Curl C API 恢复下载

Question

1. 我正在尝试恢复因 Internet 故障而失败的下载。我用来检查 curl 下载是否成功的函数是：

   curl_multi_info_read
   

   此函数在网络中断时第一次调用时返回正确的错误代码（CURLE_COULDNT_CONNECT）。如果我再次尝试调用它，它会返回 NULL 指针，这意味着没有消息。实际上，我正在使用返回错误代码来检查是否存在互联网连接。这让我很困扰，因为如果没有互联网，它在第二次调用时不会返回任何错误代码。谁能告诉我如何使用此功能检查返回码，因为此错误码（CURLE_COULDNT_CONNECT）对我检查互联网状态非常重要，因此在我回来时从停止的地方恢复下载连接....

2. 为了恢复下载，我正在使用

   curl_easy_setopt (curl, CURLOPT_RESUME_FROM, InternalOffset);
   

   每当我失去互联网连接时，我都会调用此函数来设置选项，以便在互联网连接恢复时可以继续下载...

---

给Daniel Stenberg的备注：

以下是有关平台和 libcurl 版本的一些详细信息：

• curl 版本 - libcurl 7.21.6
• 平台 - Linux (Ubuntu)

评论：

1. 是的。你的看法是对的。我从堆栈中删除了简单的句柄，通过设置新选项（curl_easy_setopt(curl, CURLOPT_RESUME_FROM, InternalOffset)）再次添加到多句柄，最后我做了多执行。如果他们没有互联网连接，它会返回正确的错误。我的问题是：每次我失去互联网连接时是否需要重复上述步骤才能得到正确的错误？如果我不执行这些步骤，curl_multi_info_read 函数是否总是返回 NULL。

2. 我的另一个观察是，当互联网连接恢复时，下载开始恢复。它从之前停止的位置开始下载。这对我来说是一个惊喜。 curl 是否在内部负责在重新连接到 Internet 时恢复下载。如果这是对的吗？我真的需要小心恢复下载或让 curl 正确处理吗？

Answer 1

您没有明确说明您使用的是multi interface 还是easy interface 可能值得一提的是您正在使用什么平台以及您正在使用什么libcurl 等等。

下面是 minimal curl easy 和 multi 针对 libcurl/7.21.6 的测试。

我已经很高兴地拔掉了网线，停止了 http 服务器等等 ~ 似乎应付得很好。

这些可能会对您有所帮助：

curl_easy_setopt(curl, CURLOPT_LOW_SPEED_LIMIT, dl_lowspeed_bytes); //bytes/sec
curl_easy_setopt(curl, CURLOPT_LOW_SPEED_TIME, dl_lowspeed_time); //seconds
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);

注意：您必须非常努力地让curl 在连接断开时摔倒。这是设计使然，但让一些人感到意外。

编辑

我怀疑你会想要使用CURLOPT_TIMEOUT。这将使传输超时。如果您的下载量很大，那么几乎肯定会花费比您准备等待找出网络连接是否存在问题的时间更长的时间 - 超时会受到影响。相比之下，CURLOPT_LOW_SPEED_TIME 超时可能永远不会受到影响，即使经过数小时的传输时间也是如此。

curltest_easy.c：

/*----------------------------------------------------
curltest_easy.c 
WARNING: for test purposes only ~ 
*/
#include <stdio.h>
#include <unistd.h>
#include <curl/curl.h>
#include <curl/types.h>
#include <curl/easy.h>
#include <sys/stat.h>



static int dl_progress(void *clientp,double dltotal,double dlnow,double ultotal,double ulnow)
{
    if (dlnow && dltotal)
        printf("dl:%3.0f%%\r",100*dlnow/dltotal); //shenzi prog-mon 
    fflush(stdout);    
    return 0;
}

static size_t dl_write(void *buffer, size_t size, size_t nmemb, void *stream)
{    
    return fwrite(buffer, size, nmemb, (FILE*)stream); 
}


int do_dl(void) 
{
    CURL *curl;
    FILE *fp;
    CURLcode curl_retval;
    long http_response;
    double dl_size;
    int retval=0;
    long dl_lowspeed_bytes=1000; //1K
    long dl_lowspeed_time=10; //sec        
    /*put something biG here, preferably on a server that you can switch off at will ;) */
    char url[] = {"http://fc00.deviantart.net/fs26/f/2008/134/1/a/Dragon_VII_by_NegativeFeedback.swf"};
    char filename[]={"blah.dl"};

    struct stat st={0};    
    if (!stat(filename, &st));    
    printf("st.st_size:[%ld]\n", st.st_size);  


    if(!(fp=fopen(filename, "ab"))) /*append binary*/
      return 1; 


    curl_global_init(CURL_GLOBAL_DEFAULT);   
    curl = curl_easy_init();

    if (curl) 
    {   
        //http://linux.die.net/man/3/curl_easy_setopt
        curl_easy_setopt(curl, CURLOPT_URL, url);

        /*callbacks*/
        curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, dl_write);
        curl_easy_setopt(curl, CURLOPT_PROGRESSFUNCTION, dl_progress);
        curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0);

        /*curl will keep running -so you have the freedom to recover 
        from network disconnects etc in your own way without
        distrubing the curl task in hand. ** this is by design :p ** */ 
        //curl_easy_setopt(curl, CURLOPT_TIMEOUT, 60);          
        //curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, 30);
        /*set up min download speed threshold & time endured before aborting*/
        curl_easy_setopt(curl, CURLOPT_LOW_SPEED_LIMIT, dl_lowspeed_bytes); //bytes/sec
        curl_easy_setopt(curl, CURLOPT_LOW_SPEED_TIME, dl_lowspeed_time); //seconds while below low spped limit before aborting


        curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
        curl_easy_setopt(curl, CURLOPT_RESUME_FROM,st.st_size);

        /*uncomment this to get curl to tell you what its up to*/
        //curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);


        if(CURLE_OK != (curl_retval=curl_easy_perform(curl)))
        {                      
            printf("curl_retval:[%d]\n", curl_retval);
            switch(curl_retval) 
            {
                //Transferred a partial file
                case CURLE_WRITE_ERROR: //can be due to a dropped connection
                break;

                //all defined in curl/curl.h 

                default: //suggest quitting on unhandled error
                retval=0;
            };    


            curl_easy_getinfo(curl, CURLINFO_CONTENT_LENGTH_DOWNLOAD, &dl_size);
            printf("CURLINFO_CONTENT_LENGTH_DOWNLOAD:%f\n", dl_size);


            curl_retval=curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &http_response);

            //see: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
            printf("CURLINFO_RESPONSE_CODE:%ld\n", http_response);

            switch(http_response)
            {
            case 0: //eg connection down  from kick-off ~suggest retrying till some max limit
            break;

            case 200: //yay we at least got to our url
            break;

            case 206:
            case 416: //http://www.checkupdown.com/status/E416.html
            printf("ouch! you might want to handle this & others\n"); 

            default: //suggest quitting on an unhandled error
            retval=0;
            };            
        }
        else
        {
            printf("our work here is done ;)\n");
            retval=2;
        }


        if (fp)
            fclose(fp);

        if (curl)
            curl_easy_cleanup(curl);
    }

    printf("retval [%d]\n", retval);
    return retval;
}


int main(void) 
{
    while (!do_dl())
    {
        usleep(5000);
    }

    return 0;
}

/* notes ----

$sudo apt-get install libcurl4-gnutls-dev
$ curl-config --libs
-L/usr/lib/i386-linux-gnu -lcurl -Wl,-Bsymbolic-functions

#oook. lets do it:
$ gcc -o curltest_easy curltest_easy.c -L/usr/lib/i386-linux-gnu -lcurl -Wl,-Bsymbolic-functions
$ ./curltest
*/

curltest_multi.c:

/*----------------------------------------------------
curltest_mult1.c
WARNING: for test purposes only ~
*/
#include <stdio.h>
#include <unistd.h>
#include <curl/curl.h>
#include <curl/types.h>
#include <curl/easy.h>
#include <sys/stat.h>

typedef struct S_dl_byte_data
{
    double new_bytes_received;  //from the latest request
    double existing_filesize;
} dl_byte_data, *pdl_byte_data;

static int dl_progress(pdl_byte_data pdata,double dltotal,double dlnow,double ultotal,double ulnow)
{
    /*dltotal := hacky way of getting the Content-Length ~ less hacky would be to first
    do a HEAD request & then curl_easy_getinfo with CURLINFO_CONTENT_LENGTH_DOWNLOAD*/
    if (dltotal && dlnow)
    {
        pdata->new_bytes_received=dlnow;
        dltotal+=pdata->existing_filesize;
        dlnow+=pdata->existing_filesize;
        printf(" dl:%3.0f%% total:%.0f received:%.0f\r",100*dlnow/dltotal, dltotal, dlnow); //shenzi prog-mon
        fflush(stdout);
    }
    return 0;
}

static size_t dl_write(void *buffer, size_t size, size_t nmemb, void *stream)
{
    return fwrite(buffer, size, nmemb, (FILE*)stream);
}

////////////////////////
int do_dl(void)
{
    CURLM *multi_handle;
    CURL *curl;
    FILE *fp;
    CURLcode curl_retval;
    int retval=0;
    int handle_count=0;
    double dl_bytes_remaining, dl_bytes_received;
    dl_byte_data st_dldata={0};
    char curl_error_buf[CURL_ERROR_SIZE]={"meh"};
    long dl_lowspeed_bytes=1000, dl_lowspeed_time=10; /* 1KBs for 10 secs*/

    /*put something biG here, preferably on a server that you can switch off at will ;) */
    char url[] = {"http://fc00.deviantart.net/fs26/f/2008/134/1/a/Dragon_VII_by_NegativeFeedback.swf"};

    char outfilename[]={"blah.swf"}, filename[]={"blah.dl"};
    struct stat st={0};


    if (!(fp=fopen(filename, "ab")) || -1==fstat(fileno(fp), &st)) //append binary
      return -1;

    if (curl_global_init(CURL_GLOBAL_DEFAULT))
      return -2;

    if (!(multi_handle = curl_multi_init()))
      return -3;

    if (!(curl = curl_easy_init()))
      return -4;


    st_dldata.new_bytes_received=st_dldata.existing_filesize=st.st_size;

    //http://curl.haxx.se/libcurl/c/curl_easy_setopt.html
    curl_easy_setopt(curl, CURLOPT_URL, url);

    /*callbacks*/
    curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, dl_write);
    curl_easy_setopt(curl, CURLOPT_PROGRESSFUNCTION, dl_progress);
    curl_easy_setopt(curl, CURLOPT_PROGRESSDATA, &st_dldata);
    curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0);

    /*curl will keep running -so you have the freedom to recover from network disconnects etc
    in your own way without distrubing the curl task in hand. ** this is by design :p **
    The follwoing sets up min download speed threshold & time endured before aborting*/
    curl_easy_setopt(curl, CURLOPT_LOW_SPEED_LIMIT, dl_lowspeed_bytes); //bytes/sec
    curl_easy_setopt(curl, CURLOPT_LOW_SPEED_TIME, dl_lowspeed_time); //seconds while below low spped limit before aborting
    //alternatively these are available in libcurl 7.25
    //curl_easy_setopt(curl, CURLOPT_TCP_KEEPALIVE,1L);
    //curl_easy_setopt(curl, CURLOPT_TCP_KEEPIDLE,10);
    //curl_easy_setopt(curl, CURLOPT_TCP_KEEPINTVL,10);

    curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);

    /*uncomment this to get curl to tell you what its up to*/
    //curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);

    curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, curl_error_buf);


    do
    {
        if (st_dldata.new_bytes_received) //set the new range for the partial transfer if we have previously received some bytes 
        {
            printf("resuming d/l..\n");
            fflush(fp);
            //get the new filesize & sanity check for file; on error quit outer do-loop & return to main
            if (-1==(retval=fstat(fileno(fp), &st)) || !(st_dldata.existing_filesize=st.st_size)) break; 
            //see also: CURLOPT_RANGE for passing a string with our own X-Y range
            curl_easy_setopt(curl, CURLOPT_RESUME_FROM, st.st_size);
            st_dldata.new_bytes_received=0;
        }
        printf("\n\nbytes already received:[%.0f]\n", st_dldata.existing_filesize);

        //re-use the curl handle again & again & again & again... lol
        curl_multi_add_handle(multi_handle, curl);

        do //curl_multi_perform event-loop
        {
            CURLMsg *pMsg;
            int msgs_in_queue;

            while (CURLM_CALL_MULTI_PERFORM == curl_multi_perform(multi_handle, &handle_count));

            //check for any mesages regardless of handle count
            while(pMsg=curl_multi_info_read(multi_handle, &msgs_in_queue))
            {
                long http_response;

                printf("\nmsgs_in_queue:[%d]\n",msgs_in_queue);
                if (CURLMSG_DONE != pMsg->msg)
                {
                    fprintf(stderr,"CURLMSG_DONE != pMsg->msg:[%d]\n", pMsg->msg);
                }
                else
                {
                    printf("pMsg->data.result:[%d] meaning:[%s]\n",pMsg->data.result,curl_easy_strerror(pMsg->data.result));
                    if (CURLE_OK != pMsg->data.result) printf("curl_error_buf:[%s]\n", curl_error_buf);
                    switch(pMsg->data.result)
                    {
                    case CURLE_OK: ///////////////////////////////////////////////////////////////////////////////////////
                    printf("CURLE_OK: ");
                    curl_easy_getinfo(pMsg->easy_handle, CURLINFO_CONTENT_LENGTH_DOWNLOAD, &dl_bytes_remaining);
                    curl_easy_getinfo(pMsg->easy_handle, CURLINFO_SIZE_DOWNLOAD, &dl_bytes_received);
                    if (dl_bytes_remaining == dl_bytes_received)
                    {
                        printf("our work here is done ;)\n");
                        rename(filename, outfilename);
                        retval=1;
                    }
                    else
                    {
                        printf("ouch! st_dldata.new_bytes_received[%f]\n",st_dldata.new_bytes_received);
                        printf("ouch! dl_bytes_received[%f] dl_bytes_remaining[%f]\n",dl_bytes_received,dl_bytes_remaining);
                        retval=dl_bytes_received < dl_bytes_remaining ? 0 : -5;
                    }
                    break; /////////////////////////////////////////////////////////////////////////////////////////////////

                    case CURLE_COULDNT_CONNECT:      //no network connectivity ?
                    case CURLE_OPERATION_TIMEDOUT:   //cos of CURLOPT_LOW_SPEED_TIME
                    case CURLE_COULDNT_RESOLVE_HOST: //host/DNS down ?
                    printf("CURMESSAGE switch handle_count:[%d]\n",handle_count);
                    break; //we'll keep trying

                    default://see: http://curl.haxx.se/libcurl/c/libcurl-errors.html
                    handle_count=0;
                    retval=-5;
                    };


                    //see: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
                    curl_retval=curl_easy_getinfo(pMsg->easy_handle, CURLINFO_RESPONSE_CODE, &http_response);
                    printf("CURLINFO_RESPONSE_CODE HTTP:[%ld]\n", http_response);
                    switch(http_response)
                    {
                    case 0:   //eg connection down  from kick-off ~suggest retrying till some max limit
                    case 200: //yay we at least got to our url
                    case 206: //Partial Content
                    break;

                    case 416:
                    //cannot d/l range ~ either cos no server support
                    //or cos we're asking for an invalid range ~ie: we already d/ld the file
                    printf("HTTP416: either the d/l is already complete or the http server cannot d/l a range\n");
                    retval=2;

                    default: //suggest quitting on an unhandled error
                    handle_count=0;
                    retval=-6;
                    };
                }
            }

            if (handle_count) //select on any active handles
            {
                fd_set fd_read={0}, fd_write={0}, fd_excep={0};
                struct timeval timeout={5,0};
                int select_retval;
                int fd_max;

                curl_multi_fdset(multi_handle, &fd_read, &fd_write, &fd_excep, &fd_max);
                if (-1 == (select_retval=select(fd_max+1, &fd_read, &fd_write, &fd_excep, &timeout)))
                {
                    //errno shall be set to indicate the error
                    fprintf(stderr, "yikes! select error :(\n");
                    handle_count=0;
                    retval=-7;
                    break;
                }
                else{/*check whatever*/}
            }

        } while (handle_count);

        curl_multi_remove_handle(multi_handle,curl);
        printf("continue from here?");
        getchar();        
    }
    while(retval==0);

    curl_multi_cleanup(multi_handle);
    curl_easy_cleanup(curl);
    curl_global_cleanup();
    if (fp) fclose(fp);

    return retval;
}

////////////////////////
int main(void)
{
    int retval;
    printf("\n\ncurl_multi d/l test ~curl version:[%s]\n", curl_version());
    while (1!=(retval=do_dl()))
    {
        printf("retval [%d] continue?\n\n", retval);
        printf("continue?");
        getchar();
    }
    printf("\nend of test!\n\n", retval);
    return retval;
}

/* notes ----

$sudo apt-get install libcurl4-gnutls-dev
$curl-config --libs
-L/usr/lib/i386-linux-gnu -lcurl -Wl,-Bsymbolic-functions

#oook. lets do it:
$gcc -o curltest_multi curltest_multi.c -L/usr/lib/i386-linux-gnu -lcurl -Wl,-Bsymbolic-functions
$./curltest_multi

*/

您可能需要记住在开始全新测试之前删除blah.dl 文件。该程序故意不这样做，因此您可以预先截断现有文件以进行测试；）

注意：对于这样的事情，你可能不应该只依赖CURLE_COULDNT_CONNECT ~你的代码应该主要是错误处理（如果你的程序是严格供个人使用的，可能更少）

编辑

我已更新 curtest_multi.c 以演示 easy_handle 的重用。

并且做注意以下来自the documentation的引用：

单次传输完成后，仍留有easy句柄 添加到多堆栈。您需要先卸下简易手柄 使用 curl_multi_remove_handle(3) 然后关闭它 curl_easy_cleanup(3)，或者可能为它设置新选项并添加它 再次使用 curl_multi_add_handle(3) 开始另一次传输。