Click here to Skip to main content
15,867,686 members
Articles / General Programming / Threads

MultiThread Download Accelerator Console

Rate me:
Please Sign up or sign in to vote.
4.94/5 (5 votes)
27 Feb 2012GPL32 min read 37.9K   938   23   10
Porting Axel a lighter download accelerator for Linux and other Unices to NATIVE WIN32

Introduction

Axel is a lighter download accelerator for Linux and other Unices. It should compile (and run) on BSD, Solaris, Darwin (Mac OS X) and Win32 (Cygwin) systems.

Porting to NATIVE WIN32 without Cygwin, what I did in the #if WIN32 #endif, for example:

  • Changed Linux file descriptor for the new socket to WIN32 SOCKET descriptor referencing the new socket
  • Changed close, write, read to closesocket, send, recv
  • Changed select blocking thread to WSAEventSelect Asynchronous I/O
  • Changed pthread_t for the thread to HANDLE referenced the thread
  • Changed pthread_create, pthread_join, pthread_cancel to CreateThread, WaitForSingleObject, TerminateThread
  • Changed Linux file descripter for the saving file to HANDLE referenced the saving state
  • Changed open, read, write, close to CreateFile, ReadFile, WriteFile, CloseHandle

Background

DEBUG Axel to see multi-thread downloading way:

  • TCP connect (socket, bind, connect) to host such as http://localhost provided by IIS
  • Send http header request:
    GET / HTTP/1.0
    Host: www.codeproject.com
    Range: bytes=1-
    User-Agent: Axel 2.4 (WIN32)
  • Get the content length from header reply:
    HTTP/1.1 206 Partial Content
    Content-Type: text/html
    Last-Modified: Tue, 20 Sep 2011 03:39:05 GMT
    Accept-Ranges: bytes
    ETag: "3c2bb1d84677cc1:0"
    Server: Microsoft-IIS/7.5
    X-Powered-By: ASP.NET
    Date: Fri, 24 Feb 2012 08:41:12 GMT
    Connection: close
    Content-Length: 688 /* Here is what I need */
    Content-Range: bytes 1-688/689
  • Divide the content range into threads' number for instance 10:
    Downloading 0-67 using conn. 0
    Downloading 68-136 using conn. 1
    Downloading 137-205 using conn. 2
    Downloading 206-274 using conn. 3
    Downloading 275-343 using conn. 4
    Downloading 344-412 using conn. 5
    Downloading 413-481 using conn. 6
    Downloading 482-550 using conn. 7
    Downloading 551-619 using conn. 8
    Downloading 620-688 using conn. 9
  • Send http header request with range [from, to] via one of the threads:
    GET / HTTP/1.0
    Host: localhost
    Range: bytes=206-274
    User-Agent: Axel 2.4 (WIN32)

recv the buffer from SOCKET, WriteFile the buffer from http header's reply to HANDLE CreateFile with temporary file *.st (Axel style), terminate && close the thread when downloaded the content range. Temporary file is helpful to the server host resuming available.

Using the Code

The source file text.c is the main entry considering as a good example and testcase about how to use Axel API.

C++
conf_t conf;
conf_init(&conf);
conf.num_connections = 10;  /* multi-thread numbers */
conf.add_header_count = 0;
/* axel_t object construct */
axel_t *axel = axel_new(&conf, 0, "http://localhost");
strcpy(axel->filename, "filename.htm");
axel_open(axel);
axel_start(axel);
/* Have not downloaded the total content */
while (!axel->ready) 
{
    axel_do(axel);
    /* printf some verbose about download threads info such as speed and complete percent */
}
/* axel_t object destruct */
axel_close(axel);

Points of Interest

Cross platform development - played with Axel console under Gentoo Linux to see the Multi-thread download accelerator working way, then ported with NATIVE WIN32 API to Windows. I complied the source code with VS2005 (Choose mbstring project setting instead of Unicode which brings ^M CTRL-V CTRL-M, unrecognized Chinese character, and binary file trash issue) under Windows 7 64-bit, so it needs more contributors to debug under other Windows distribution.

Improvement

There is WSAPoll supported only by Windows Vista, 7 && Server 2008 to determine the status of one or more sockets. It might be more efficient than WSAEventSelect.

History

  • 2012-02-27 xzhai BUG FREE :)
  • 2012-02-26 xzhai 
    • Fixed Re-connection bug
    • Fixed BIGSIZE file freezing bug
  • 2012-02-24 xzhai
    • Ported Axel v2.4 to WIN32 without Cygwin

License

This article, along with any associated source code and files, is licensed under The GNU General Public License (GPLv3)


Written By
Engineer
China China
An individual human existence should be like a river - small at first, narrowly contained within its banks, and rushing passionately past boulders and over waterfalls. Gradually the river grows wider, the banks recede, the waters flow more quietly, and in the end, without any visible break, they become merged in the sea, and painlessly lose their individual being.

Comments and Discussions

 
GeneralMy vote of 4 Pin
freeliey24-Feb-12 19:09
freeliey24-Feb-12 19:09 
GeneralRe: My vote of 4 Pin
Leslie Zhai24-Feb-12 20:09
Leslie Zhai24-Feb-12 20:09 
GeneralTODO Pin
Leslie Zhai24-Feb-12 19:03
Leslie Zhai24-Feb-12 19:03 
QuestionAny measurements? Pin
Jamming124-Feb-12 16:34
Jamming124-Feb-12 16:34 
AnswerRe: Any measurements? Pin
Leslie Zhai24-Feb-12 17:40
Leslie Zhai24-Feb-12 17:40 
GeneralRe: Any measurements? Pin
Jamming124-Feb-12 17:58
Jamming124-Feb-12 17:58 
GeneralRe: Any measurements? Pin
Leslie Zhai24-Feb-12 18:54
Leslie Zhai24-Feb-12 18:54 
GeneralRe: Any measurements? Pin
Michael McConnell27-Feb-12 3:06
Michael McConnell27-Feb-12 3:06 
GeneralRe: Any measurements? Pin
Leslie Zhai27-Feb-12 17:44
Leslie Zhai27-Feb-12 17:44 
The comparison or benchmark list in TODO, these days I fixed bug, BUG FREE, increased performance choosing WSAEventSelect instead of original select. I will give the benchmarks ASAP.
QuestionHUNTING BUG Pin
Leslie Zhai24-Feb-12 15:28
Leslie Zhai24-Feb-12 15:28 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.