MultiThread Download Accelerator Console

• Download source - 45 KB
• SVN goes here https://xsandbox.googlecode.com/svn/trunk/axel-win32

Introduction

Axel is a lighter download accelerator for Linux and other Unices. It should compile (and run) on BSD, Solaris, Darwin (Mac OS X) and Win32 (Cygwin) systems.

Porting to NATIVE WIN32 without Cygwin, what I did in the #if WIN32 #endif, for example:

• Changed Linux file descriptor for the new socket to WIN32 SOCKET descriptor referencing the new socket
• Changed close, write, read to closesocket, send, recv
• Changed select blocking thread to WSAEventSelect Asynchronous I/O
• Changed pthread_t for the thread to HANDLE referenced the thread
• Changed pthread_create, pthread_join, pthread_cancel to CreateThread, WaitForSingleObject, TerminateThread
• Changed Linux file descripter for the saving file to HANDLE referenced the saving state
• Changed open, read, write, close to CreateFile, ReadFile, WriteFile, CloseHandle

Background

DEBUG Axel to see multi-thread downloading way:

• TCP connect (socket, bind, connect) to host such as http://localhost provided by IIS
• Send http header request:

  GET / HTTP/1.0
  Host: www.codeproject.com
  Range: bytes=1-
  User-Agent: Axel 2.4 (WIN32)

• Get the content length from header reply:

  HTTP/1.1 206 Partial Content
  Content-Type: text/html
  Last-Modified: Tue, 20 Sep 2011 03:39:05 GMT
  Accept-Ranges: bytes
  ETag: "3c2bb1d84677cc1:0"
  Server: Microsoft-IIS/7.5
  X-Powered-By: ASP.NET
  Date: Fri, 24 Feb 2012 08:41:12 GMT
  Connection: close
  Content-Length: 688 /* Here is what I need */
  Content-Range: bytes 1-688/689

• Divide the content range into threads' number for instance 10:

  Downloading 0-67 using conn. 0
  Downloading 68-136 using conn. 1
  Downloading 137-205 using conn. 2
  Downloading 206-274 using conn. 3
  Downloading 275-343 using conn. 4
  Downloading 344-412 using conn. 5
  Downloading 413-481 using conn. 6
  Downloading 482-550 using conn. 7
  Downloading 551-619 using conn. 8
  Downloading 620-688 using conn. 9

• Send http header request with range [from, to] via one of the threads:

  GET / HTTP/1.0
  Host: localhost
  Range: bytes=206-274
  User-Agent: Axel 2.4 (WIN32)

recv the buffer from SOCKET, WriteFile the buffer from http header's reply to HANDLE CreateFile with temporary file *.st (Axel style), terminate && close the thread when downloaded the content range. Temporary file is helpful to the server host resuming available.

Using the Code

The source file text.c is the main entry considering as a good example and testcase about how to use Axel API.

conf_t conf;
conf_init(&conf);
conf.num_connections = 10;  /* multi-thread numbers */
conf.add_header_count = 0;
/* axel_t object construct */
axel_t *axel = axel_new(&conf, 0, "http://localhost");
strcpy(axel->filename, "filename.htm");
axel_open(axel);
axel_start(axel);
/* Have not downloaded the total content */
while (!axel->ready) 
{
    axel_do(axel);
    /* printf some verbose about download threads info such as speed and complete percent */
}
/* axel_t object destruct */
axel_close(axel);

Points of Interest

Cross platform development - played with Axel console under Gentoo Linux to see the Multi-thread download accelerator working way, then ported with NATIVE WIN32 API to Windows. I complied the source code with VS2005 (Choose mbstring project setting instead of Unicode which brings ^M CTRL-V CTRL-M, unrecognized Chinese character, and binary file trash issue) under Windows 7 64-bit, so it needs more contributors to debug under other Windows distribution.

Improvement

There is WSAPoll supported only by Windows Vista, 7 && Server 2008 to determine the status of one or more sockets. It might be more efficient than WSAEventSelect.

History

• 2012-02-27 xzhai BUG FREE :)
• 2012-02-26 xzhai 
• Fixed Re-connection bug
• Fixed BIGSIZE file freezing bug
• 2012-02-24 xzhai
• Ported Axel v2.4 to WIN32 without Cygwin