Mar 24 2011

WebClient class. Troubleshooting the 403 (Forbidden) error

Category: web applicationsadmin @ 4:29 pm

The WebClient class included in the System.Web library has a method to download files from a website. The method is called DownloadFile.

I was facing problems with some websites.  The only particular thing about them was that they are all seemed to be from .sz domains (Swaziland).

The problem was that a 403 (forbidden) error was being returned, for example:

WebClient client = new WebClient();
string downloadURL= @"http://example.co.sz";
string localFile=@"file.htm";
//
//example taken from http://csharphowto.blogopogo.com
//
client.Headers.Add("User-Agent", "(my client app)");
client.DownloadFile(downloadURL, localFile);

The first thing I tried was to reproduce using the wget utility in Ubuntu:

wget -U “(my client app)” -d http://example.co.sz

The command above:

  1. Uses an alternate user agent string
  2. Shows debug information to confirm the contents of headers sent to the server when making the web request

and the result was that the page was retrieved OK, so there was nothing wrong with the user agent string.  I looked at the debug output produced by the above example (the -d option) and found the following additional headers sent by wget:

---request begin---
GET / HTTP/1.0
User-Agent: (my client app)
Accept: */*
Host: example.com.sz
Connection: Keep-Alive

So all I did was to add an additional header to my request to include the ‘Accept’ header:

WebClient client = new WebClient();
string downloadURL= @"http://example.co.sz";
string localFile=@"file.htm";
client.Headers.Add("User-Agent", "(my client app)");
client.Headers.Add("Accept", "*/*");
client.DownloadFile(downloadUrl, localFile);

and this resolved the issue.

Leave a Reply

*
= 4 + 2