On some sites the head request takes a long time

[ericdesouza]

I have simple code that does a head request for a URL and then prints the response headers. I've noticed that on some sites, this can take a long time to complete.

For example, requesting //http://www.arstechnica.com// takes about two minutes. I've tried the same request using another web site that does the same basic task, and it comes back immediately. So there must be something I have set incorrectly that's causing this delay.

Here's the code I have:

$ch = curl_init();

curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);

curl_setopt ($ch, CURLOPT_URL, $url);

curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, 20);

curl_setopt ($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);

// Only calling the head

curl_setopt($ch, CURLOPT_HEADER, true); // header will be at output

curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'HEAD'); // HTTP request is 'HEAD'

$content = curl_exec ($ch);

curl_close ($ch);

Here's a link to the web site that does the same function:// http://www.seoconsultants.com/tools/headers.asp//

The code above, at least on my server, takes two minutes to retrieve www.arstechnica.com, but the service at the link above returns it right away.

What am I missing?