Try HEAD vs GET requests first #3
Description
GET requests can fetch large resources that consume lots of bandwidth unnecessarily.
Example:
2015-06-12 15:05:56 DEBUG [Url][557b3ba416813]Found url: http://upload.wikimedia.org/wikipedia/commons/4/4f/Funny_Car_AAA.JPG []
...snip...
2015-06-12 15:05:56 DEBUG [Url][557b3ba416813]Download complete (after 0.18480515480042s):
Implement a strategy whereby HEAD requests are tried first, so that the resource body isn't downloaded since a lot of desired information is generally in the response headers anyway.
There are instances where this is not the case. For example, if chunked transfer encoding is used, the size of the resource won't be available. While most responses should include a content type header, it's possible they may not. Finally, some resources won't support HEAD (in which case a 405 response should be returned). So, use GET as a fallback in cases where desired information isn't available in HEAD responses.
(Originally opened by @elazar on a different repo, moving issue to this rep)