I mainly want to write this down somewhere so I can refer back to it. HTTP never ceases to surprise me and I never know when I'll need this info again. Yesterday I started looking into a bug with our Web Macro Recorder (WMR) tool. In the world of web scanners being able to record login macros and knowing when to replay them is essential to quality coverage during a scan. The WMR would hang when attempting go through the login process on some sites, e.g. Yahoo.com. After receiving a 301 redirect to the HTTPS login page it would just quit, give up, throw in the proverbial towel. I watched this happen from WireShark, there was no RST there was nothing. The response form Yahoo is: HTTP/1.0 301 Moved Permanently
Location: https://login.yahoo.com/config/login?.src=fpctx&.done=http://www.yahoo.com
Cache-Control: no-cache
Set-Cookie: D=; expires=Thu, 01-Jan-1970 00:00:00 GMT; path=/; domain=.yahoo.com
I decided to try and recreate this on a server I control so that I could mess around. After I noticed it was using HTTP/1.0 I tried using the headers() function in PHP to give responses as http proto ver 1.0. No luck... There is a bug with the headers() function in PHP that requires you to add some additional voodoo. ?php
apache_setenv("downgrade-1.0", "true");
apache_setenv("force-response-1.0", "true");
header("HTTP/1.0 301 Moved Permanently");
header("Location: https://login.yahoo.com/config/login?.src=fpctx&.done=http://$
header("Cache-Control: no-cache");
header("Set-Cookie: D=; expires=Thu, 01-Jan-1970 00:00:00 GMT path=/; domain=.y$
exit("Redirect Success");
?
After implementing this I wasn't able to reproduce the bug in WMR that was occurring on Yahoo.com. The Apache server I was using was also configured to send additional headers that Yahoo wasn't including. The one that caught my suspicion was the Connection: Keep-Alive header. Using interactive mode I tried removing the Connection header sent by my server and SUCCESS was able to reproduce the same behavior that occurs with Yahoo.com. So the one issue is that the WMR is not properly handling HTTP/1.0 redirects. In other testing it was determined that WMR would work sometimes but not every time. e.g. We saw it work about 66% of the time on Citigroup.com. After the revelation with Yahoo.com I decided to go back and look at Citigroup's HTTP responses. I noticed this: HTTP/1.1 302 Moved Temporarily
Date: Thu, 12 Feb 2009 16:11:20 GMT
Server: Hitbox Gateway 9.3.6-rc1
P3P: policyref="/w3c/p3p.xml", CP="NOI DSP LAW NID PSA ADM OUR IND NAV COM"
Set-Cookie: CTG=1234455080; path=/; domain=vendorweb.citibank.com; expires=Thu, 19-Feb-2009 16:11:20 GMT; max-age=604800
nnCoection: close
Pragma: no-cache
Cache-Control: max-age=0, private, proxy-revalidate
Expires: Thu, 12 Feb 2009 16:11:21 GMT
Location: http://vendorweb.citibank.com/HGct
...
Notice the Connection header is misspelled. I figured this probably wasn’t an accident. After Googling I found that sometimes a network device or load balancer will rewrite it to nnCoection: close because (unbeknown to the web server) it is willing and able to maintain the connection. It is a technique usually only applied in environments where there is a lot of load balancing going on. Other ways this same header has been rewritten include Cneonction: close and nncoection: close. This is a technique used as an alternative to rewriting it to Connection: Keep-Alive. The Missed Cneonctions section described in the "Fun with headers" write up has some other interesting examples of that behavior. In conclusion it is best to have a well defined default behavior when implementing a proxy because HTTP might surprise you and not give you what is expected. HTTP and Night Of The Living Proxy | 
