Linker scan.php produces HTTP errors (405/501)

[ray]

As several recent forum entries suggest(http://xinha.gogo.co.nz/punbb/viewtopic.php?id=1042 , http://xinha.gogo.co.nz/punbb/viewtopic.php?id=1061 , http://xinha.gogo.co.nz/punbb/viewtopic.php?id=1060 ) there are some web servers that choke on Linker's scan.php.

Whatever the particular reasons might be, the problem seems to be that scan.php is requested by HTTP POST.

[ray]

rev [837]: request method is now GET, which should be served in all cases without problems

[gogo]

Hmmm, this doesn't seem like a good fix. It doesn't address why postback fails for these people, it certainly shouldn't and if it does it here, then it's likely to also be a problem in other cases.

[gogo]

jwendt wrote:

From the modsecurity configuration files

# TODO Most applications support only two encodings for request bodies
#      because that is all browsers know how to produce. If you are using
#      automated tools to talk to the application you may be using other
#      content types and would want to change the list of supported encodings.
#
#      Note though that ModSecurity parses only three content encodings:
#      application/x-www-form-urlencoded, multipart/form-data request and
#      text/xml. The protection provided for any other type of encoding is
#      inferior.

SecRule REQUEST_METHOD "!^(?:get|head|propfind|options)$" \
    "chain, t:lowercase, deny,log,auditlog,status:501,msg:'Request content encoding is not allowed by policy',id:'960010',severity:'4'"
SecRule REQUEST_HEADERS:Content-Type "!(?:^(?:application/x-www-form-urlencoded$|multipart/form-data;)|text/xml)"

I don't know the mod_security config syntax, but by looking at it I'm guessing this ruleset would appear to trigger if the method was NOT get, head, propfind or options, and the contenttype was not application/x-www-form-urlencoded or multipart/form-data. That is, if it was POST and not either of those content types.

However note that the regexp there for application/x-www-form-urlencoded has an end-anchor there. In XinhaCore?.js we send the content type for posts as application/x-www-form-urlencoded; charset=UTF-8 so it's not matching the regexp which will be why mod_security is balking.

Now, I havn't checked the spec, but from memory I think we are correct in saying that the charset addition is perfectly valid (and infact, maybe it's required for some reason, I doubt any of us would have put it there for no real reason), and probably the problem should be reported to mod_security's maintainers.

[gogo]

But, given that mod_security is becoming increasingly common, it may be prudent to remove the charset for the meantime.

[gogo]

RFC 2616

14.17 Content-Type

The Content-Type entity-header field indicates the media type of the entity-body sent to the recipient or, in the case of the HEAD method, the media type that would have been sent had the request been a GET.

       Content-Type   = "Content-Type" ":" media-type

Media types are defined in section 3.7. An example of the field is

       Content-Type: text/html; charset=ISO-8859-4

Further discussion of methods for identifying the media type of an entity is provided in section 7.2.1. 

(Recipient refers to the server in this case, see section 7 of the RFC)

So we are correct and within specification in appending the charset and it's mod_security that is not playing by the rules.

Four options present themselves for discussion:

1. We could remove the charset, however I have a feeling it was added for a specific reason
2. We could not remove the charset and leave it up to users (or developers) of mod_security to fix their problem
3. We could make it configurable, defaulting to sending the charset.
4. There is the possibility that perhaps we could detect the error and switch automagically to not sending the charset - the danger of this though is that problems might go un-noticed.

[gogo]

changset:869 - will now automatically turn off the charset being sent when an error is detected the first time a postback is tried (and then try again). Reverted the Linker back to postback. Lets see how it works.

[ray]

Gogo, what's wrong with the GET method? There is no data sent, so I find it seem quite logical to use GET.