Search with SharePoint 2007 and ISA 2006

Our SharePoint implementation is a bit unique although I'm surprised more people don't have it setup this way. We have 2 FE, 1 App server, and 2 clustered SQL servers. All of that is behind ISA and ISA handles the authentication. It is set to bridge HTTPS(ISA FBA and NTLM) to HTTP(SharePoint NTLM). The reasoning we have HTTP on the SharePoint servers is due to the fact it would reduce load and it wasn't really necessary. This caused us a problem however. The crawler couldn't access the local SharePoint sites. ISA would kill the connection whenever the crawler ran due to a loop. ISA doesn't like to proxy an internal server to itself. This makes sense. A server shouldn't have to proxy through ISA in order to index content on itself.

The obvious solution would be to have the crawler talk directly to the server and bypass ISA. That's where I ran into the next problem. I setup the HOSTS file so it points to the IP of the server. The crawl still failed. The trouble was with the crawl rules. Remember I have HTTP on the SP servers? The crawl rules were trying to connect with HTTPS which would fail. Unfortunately the search rules wouldn't allow me to change it to http://. Every time I did, it changed it back to HTTPS thanks to the AAM. The only way I could find to get around this was to turn on SSL for the back-end. This is overkill but it works. And sense crawling is happening on the same server hosting the index, load is minimal.