We have two identical load balanced web servers (ws1 and ws2) serving the same website. Both have a few crawlers on them at the moment, crawling things like PDFs and pages that contain lots of images (maybe 25/page average). There are ~60 concurrent users between both servers, but the crawlers are by far making the most requests. The IIS App Pool on ws1 is consuming ~500mb RAM and the App Pool on ws2 is consuming ~2gb RAM (4gb available on each server). These consumption levels have been sustained for several hours. I have IIS configured to do static caching and static compression, but not much more I can think of that would impact RAM consumption.The problem is that ws2's RAM is just about maxed out and the web pages on that server are loading considerably slower than on ws1, and that's not good for our end users.My questions:
- 50% of the available RAM (~2gb on ws2) seems like a lot for an App Pool to consume - is it?
- Are there any programming/configuration considerations I should be making to help the server manage its resources, particularly RAM, better, especially when it comes to larger request like PDFs and images?