DJ .DuMbAsS
New Member
I have a crawlerbot which downloads various pages in parallel using parallel.for and parallel.foreach. The MaxDegreeOfParallelism is set to around 20 since increasing it does not seem to improve the performance further and I do not want to overload any web servers.When I start the crawling from an ASP.NET website (of all places), it is running with acceptable performance and it all takes a few hours. Due to the design issues with this setup, I am changing this so I can start it from the console instead.All the code I need is in a dll that can be accessed by both the website and the console app.There is no output to the console except for a startup message.The problem is that when I run this from the console (or from the debugger), it takes about 10 times longer(!) than when I run it from the ASP.NET website. I would have found it easier to understand if the ASP.NET version would be slower than the console.Why is this so or how can I debug this or find out why it is so or fix the problem?Edit: This also happens in when trying both variants through the visual studio debugger.The output to the 'Output' window from 'Debug' is similar to:The thread '' (0x1d34) has exited with code 0 (0x0).The thread '' (0xf20) has exited with code 0 (0x0).The thread '' (0x3414) has exited with code 0 (0x0).The thread '' (0x35c8) has exited with code 0 (0x0).The thread '' (0xdc) has exited with code 0 (0x0).The thread '' (0x1c98) has exited with code 0 (0x0).The thread '' (0x3308) has exited with code 0 (0x0).The thread '' (0x2b00) has exited with code 0 (0x0).The thread '' (0x2f7c) has exited with code 0 (0x0).The thread '' (0x345c) has exited with code 0 (0x0).The thread '' (0x2a18) has exited with code 0 (0x0).The thread '' (0x138) has exited with code 0 (0x0).The thread '' (0x3650) has exited with code 0 (0x0).The thread '' (0x376c) has exited with code 0 (0x0).except that in the ASP.NET case this text is written about 10 times faster.Perhaps it could also be worth mentioning that the dll invokes 7zip via COM for each downloaded page to compress it.