Parallell processing using PHP

VotaTyptoda

New Member
I have to scrap a web site where i need to fetch multiple URLs and then process them one by one. The current process somewhat goes like this..I fetch a base URL and get all secondary URLs from this page, then for each secondary url I fetch that URL, process found page, download some photos (which takes quite a long time) and store this data to database, then fetch next URL and repeat the process.In this process, I think I am wastingn some time in fetching secondary url at the start of each iteration. So i am trying to fetch next URLs parallely while processing first iteration.The solution in my mind is, from main process call a php script, say downloader, which will download all the URL ( with curl_multi or wget ) and store them in some database. My questions are
- How to call such downloder ASYNCHRONOUSLY, I don't want my main script to wait till downloder completes.
- Any location to store downloaded data, such as shared memory. Of course, other than database.
- There any chances that data gets currupt while storing and retriving, how to avoide this?
- Also, please guide me know if anyone have a better plan.
 
Back
Top