telelcaubron
New Member
I know about the option to set the internal memory \[code\]ini_set("memory_limit","30M");\[/code\]But I wanted to know if there is a better approach for querying data?I have a WHILE LOOP that checks to see if I need to query for another 1000 records.using the offset as the starting record number and the limit as the returned records, I search for all records matching my data request. I hit about 100K in records before I get the error.Now during testing I found that I get the 'Fatal error: Allowed memory size...' error. I've read by setting the above ini_set() to allow for the increase in memory but I wanted to know if I could just code it better?Each time I execute the code below in the WHILE LOOP, the memory usage grows very large. Even if I unset($curl). I think it could be reduced if I could unset the $result and $curl variables after I have parsed out the results before the next cURL query.\[code\]function getRequest($url,$user,$pwd) { $curl = curl_init(); curl_setopt($curl, CURLOPT_VERBOSE, 1); curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 2); curl_setopt($curl, CURLOPT_HEADER, 0); curl_setopt($curl, CURLOPT_HTTPAUTH, CURLAUTH_BASIC); curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1); curl_setopt($curl, CURLOPT_USERPWD, "$user:$pwd"); curl_setopt($curl, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1); curl_setopt($curl, CURLOPT_ENCODING, ''); curl_setopt($curl, CURLOPT_URL, $url); $result = curl_exec($curl); $httpResponseCode = (int)curl_getinfo($curl, CURLINFO_HTTP_CODE); switch ($httpResponseCode) { case 500: // Send problem email break; case 200: // GET was good break; default: // Send problem email break; } curl_close($curl); return $result;} \[/code\]WHILE LOOP (Slim version)\[code\]while($queryFlag) { // $queryFlag is TRUE // Check if we have more records to query, if not set $queryFlag to FALSE // Build cURL URL echo "Before Call Memory Usage: ".memory_get_usage()."\n"; $resultXML = getRequest($query,$user,$pass); echo "After Call Memory Usage: ".memory_get_usage()."\n"; $results = new ParseXMLConfig((string)$resultXML); // This is basically a class for $this->xml = simplexml_load_string($xml); // Loop through results and keep what I'm looking for foreach($results as $resultsKey => $resultsData) { if(preg_match('|^[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}$|i', $resultsData)) { $resultsArr["$resultsData"] = $resultsData; } } }\[/code\]Some memory numbers
- Before Call Memory Usage: 1819736
- After Call Memory Usage: 2285344
- keep data I need
- dump data I don't need
- Next LOOP Iteration
- Before Call Memory Usage: 2084128
- After Call Memory Usage: 2574952