PHP+MySQL Log Analysis (Apache) Part 2

admin

Administrator
Staff member
This is a re-post of my project "Real-Time Access Counter" after previous discussion and modification from "Coding Help" forum. I believe it's now a database issue.

I would like to use PHP + MySQL to perform Apache log file analysis every 5 minutes via cron job with log rotation.

I was wondering what will happen if I can't manage to finish processing the log files within 5 minutes?

What will happen when there is multiple instances of the log analysis script running at the same time?

I have 3 proposal for the implementation of my log analysis script, can anyone give me some suggestions of which implementation would be most efficient:

1. Execute an external call to the Linux shell command, GREP the lines that I'm interested in from the log file, storing the result as arrary of lines, parse each line by making array of variables, storing number of hits. after the parsing is done, update the data in the MySQL database.

2. Same as #1, storing the result after GREP in array of lines, but parse it and update the MySQL at the same time instead of doing the process via array of variables.

3. Read the whole log file, parse it line by line, and make necessary update to MySQL as #2.

I'm not quite sure how GREP works in the respect of efficiency.

Does it have to do pattern matching for each line to generate my desired result?

In that case, will #3 be more efficient comparing to #1 and #2 since there s only one pass?

PS: I have special needs for making log analysis every 5 minutes, with large log files.
 
Back
Top