FTP traffic - Help with php code

[erosbk]

I posted yesterday a piece of code for calculate ftp traffic in the tasklist (it will update ftp traffic every minute, like web traffic).

I need to know right now, which is the best way to read and show that information to user. I have a multiserver enviroment, and ftp log is in the same server that ISPConfig main is running...

But if my ftp/site server is in a different server (ispconfig in another), how can I read it remotely? what is the best way? do I have to update always the table of ispconfig main server, copying date from ftp server?

http://bugtracker.ispconfig.org/inde...otes&sort=desc

Thanks!

[till]

Data can not be read from the master remotely on the slave. For that reason, you will have to run a script on the slave that updates the traffic data in the mysql database of the master server.

[erosbk]

Ok till, is should be easy. How can I detect in a slave which is the master server?, or in ispconfig, which ones are the variables used to connect to master server?

[till]

Please see ispconfig configuration file /usr/local/ispconfig/server/lib/config.inc.php. The mysql master login details are in there. Please be aware that the mysql user has only write access to the monitor_data and web traffic table. If you need access to your table as well, you will have to grant specific permissions for that table in the installer. Do not grant access to all tables as this would cause security problems when a slave gets hacked!

[erosbk]

Thanks Till. Please give me advise. I am coding this for ispconfig, not just for me. So, is any chance that the code become part of ispconfig or you will writte your own for this? I am trying to consider it for multiserver and for different distros (but I think that I will not have change to test in all of them...). The code will be posted in the tasklist of course.

thank you

[till]

You code can be added to ispcongig main version if it works well. Here a few things that might be worth to consider.

1) Shall the statistics aggregated by FTP user or by website? If we want to show them by website, then it might be better to add a column to the web_stats table and insert the traffic data there, so we dont need a new database table then and dont have to deal with the extra permissions during setup. This would also enable is to show the statistics as part of the website traffic stats easily without the need of a separate list.

2) In your script you read the ftp log in one chunk with the file() function. This function loads the complete file into memory in one step. If you have a lrge ftp file (e.g. 1 GB, then this will fail. I thin its better to use this approach to go trough all log lines which can deal with any file size:

$handle = fopen ("/tmp/inputfile.txt", "r");
while (!feof($handle)) {
$buffer = fgets($handle, 4096);
echo $buffer;
}
fclose ($handle);

so you process the lines inside the while loop.

3) It might be nescessary to add some code that ensures that you dont read the ftp traffic in twice e.g. by storing the date / time of the last processed ftp log line in a file and when the script is executed again, the skip all ftp log lines where the date is older then the date of that last log line.

[erosbk]

Thanks till, I have little knowledge in programming (well... not so little), but I am new in php.

Addin ftp traffic to web site traffic will be easy. But in this case, with ftp traffic by user, if there are two or more users per site, clients could see who is using the most traffic. If you limit traffic per month, I think that this approach would be usefull for clients.

It is up to you to decide which one is the better approach! I will wait your answer before starting coding. I want to make this for ispconfig, not just for me!

Anyway, in my script I am saving in table every last timestamp per using, calculating the max in next run and comparing it with each log line to know if that line must be added or not to the table... it is not the best... I think that using a file should be the best... but, where to save that file? is a directory in ispconfig for this? does it have a table for this?

thanks again

edit: I think that adding a new table will be the best... you can get a lot of information with it... is good for both clients and users, in every approach... site traffic is site traffic... ftp traffic is ftp traffic...

[erosbk]

Add ispconfig system table (multiuse to store values of different types, for different modules that could need it in a future)
- With this, you can save a variable value, needed to be updated from time to time:
- ( ftp_traffic, traffic_timestamp, "", 0, 1234512345 ) (instead of saving data to a file if needed, or saving a value

multiple times in differents columns as it was my original idea...)

sys_vars
* var_module (char)
* var_name (char)
* var_string (varchar)
* var_integer (bigint)
* var_timestamp (timestamp)

Add ftp_traffic table

ftp_traffic
* hostname
* ftp_user
* traffic_date
* traffic_type
* traffic_bytes

Basically, this should be the program (in pseudo-pseudo code :P)

If server = FTP server (where to obtain this from a slave??)

read last update from sys_vars.var_timestamp and save in lasttimestamp
read ftplogfile

while !eof(ftplogfile)

if ftplogfile.line.timestamp > lasttimestamp

updatearray[] = ftplogfile.line.values, hostname of ftplogfile.line.ftp_user
x = x + 1
endif

if x = 10
call addtraffic
x = 0
end if
end while

if x > 0
call addtraffic
end if


function addtraffic
begin transaction
if exists in ftp_traffic
update ftp_traffic (values and hostname)
else
insert ftp_traffic (values and hostname)
endif
update sys_vars.var_timestamp with updatearray[].timestamp
end transaction
end function

end if

I hope you understand my idea, sorry for my poor and limited english...

[till]

ISPConfig has already a config system to store values, so please do not duplicate that.

Using a separate ftp traffic table is ok, even if I would prefer to use the existing web_traffic table.

[erosbk]

Ok, where in ispconfig should I save the last timestamp of the logfile? with your answer you will save me a lot of work xD