FTP traffic - Help with php code

Discussion in 'Developers' Forum' started by erosbk, Jun 18, 2011.

  1. erosbk

    erosbk New Member

    I posted yesterday a piece of code for calculate ftp traffic in the tasklist (it will update ftp traffic every minute, like web traffic).

    I need to know right now, which is the best way to read and show that information to user. I have a multiserver enviroment, and ftp log is in the same server that ISPConfig main is running...

    But if my ftp/site server is in a different server (ispconfig in another), how can I read it remotely? what is the best way? do I have to update always the table of ispconfig main server, copying date from ftp server?


  2. till

    till Super Moderator Staff Member ISPConfig Developer

    Data can not be read from the master remotely on the slave. For that reason, you will have to run a script on the slave that updates the traffic data in the mysql database of the master server.
  3. erosbk

    erosbk New Member

    Ok till, is should be easy. How can I detect in a slave which is the master server?, or in ispconfig, which ones are the variables used to connect to master server?
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    Please see ispconfig configuration file /usr/local/ispconfig/server/lib/config.inc.php. The mysql master login details are in there. Please be aware that the mysql user has only write access to the monitor_data and web traffic table. If you need access to your table as well, you will have to grant specific permissions for that table in the installer. Do not grant access to all tables as this would cause security problems when a slave gets hacked!
  5. erosbk

    erosbk New Member

    Thanks Till. Please give me advise. I am coding this for ispconfig, not just for me. So, is any chance that the code become part of ispconfig or you will writte your own for this? I am trying to consider it for multiserver and for different distros (but I think that I will not have change to test in all of them...). The code will be posted in the tasklist of course.

    thank you
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    You code can be added to ispcongig main version if it works well. Here a few things that might be worth to consider.

    1) Shall the statistics aggregated by FTP user or by website? If we want to show them by website, then it might be better to add a column to the web_stats table and insert the traffic data there, so we dont need a new database table then and dont have to deal with the extra permissions during setup. This would also enable is to show the statistics as part of the website traffic stats easily without the need of a separate list.

    2) In your script you read the ftp log in one chunk with the file() function. This function loads the complete file into memory in one step. If you have a lrge ftp file (e.g. 1 GB, then this will fail. I thin its better to use this approach to go trough all log lines which can deal with any file size:

    $handle = fopen ("/tmp/inputfile.txt", "r");
    while (!feof($handle)) {
    $buffer = fgets($handle, 4096);
    echo $buffer;
    fclose ($handle);

    so you process the lines inside the while loop.

    3) It might be nescessary to add some code that ensures that you dont read the ftp traffic in twice e.g. by storing the date / time of the last processed ftp log line in a file and when the script is executed again, the skip all ftp log lines where the date is older then the date of that last log line.
  7. erosbk

    erosbk New Member

    Thanks till, I have little knowledge in programming (well... not so little), but I am new in php.

    Addin ftp traffic to web site traffic will be easy. But in this case, with ftp traffic by user, if there are two or more users per site, clients could see who is using the most traffic. If you limit traffic per month, I think that this approach would be usefull for clients.

    It is up to you to decide which one is the better approach! I will wait your answer before starting coding. I want to make this for ispconfig, not just for me!

    Anyway, in my script I am saving in table every last timestamp per using, calculating the max in next run and comparing it with each log line to know if that line must be added or not to the table... it is not the best... I think that using a file should be the best... but, where to save that file? is a directory in ispconfig for this? does it have a table for this?

    thanks again

    edit: I think that adding a new table will be the best... you can get a lot of information with it... is good for both clients and users, in every approach... site traffic is site traffic... ftp traffic is ftp traffic...
    Last edited: Jun 22, 2011
  8. erosbk

    erosbk New Member

    Add ispconfig system table (multiuse to store values of different types, for different modules that could need it in a future)
    - With this, you can save a variable value, needed to be updated from time to time:
    - ( ftp_traffic, traffic_timestamp, "", 0, 1234512345 ) (instead of saving data to a file if needed, or saving a value

    multiple times in differents columns as it was my original idea...)

    * var_module (char)
    * var_name (char)
    * var_string (varchar)
    * var_integer (bigint)
    * var_timestamp (timestamp)

    Add ftp_traffic table

    * hostname
    * ftp_user
    * traffic_date
    * traffic_type
    * traffic_bytes

    Basically, this should be the program (in pseudo-pseudo code :p)

    If server = FTP server (where to obtain this from a slave??)

    read last update from sys_vars.var_timestamp and save in lasttimestamp
    read ftplogfile

    while !eof(ftplogfile)

    if ftplogfile.line.timestamp > lasttimestamp

    updatearray[] = ftplogfile.line.values, hostname of ftplogfile.line.ftp_user
    x = x + 1

    if x = 10
    call addtraffic
    x = 0
    end if
    end while

    if x > 0
    call addtraffic
    end if

    function addtraffic
    begin transaction
    if exists in ftp_traffic
    update ftp_traffic (values and hostname)
    insert ftp_traffic (values and hostname)
    update sys_vars.var_timestamp with updatearray[].timestamp
    end transaction
    end function

    end if

    I hope you understand my idea, sorry for my poor and limited english...
  9. till

    till Super Moderator Staff Member ISPConfig Developer

    ISPConfig has already a config system to store values, so please do not duplicate that.

    Using a separate ftp traffic table is ok, even if I would prefer to use the existing web_traffic table.
  10. erosbk

    erosbk New Member

    Ok, where in ispconfig should I save the last timestamp of the logfile? with your answer you will save me a lot of work xD
  11. till

    till Super Moderator Staff Member ISPConfig Developer

    You can use the sys_config table.
  12. erosbk

    erosbk New Member

    Is there a variable where to check if I am a slave or if I am master server?

    Right now, I am checking if $conf['dbmaster_host'] is empty or not to know if I am a slave or master... this is not the best way to check it...

    (as I could see, in master server that variable is empty, if it is slave, it has the hostname of the master server...)

    I think that there is a better way to check this... more simple I think... could you help me? I did'nt have enough time to check all ispconfig structure and variables...

    My approach is:

    If I am a slave, I have to save locally timestamp of last read line in ftptraffic log and next save traffic in ftp_traffic in master server.

    If I am master, I have to save both locally, timestamp and ftp traffic.
  13. till

    till Super Moderator Staff Member ISPConfig Developer

    if ($app->dbmaster == $app->db) {
    echo "This is a master server";
  14. erosbk

    erosbk New Member

    I made the script, I can assume that it will work for a multiserver (with one or more ftp/web servers) environment -Debian, pureftpd, log set as stats:-

    Only tested with ftp/web server in master server... I will test it in VMs in the next weeks to see if it is working correctly (only thing I need to test, is to add ftp_traffic in master server, give INSERT, UPDATE, SELECT privileges for each slave (or just for ftp slave?) and test, test and test...

    Right now, It is just crude code, including this files: require('/usr/local/ispconfig/server/lib/config.inc.php');

    But only using a few variables/classes/functions of them...

    I don't have enought time to investigate all ispconfig as I wish, so... I need your guidance...

    1) Have a look here please if you want to see the "ugly working code" (will be improved soon...)


    2) How can I replace this check? (do I am a FTP server??)

    $myhostname = gethostname();
    $conn = mysql_connect($conf['db_host'], $conf['db_user'], $conf['db_password']) or die                      ('Error connecting to mysql');
    $result = mysql_query("select count(server_id) FROM server WHERE file_server=1 and server_name='$myhostname'");
    $ftpvalid = mysql_fetch_array($result);
    if ($ftpvalid[0] > 0) //Do I am a FTP server???, if yes... continues, master or slave is the same

    3) I think that ISPConfig3 has some functions to handle update/insert to dbmaster/dblocal... how can I replace this?

    $conn = mysql_connect($conf['dbmaster_host'], $conf['dbmaster_user'], $conf['dbmaster_password']) or die            $
    $result = mysql_query("select count(username) from ftp_traffic where username='$ftp_user' and traffic_date='$ftp_dat$
    $row = mysql_fetch_array($result);
    if ($row[0] == 0) // No record found, so... insert in master into ftp_traffic
            $query = "insert into ftp_traffic values ('$ftp_domain[0]','$ftp_user','$ftp_date',$ftp_type,'$ftp_bytes')";
    else // Update ftp_traffic in master
           $query = "update ftp_traffic set traffic_bytes=traffic_bytes+'$ftp_bytes' where username='$ftp_user' and tra$
    $conn = mysql_connect($conf['dbmaster_host'], $conf['dbmaster_user'], $conf['dbmaster_password']) or die            $

Share This Page