HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials

HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials (http://www.howtoforge.com/forums/index.php)
-   Installation/Configuration (http://www.howtoforge.com/forums/forumdisplay.php?f=16)
-   -   Traffic stats ok? (http://www.howtoforge.com/forums/showthread.php?t=1569)

BobbyQ 7th December 2005 09:39

Traffic stats ok?
 
Hi!

I'm running ISPConfig since a week now, until now only set up one Web for testing purposes, have a big file in that web that was donloaded several times.

I checked now the stats in ISP Config and its telling me this:

Trafficübersicht (in MB):
Monat Web FTP Mail Gesamt
12/2005: 86375.99 0.00 0.00 86375.99
12/2005 86375.99 0.00 0.00 86375.99
Gesamt: 86375.99 0.00 0.00 86375.99

Now I have a look into the trafficcount that my Provider gives me for this machine and the month of december and they say:

Traffic incoming in MB Traffic outgoing in MB Traffic gesamt in MB
313,7711 9.788,8306 10.102,6017
Gesamt: 313,7711 9.788,8306 10.102,6017

So we got a difference of like 76 gig after only one week, how can this be??
I'm quite sure that the missing 76 Gig WERE NOT MOVED WITHIN MY Providers Network, thats the only reason I know it wouldn't appear on their stats, so where is the problem here??

Are aborted transfers counted like complete transfer by httpd.log or Webalizer?

If there is really such a difference in counting, the traffic limitations on the webs within ISPConfig would be somehow useless....
Thanks in advance for any help!
Robert

till 7th December 2005 09:50

ISPConfig and your provider are measuring something different.

ISPConfig shows you the traffic that has been logged by apache in the logfiles (web) and proftpd (FTP) and postfix (Mail).

Your provider measures the IP traffic that goes trough your network card.

If you dont have other services running on your server that are not measured by ISPConfig and the 76 gig are protocol overhead, i recommend to scan your system with a rootkit checker like rkhunter (www.rootkit.nl)

BobbyQ 7th December 2005 10:16

Ok, done this is the result:

---------------------------- Scan results ----------------------------

MD5
MD5 compared: 0
Incorrect MD5 checksums: 0

File scan
Scanned files: 342
Possible infected files: 0

Application scan
Vulnerable applications: 1 (the only warning it gave me is that root still can log on over ssh.... but thats how I wanted it for setting things up, will change later when machine is running like I want...)

Scanning took 78 seconds
---------------------------------

So practically everything came out with 'OK' and 'clean', so apparently the problem is not in this...

(Would wonder me as there is NO third party app (PHPs in Webs or so..) on the server until now, just 'Perfect-Debian-Setup' and ISPConfig....

FTP AND DNS Server are switched OFF on this machine, and the ISPConfig stats claims the 86 GIG went through WEB transfer only!!!!

Thanks anyway...

Oh, just found another possibility for Error:
The incoming traffic that my Provider gives me is wrong, as I for sure uploaded a 1 Gig testfile to that web and they claim there was only 313MB incoming... Furthermore they told me something like: 'Traffic is counted as soon as a connection/transfer is closed' - so that would mean my Machine should still have somthing like 80 connections opened that still didn't finished transferring the 1Gig file?? Naah... We had for testing purposes something around 20 ppl trying to dl that file... When I dig deeper into the stats of the web, it came out that ONE SINGLE IP produced >40GIG alone downloading the testfile... In the end all testers confirmed they downloaded the file only once, although some had to try several times to get it completely, but 40 times??? Naaa, I still believe someones counting is weird here...

till 7th December 2005 10:27

Oh, i just saw its 76 MB, not gig :)
And the csan with rkhunter is OK too, so i wont worry about the difference. 76 MB are not very much and its not easy to compare the two measurement systems if you dont know exactly what your ISP measures. You can try to measure the traffic of your network card with iptables on your server.

BobbyQ 7th December 2005 10:38

Quote:

Originally Posted by till
Oh, i just saw its 76 MB, not gig :)
And the csan with rkhunter is OK too, so i wont worry about the difference. 76 MB are not very much and its not easy to compare the two measurement systems if you dont know exactly what your ISP measures. You can try to measure the traffic of your network card with iptables on your server.


AM I reading something wrong?

Trafficübersicht (in MB):
Monat Web
12/2005: 86375.99

thats over 80 Gig to me.... or do we have the comma 1 position too far to the right??
and the Provider tells me total of 10 Gig....

I wouldn't complain even if the Difference was 2:1, but this is 8:1.....

till 7th December 2005 10:42

Youre right. I will go and get a cup of coffe to wake up a bit more this morning ;)

BobbyQ 7th December 2005 20:16

Hmm so everyone else is enjoying sensefull stats??
Aaargh why me then?!?!

falko 7th December 2005 20:26

Traffic statistics are never reliable. ISPConfig says something different than webalizer which says something different than the statistics from your provider.
Traffic measurement is a very difficult field...

BobbyQ 10th December 2005 15:04

Well, the problem I see with unreliable stats, is that creating 'New Web Templates' or adding 'New Webs' that should contain a certain amount of traffic is then completely useless with ISPConfig... Instead you have to control the traffic by hand all the time, as the internal traffic counting of ISP Config seems to be broken and no one apparently knows or cares where its going nuts...

Now we also clarified with the one user that appeared to have downloaded >45 Gig within three visits on two days (where only a single 1 Gig Testfile was available to download on the whole site and each tester should and did try to only download it ONCE in complete), he used a download manager and left it on over night, next morning he found the transfer broken at 60%, then resumed it within the download manager and it completed till 90%. THATS ALL!!!
ISP Config is counting for his IP alone 45 Gig transferred, when in reality he did around 900 MByte plus one Resume overhead!!!


To me this smells A LOT like every - even unfinished and broken - transfers are counted like the transfer of the file was completed... not good....and it gets worse the bigger the files are on the site....
Probably I should better add this to the buglist or new features wishlist ...


Just for the fun of numbers, heres the actual comparison between ISPConfigs stats and my Providers stat:
----------------------------------
ISPConfig says:
---------------
Trafficübersicht (in MB):
Monat Web FTP Mail Gesamt
12/2005: 143224.71 0.00 0.00 143224.71

so thats around 140 Gig to me...

Provider says:
--------------
Traffic outgoing in MB Traffic gesamt in MB
14.915,5264 15.371,4832

thats something over 14 Gig...

till 10th December 2005 15:10

The stats in ISPConfig are as reliable as the logfiles of apache are. If the size of downloaded files is reported incorrect by apache in its logfile, then this is not an ISPConfig bug. Have you checked the logfile (webalizer) for the downloaded traffic? Maybe you shall use the download manager to download a file and watch the apache logfile which downloads where counted there.


All times are GMT +2. The time now is 00:22.

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.