Ultimate Security Proxy With Tor

Nowadays, within the growing web 2.0 environment you may want to have some anonymity, and use other IP addresses than your own IP. Or, for some special purposes - a few IPs or more, frequently changed. So no one will be able to track you. A solution exists, and it is called Tor Project, or simply tor. There are a lot of articles and howtos giving you the idea of how it works, I'm not going to describe here onion routing and its principles, I'll rather tell you how practically pull out the maximum out of it.

So, let's start.

Major disadvantages of tor, with all its benefits like security (however, tor manufacturers advice not to rely on its strong anonymity, because there are some cases, when it is possible to track young h4x0r...) are:

1. Long enough session (60 seconds for one connection per one exit node, that means, that you will have one external IP for a minute)
2. Slow perfomance (request takes up to 20 seconds to complete, what makes surfing sites with lot of elements a disaster)
3. All the requests come through one node, and possibly route how your requests arrive can be calculated.

In order to fight these three we are going to use:

1. 8 tor processes, each using separate spool directory
2. 8 privoxy processes, each configured to talk to separate tor.
3. First squid, with malware domains blacklist, will have 8 round robin cache peers configured. (squid-in)
4. Havp, with squid-in as parent. (Anti-virus proxy, using clamav :) )
5. Second squid, that will use havp as parent (squid-out). Users will connect to this one.

Schema:

Tor Proxies

Okay, so you got it, how all it looks like. Now let's do all the required configuration.

For squid-in about 2Gb of disc cache will be enough.
For squid-out it is better to have about 12-15 Gb of disc cache to store clean elements.
So you'll need at least 20 Gb HDD.


Now about RAM.

Each tor process consumes ~ 6 Mb of Ram. 6x8 = 48 ~ 64 Mb of RAM for Tor processes.
Privoxy eats about 2 Mb each. 2x8 = 16 ~ 32 Mb of RAM.
So, 128 Mb of RAM will be enough for Tor and Privoxy processes.
Squid-in will have memory footprint approx 128 Mb. We assume it eats 256 Mb.
Same for Squid-out. We got 768 Mb of RAM total required. If it is gonna be a real system - we assume 1 Gb hardware RAM
installed.

CPU.

As you know, anti-virus scanning is very CPU demanding.
I've used a 2.0 GHz P4.
I've tested this system under concurrent load under 10 users, and it did now show it was exhausted.

Finally, we have:

CPU: 2.0 GHz
RAM: 1.0 Gb
HDD: 20 Gb

Used software:

 

Privoxy

privoxy --version

Privoxy version 3.0.8 (http://www.privoxy.org/)

 

Tor

tor --version
Oct 11 19:49:27.394 [notice] Tor v0.1.2.19. This is experimental software. Do not rely on it for strong anonymity.
Tor version 0.1.2.19.

 

Squid

squid -v

Squid Cache: Version 2.7.STABLE3
configure options: '--prefix=/usr' '--host=x86_64-pc-linux-gnu' '--mandir=/usr/share/man'
'--infodir=/usr/share/info' '--datadir=/usr/share' '--sysconfdir=/etc' '--localstatedir=/var/lib'
'--sysconfdir=/etc/squid' '--libexecdir=/usr/libexec/squid' '--localstatedir=/var'
'--datadir=/usr/share/squid' '--enable-auth=basic,digest,ntlm' '--enable-removal-policies=lru,heap'
'--enable-digest-auth-helpers=password'
'--enable-basic-auth-helpers=SASL,PAM,SMB,multi-domain-NTLM,getpwnam,NCSA,MSNT'
'--enable-external-acl-helpers=wbinfo_group,ip_user,session,unix_group'
'--enable-ntlm-auth-helpers=SMB,fakeauth' '--enable-ident-lookups' '--enable-useragent-log'
'--enable-cache-digests' '--enable-delay-pools' '--enable-referer-log' '--enable-arp-acl' '--with-pthreads'
'--with-large-files' '--enable-htcp' '--enable-carp' '--enable-follow-x-forwarded-for' '--with-maxfd=8192'
'--enable-snmp' '--enable-ssl' '--enable-storeio=ufs,diskd,coss,aufs,null' '--enable-async-io'
'--enable-linux-netfilter' '--libdir=/usr/lib64' '--build=x86_64-pc-linux-gnu'
'build_alias=x86_64-pc-linux-gnu' 'host_alias=x86_64-pc-linux-gnu' 'CC=x86_64-pc-linux-gnu-gcc'
'CFLAGS=-march=k8 -O2 -pipe -fomit-frame-pointer' 'LDFLAGS=-Wl,-O1'

 

Tor Configuration

/etc/tor/torrc
SocksPort 9050 # what port to open for local application connections
SocksBindAddress 127.0.0.1 # accept connections only from localhost
AllowUnverifiedNodes middle,rendezvous
Log notice syslog
RunAsDaemon 1
User tor
Group tor
DataDirectory /var/lib/tor
/etc/tor/torrc2
SocksPort 9150 # what port to open for local application connections
SocksBindAddress 127.0.0.1 # accept connections only from localhost
AllowUnverifiedNodes middle,rendezvous
Log notice syslog
RunAsDaemon 1
User tor
Group tor
DataDirectory /var/lib/tor2
ControlPort 9151
/etc/tor/torrc3
SocksPort 9250 # what port to open for local application connections
SocksBindAddress 127.0.0.1 # accept connections only from localhost
AllowUnverifiedNodes middle,rendezvous
Log notice syslog
RunAsDaemon 1
User tor
Group tor
DataDirectory /var/lib/tor3
ControlPort 9251
/etc/tor/torrc4
SocksPort 9350 # what port to open for local application connections
SocksBindAddress 127.0.0.1 # accept connections only from localhost
AllowUnverifiedNodes middle,rendezvous
Log notice syslog
RunAsDaemon 1
User tor
Group tor
DataDirectory /var/lib/tor4
ControlPort 9351
/etc/tor/torrc5
SocksPort 9450 # what port to open for local application connections
SocksBindAddress 127.0.0.1 # accept connections only from localhost
AllowUnverifiedNodes middle,rendezvous
Log notice syslog
RunAsDaemon 1
User tor
Group tor
DataDirectory /var/lib/tor5
ControlPort 9451
/etc/tor/torrc6
SocksPort 9550 # what port to open for local application connections
SocksBindAddress 127.0.0.1 # accept connections only from localhost
AllowUnverifiedNodes middle,rendezvous
Log notice syslog
RunAsDaemon 1
User tor
Group tor
DataDirectory /var/lib/tor6
ControlPort 9551
/etc/tor/torrc7
SocksPort 9650 # what port to open for local application connections
SocksBindAddress 127.0.0.1 # accept connections only from localhost
AllowUnverifiedNodes middle,rendezvous
Log notice syslog
RunAsDaemon 1
User tor
Group tor
DataDirectory /var/lib/tor7
ControlPort 9651
/etc/tor/torrc8
SocksPort 9750 # what port to open for local application connections
SocksBindAddress 127.0.0.1 # accept connections only from localhost
AllowUnverifiedNodes middle,rendezvous
Log notice syslog
RunAsDaemon 1
User tor
Group tor
DataDirectory /var/lib/tor8
ControlPort 9751

 

Configuration files for privoxy

/etc/privoxy/config
trust-info-url  http://www.example.com/why_we_block.html
trust-info-url  http://www.example.com/what_we_allow.html
confdir /etc/privoxy
logdir /var/log/privoxy
actionsfile standard.action  # Internal purpose, recommended
actionsfile default.action   # Main actions file
actionsfile user.action      # User customizations
filterfile default.filter
logfile logfile
listen-address  127.0.0.1:8118
toggle  1
enable-remote-toggle  0
enable-remote-http-toggle  0
enable-edit-actions 0
enforce-blocks 0
buffer-limit 4096
forward-socks4a   /               127.0.0.1:9050 .
forwarded-connect-retries  0
accept-intercepted-requests 0
allow-cgi-request-crunching 0
split-large-forms 0
/etc/privoxy/config2
trust-info-url  http://www.example.com/why_we_block.html
trust-info-url  http://www.example.com/what_we_allow.html
confdir /etc/privoxy
logdir /var/log/privoxy2
actionsfile standard.action  # Internal purpose, recommended
actionsfile default.action   # Main actions file
actionsfile user.action      # User customizations
filterfile default.filter
logfile logfile
listen-address  127.0.0.1:8129
toggle  1
enable-remote-toggle  0
enable-remote-http-toggle  0
enable-edit-actions 0
enforce-blocks 0
buffer-limit 4096
forward-socks4a   /               127.0.0.1:9150 .
forwarded-connect-retries  0
accept-intercepted-requests 0
allow-cgi-request-crunching 0
split-large-forms 0
/etc/privoxy/config3
trust-info-url  http://www.example.com/why_we_block.html
trust-info-url  http://www.example.com/what_we_allow.html
confdir /etc/privoxy
logdir /var/log/privoxy3
actionsfile standard.action  # Internal purpose, recommended
actionsfile default.action   # Main actions file
actionsfile user.action      # User customizations
filterfile default.filter
logfile logfile
listen-address  127.0.0.1:8230
toggle  1
enable-remote-toggle  0
enable-remote-http-toggle  0
enable-edit-actions 0
enforce-blocks 0
buffer-limit 4096
forward-socks4a   /               127.0.0.1:9250 .
forwarded-connect-retries  0
accept-intercepted-requests 0
allow-cgi-request-crunching 0
split-large-forms 0
/etc/privoxy/config4
trust-info-url  http://www.example.com/why_we_block.html
trust-info-url  http://www.example.com/what_we_allow.html
confdir /etc/privoxy
logdir /var/log/privoxy4
actionsfile standard.action  # Internal purpose, recommended
actionsfile default.action   # Main actions file
actionsfile user.action      # User customizations
filterfile default.filter
logfile logfile
listen-address  127.0.0.1:8321
toggle  1
enable-remote-toggle  0
enable-remote-http-toggle  0
enable-edit-actions 0
enforce-blocks 0
buffer-limit 4096
forward-socks4a   /               127.0.0.1:9350 .
forwarded-connect-retries  0
accept-intercepted-requests 0
allow-cgi-request-crunching 0
split-large-forms 0
/etc/privoxy/config5
trust-info-url  http://www.example.com/why_we_block.html
trust-info-url  http://www.example.com/what_we_allow.html
confdir /etc/privoxy
logdir /var/log/privoxy5
actionsfile standard.action  # Internal purpose, recommended
actionsfile default.action   # Main actions file
actionsfile user.action      # User customizations
filterfile default.filter
logfile logfile
listen-address  127.0.0.1:8421
toggle  1
enable-remote-toggle  0
enable-remote-http-toggle  0
enable-edit-actions 0
enforce-blocks 0
buffer-limit 4096
forward-socks4a   /               127.0.0.1:9450 .
forwarded-connect-retries  0
accept-intercepted-requests 0
allow-cgi-request-crunching 0
split-large-forms 0
/etc/privoxy/config6
trust-info-url  http://www.example.com/why_we_block.html
trust-info-url  http://www.example.com/what_we_allow.html
confdir /etc/privoxy
logdir /var/log/privoxy6
actionsfile standard.action  # Internal purpose, recommended
actionsfile default.action   # Main actions file
actionsfile user.action      # User customizations
filterfile default.filter
logfile logfile
listen-address  127.0.0.1:8522
toggle  1
enable-remote-toggle  0
enable-remote-http-toggle  0
enable-edit-actions 0
enforce-blocks 0
buffer-limit 4096
forward-socks4a   /               127.0.0.1:9550 .
forwarded-connect-retries  0
accept-intercepted-requests 0
allow-cgi-request-crunching 0
split-large-forms 0
/etc/privoxy/config7
trust-info-url  http://www.example.com/why_we_block.html
trust-info-url  http://www.example.com/what_we_allow.html
confdir /etc/privoxy
logdir /var/log/privoxy7
actionsfile standard.action  # Internal purpose, recommended
actionsfile default.action   # Main actions file
actionsfile user.action      # User customizations
filterfile default.filter
logfile logfile
listen-address  127.0.0.1:8623
toggle  1
enable-remote-toggle  0
enable-remote-http-toggle  0
enable-edit-actions 0
enforce-blocks 0
buffer-limit 4096
forward-socks4a   /               127.0.0.1:9650 .
forwarded-connect-retries  0
accept-intercepted-requests 0
allow-cgi-request-crunching 0
split-large-forms 0
/etc/privoxy/config8
trust-info-url  http://www.example.com/why_we_block.html
trust-info-url  http://www.example.com/what_we_allow.html
confdir /etc/privoxy
logdir /var/log/privoxy8
actionsfile standard.action  # Internal purpose, recommended
actionsfile default.action   # Main actions file
actionsfile user.action      # User customizations
filterfile default.filter
logfile logfile
listen-address  127.0.0.1:8724
toggle  1
enable-remote-toggle  0
enable-remote-http-toggle  0
enable-edit-actions 0
enforce-blocks 0
buffer-limit 4096
forward-socks4a   /               127.0.0.1:9750 .
forwarded-connect-retries  0
accept-intercepted-requests 0
allow-cgi-request-crunching 0
split-large-forms 0

 

Startup scripts

Privoxy: /etc/rc.d/privoxy (Arch linux)

#!/bin/bash
# source application-specific settings
[ -f /etc/conf.d/privoxy ] && . /etc/conf.d/privoxy
. /etc/rc.conf
. /etc/rc.d/functions
#PID=`pidof -o %PPID /usr/sbin/privoxy`
start() {
  stat_busy "Starting Privoxy, squid, havp"
  chown privoxy:privoxy /var/log/privoxy
  # create missing logfiles
  for i in logfile jarfile; do 
      touch /var/log/privoxy/$i
      chgrp $PRIVOXY_GROUP /var/log/privoxy/$i
      chmod 660 /var/log/privoxy/$i
  done
  chown privoxy:privoxy /var/log/privoxy2
  # create missing logfiles
  for i in logfile jarfile; do 
      touch /var/log/privoxy2/$i
      chgrp $PRIVOXY_GROUP /var/log/privoxy2/$i
      chmod 660 /var/log/privoxy2/$i
  done
  chown privoxy:privoxy /var/log/privoxy3
  # create missing logfiles
  for i in logfile jarfile; do 
      touch /var/log/privoxy3/$i
      chgrp $PRIVOXY_GROUP /var/log/privoxy3/$i
      chmod 660 /var/log/privoxy3/$i
  done
chown privoxy:privoxy /var/log/privoxy4
  # create missing logfiles
  for i in logfile jarfile; do 
      touch /var/log/privoxy4/$i
      chgrp $PRIVOXY_GROUP /var/log/privoxy4/$i
      chmod 660 /var/log/privoxy4/$i
  done
   chown privoxy:privoxy /var/log/privoxy5
  # create missing logfiles
  for i in logfile jarfile; do 
      touch /var/log/privoxy5/$i
      chgrp $PRIVOXY_GROUP /var/log/privoxy5/$i
      chmod 660 /var/log/privoxy5/$i
  done
  
    chown privoxy:privoxy /var/log/privoxy6
  # create missing logfiles
  for i in logfile jarfile; do 
      touch /var/log/privoxy6/$i
      chgrp $PRIVOXY_GROUP /var/log/privoxy6/$i
      chmod 660 /var/log/privoxy6/$i
  done
  chown privoxy:privoxy /var/log/privoxy7
  # create missing logfiles
  for i in logfile jarfile; do 
      touch /var/log/privoxy7/$i
      chgrp $PRIVOXY_GROUP /var/log/privoxy7/$i
      chmod 660 /var/log/privoxy7/$i
  done
    chown privoxy:privoxy /var/log/privoxy8
  # create missing logfiles
  for i in logfile jarfile; do 
      touch /var/log/privoxy8/$i
      chgrp $PRIVOXY_GROUP /var/log/privoxy8/$i
      chmod 660 /var/log/privoxy8/$i
  done
#[ -z "$PID" ] && /usr/sbin/privoxy $PRIVOXY_ARGS 2>/dev/null
  /usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy.pid /etc/privoxy/config
  /usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy2.pid /etc/privoxy/config2
  /usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy3.pid /etc/privoxy/config3
  /usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy4.pid /etc/privoxy/config4
  /usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy5.pid /etc/privoxy/config5
  /usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy6.pid /etc/privoxy/config6
  /usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy7.pid /etc/privoxy/config7
  /usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy8.pid /etc/privoxy/config8
  havp -c /etc/havp/havp.config  
  
  squid -f /etc/squid/squid-in.conf -DYC
  squid -f /etc/squid/squid-out.conf -DYC
  
  if [ $? -gt 0 ]; then
    stat_fail
  else
    add_daemon privoxy
    stat_done
  fi
}
stop() {
  stat_busy "Stopping Privoxy, Squid, Havp"
  #[ ! -z "$PID" ]  && kill $PID &> /dev/null
  killall privoxy
  killall squid
  killall havp
  
  if [ $? -gt 0 ]; then
    stat_fail
  else
    rm_daemon privoxy
    stat_done
  fi
}
case "$1" in
  start)
    start
    ;;
  stop)
    stop
    ;;
  restart)
    $0 stop
    sleep 1
    $0 start
    ;;
  *)
    echo "usage: $0 {start|stop|restart}"  
esac
exit 0

 

Tor startup script: /etc/rc.d/tor (Arch linux)

#!/bin/bash
. /etc/rc.conf
. /etc/rc.d/functions
#PID=`pidof -o %PPID /usr/bin/tor`
tor='/usr/bin/tor'
case "$1" in
        start)
                stat_busy "Starting Tor Daemon"
                #[ -z "$PID" ] && /usr/bin/tor &>/dev/null
                ${tor} -f /etc/tor/torrc
                ${tor} -f /etc/tor/torrc2
                ${tor} -f /etc/tor/torrc3
                ${tor} -f /etc/tor/torrc4
                ${tor} -f /etc/tor/torrc5
                ${tor} -f /etc/tor/torrc6
                ${tor} -f /etc/tor/torrc7
                ${tor} -f /etc/tor/torrc8
    
                if [ $? -gt 0 ]; then
                        stat_fail
                else
                        add_daemon tor
                        stat_done
                fi
                ;;
        stop)
                stat_busy "Stopping Tor Daemon"
                #[ ! -z "$PID" ] && kill $PID &> /dev/null
                killall tor
                if [ $? -gt 0 ]; then
                        stat_fail
                else
                        rm_daemon tor
                        stat_done
                fi
                ;;
		restart)
                $0 stop
                sleep 3
                $0 start
                ;;
        *)
                echo "usage: $0 {start|stop|restart}"
esac
exit 0
# vim: ft=sh ts=2 sw=2

 

Squid-in config

/etc/squid/squid-in.conf
acl all src all
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl Safe_ports port 901         # SWAT
acl purge method PURGE
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
acl malware_domains url_regex '/etc/squid/Malware-domains.txt'
http_access deny malware_domains
http_access allow localhost
http_access deny all
icp_access deny all
http_port 3400
icp_port 0
hierarchy_stoplist cgi-bin ?
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
refresh_pattern .               0       20%     4320
cache_peer localhost parent 8118 0 round-robin no-query
cache_peer localhost2 parent 8129 0 round-robin no-query
cache_peer localhost3 parent 8230 0 round-robin no-query
cache_peer localhost4 parent 8321 0 round-robin no-query
cache_peer localhost5 parent 8421 0 round-robin no-query
cache_peer localhost6 parent 8522 0 round-robin no-query
cache_peer localhost7 parent 8623 0 round-robin no-query
cache_peer localhost8 parent 8724 0 round-robin no-query
never_direct allow all
always_direct deny all
acl apache rep_header Server ^Apache
broken_vary_encoding allow apache
forwarded_for off
coredump_dir /var/cache/squid-in
cache_dir ufs /var/cache/squid-in 100 16 256
pid_filename /var/run/squid-in.pid
access_log /var/log/squid/access.squid-in.log
cache_store_log /var/log/squid/store.squid-in.log
cache_log /var/log/squid/cache.squid-in.log

 

Squid-out config

/etc/squid/squid-out.conf
acl all src all
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8
acl localnet src 10.0.0.0/8     # RFC1918 possible internal network
acl localnet src 172.16.0.0/12  # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl Safe_ports port 901         # SWAT
acl purge method PURGE
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localnet
http_access allow localhost
http_access deny all
icp_access deny all
http_port 3128
icp_port 0
hierarchy_stoplist cgi-bin ?
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
refresh_pattern .               0       20%     4320
cache_peer localhost parent 3410 0 round-robin no-query
never_direct allow all
always_direct deny all
acl apache rep_header Server ^Apache
broken_vary_encoding allow apache
forwarded_for off
coredump_dir /var/cache/squid-out
cache_dir ufs /var/cache/squid-out 100 16 256
pid_filename /var/run/squid-out.pid
access_log /var/log/squid/access.squid-out.log
cache_store_log /var/log/squid/store.squid-out.log
cache_log /var/log/squid/cache.squid-out.log

 

Havp antivirus proxy config

/etc/havp/havp.config
USER havp
GROUP havp
DAEMON true
PIDFILE /tmp/havp.pid
SERVERNUMBER 4
MAXSERVERS 256
SCANTEMPFILE /tmp/havp-XXXXXX
TEMPDIR /tmp
DBRELOAD 60 
PARENTPROXY localhost
PARENTPORT 3400
FORWARDED_IP false
X_FORWARDED_FOR false
PORT 3410
BIND_ADDRESS 127.0.0.1
ENABLECLAMLIB true
ENABLECLAMD false
ENABLEFPROT false
ENABLEAVG false
ENABLEAVESERVER false
ENABLESOPHIE false
ENABLETROPHIE false
ENABLENOD32 false
ENABLEAVAST false
ENABLEARCAVIR false
ENABLEDRWEB false

 

Domain List updater script (can be launched from cron)

#!/usr/bin/python
# -*- coding: utf-8 -*-
# Author: Bohdan Turkynewych, AKA Gh0st, 2006-2008, tb0hdan[at]gmail[dot]com
# Distributed under the terms of GPLv2
# Parts of code from neutron jabber bot project, http://code.google.com/p/neutron/
from urllib2 import Request as urllib2_Request, urlopen as urllib2_urlopen
from sys import exit as sys_exit
from re import search as re_search
from time import ctime as time_ctime, time as time_time
from re import compile as re_compile
def t_conv(timestamp):
        reply = ''
        seconds = timestamp % 60
        minutes = (timestamp / 60) % 60
        hours = (timestamp / 3600) % 60
        days = timestamp / 216000
        if days: reply += str(days) + ' day(s) '
        if hours: reply += str(hours) + ' hour(s) '
        if minutes: reply += str(minutes) + ' minute(s) '
        reply += str(seconds) + ' second(s)'
        return reply
    
def uptime():
        global BOOTUP_TIMESTAMP
        if BOOTUP_TIMESTAMP:
                idletime = int(time_time() - BOOTUP_TIMESTAMP)
                reply = t_conv(idletime)
        else:
                reply = 'Unknown'
        return reply
strip_tags = re_compile(r'<[^]+>')
def decode(text):
    data = text.replace('<br>','\n').replace(' ', ' ').replace('<', '<').replace('>', '>').replace('"', '"').replace('<br>','\n').replace('
<li>','\r\n')
    return strip_tags.sub('', data)
def get_data(message):
    reply=[]
    try:
        for line in message.split('\n'):
            if re_search('wbr',line):
                #
                od = re_search('',line)
                d0main = line[od.end():]
                d0main = d0main[:re_search('',d0main).start()].strip()
                #
                if d0main == '-':
                        od = re_search('-',line)
                        bookurl = line[od.end():]
                        bookurl = bookurl[:re_search('',bookurl).start()].strip()
                        bookurl = decode(bookurl)
                else:
                    bookurl = decode(d0main)
                d0main = ''
                stripper = '[/|:]'
                if re_search(stripper,bookurl):
                      #d0main = bookurl
                     od = re_search(stripper,bookurl)
                     d0main = bookurl[od.end():]
                     d0main = bookurl[:re_search(stripper,bookurl).start()].strip()
                else:
                     d0main = bookurl
                if not d0main in reply:
                    reply.append(d0main)
        return reply
    except:
         raise
def update_list(filename):
    req2 = urllib2_Request('http://www.malwaredomainlist.com/mdl.php?sort=Domain&ascordesc=DESC&search=&colsearch=All&quantity=All')
    req2.add_header = ('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.9) Gecko/20071030 SeaMonkey/1.1.6')
    print '[+] Update process started at ' + time_ctime()
    try:
        r = urllib2_urlopen(req2)
        target = r.read()
        urls = get_data(target)
    except:
        raise
        print '[-] Error occured while fetching / parsing data.'
        sys_exit(1)
    print '[+] Got ' + str(len(urls)) + ' domains/ips.'
    try:
        file = open(filename,'w')
        for line in urls:
            file.write(line + '\n')
        file.close()
    except:
        print '[-] Error occured while flushing data to file'
        sys_exit(5)
    elapsed = uptime()
    print '[+] Update process finished at ' + time_ctime() + '. Took ' + elapsed
BOOTUP_TIMESTAMP = time_time()
update_list('/etc/squid/Malware-domains.txt')

Have fun!

P.S. Future optimisation requires 8 havp instances as well due to slowup.

P.S.S You may exclude havp and squid-out proxy from your system, and use only squid-in. That will be much, much faster.

Share this page:

Suggested articles

31 Comment(s)

Add comment

Comments

By: Bohdan Turkynewych

Sorry about that, I've forgot to add /etc/hosts file in the article...

That is how i fooled squid.

 

Here is the file:

 

127.0.0.1  localhost

127.0.0.1 localhost2

127.0.0.1 localhost3 

127.0.0.1 localhost4

127.0.0.1 localhost5

127.0.0.1 localhost6

127.0.0.1 localhost7

127.0.0.1 localhost8

By: Pete Ashdown

Yes, but you can run HAProxy off of Privoxy and SOCKS quite well.  I've cut out squid entirely for a client server.  Here is a config:

global
    maxconn 4096
    ulimit-n 65536
    quiet
    daemon
    nbproc 2
    user haproxy
    group haproxy

defaults
    retries    3
    redispatch
    maxconn    2000
    contimeout    5000
    clitimeout    50000
    srvtimeout    50000

listen privoxytor :8118,:443
    mode tcp
    balance roundrobin

    server privoxy 127.0.0.1:8119
    server privoxy1 127.0.0.1:8129
    server privoxy2 127.0.0.1:8230
    server privoxy3 127.0.0.1:8321
    server privoxy4 127.0.0.1:8421
    server privoxy5 127.0.0.1:8522
    server privoxy6 127.0.0.1:8623
    server privoxy7 127.0.0.1:8724
    server privoxy8 127.0.0.1:8825

listen socks :9050
    mode tcp
    balance roundrobin

    server tor 127.0.0.1:9052
    server tor1 127.0.0.1:9150
    server tor2 127.0.0.1:9250
    server tor3 127.0.0.1:9350
    server tor4 127.0.0.1:9450
    server tor5 127.0.0.1:9550
    server tor6 127.0.0.1:9650
    server tor7 127.0.0.1:9750
    server tor8 127.0.0.1:9850

By:

Interesting idea. Thanks for the good read!

By: Woods

Very interesting indeed. A good privacy service is a MUST have on todays internet.Jiff

www.online-anonymity.kr.tc 

By:

let me thank you for that interesting and intriguing idea.

By: syniack

nice tutorials thanks dude for sharing such a nice tutorial.

By: Anonymous

Damn, forgot the picture to make it somewhat clearer:

 http://img300.imageshack.us/img300/636/torprivoxysquid.png

By: Anonymous

Well I have some questions for my setup.

Internet ? Router ? FreeBSD ? Local machines

FreeBSD currently contains only Squid. But i would love to include tor as shown in the tutorial above. Currently I am having all local users to have the HTTP proxy set up in browsers to use squid on FreeBSD port 3128 ( default by squid ).

If I add tor to it, how the users connect to this server ?  What port and configurations required for browser / client. How the load of single client is handled by the network of tor, as you mentioned to disable the squid

Thanks

 

By: Anonymous

First, let me thank you for that interesting and intriguing idea. Tried it in a VM and came across some problems, so i thought i'd share the solutions and improvements with the rest of you. I'm sorry for any grammar-related fuck-ups, as i'm a non-native english speaker.

Software used: Arch i686 2.6.29-3, Tor v0.2.0.34 r18423, Privoxy 3.0.12, Squid 2.7 STABLE6

I only used Squid-IN as caching proxy collecting from the 8 Privoxy instances. Next step will be inlcuding havp into the chain.

1. Tor config

- The "Group" option is deprecated an no longer needed.
- What exactly is the ControlPort for? I didn't see it used anywhere, just Tor throwing a warning about the ControlPort used without authorization. I simply deleted the line, everything still works fine.
- Once everything runs smoothly, logging isn't really needed any longer, is it? Same goes for the other log files.

2. Privoxy config

- The "standard.action" is now called "match-all.action". Privoxy won't start with "standard.action"

3. Privoxy startup script

- The "$PRIVOXY_ARGS" when starting the Privoxy instances needs to be removed. Privoxy won't start, but without them they work as expected.
- For testing purposes and as i'm only using one squid, i deleted the havp and squid lines and used the existing /etc/rc.d/squid, so every chain link can be started/stopped seperately.

4. Squid config

I had a problem getting your config up and running, haven't found out exactly where it hangs. I edited the squid.conf.default to mainly do the same.

- "cache_peer localhost2 parent 8129 0 round-robin no-query" won't work, as squid understandably can't resolve "localhost2".

Instead use:

"cache_peer localhost parent 8129 0 round-robin no-query name=localhost2" and corresponding, which will work just fine.

Problem with the first line: it works, but only the first entry (which correctly reads "localhost" as host), so it doesn't really improve anonymity with only one Tor-session effectively used.

Using this config i got the whole chain up and running quite stable. Btw, i'm using encryption for the VM-disk, one never knows when it may come in handy. Hope this helps someone who tried it and ran into the same problems.

By: Anonymous

Or nginx.

By: Anonymous

Tor is fine as far as security, but it's so slow, you'll give up really fast. I know 'cause I've tried it. There's also a Firefox addon for Tor.

By: HeroicLife

It would be great if someone provided this setup as a virtual machine.  I'd like to be able to fire it up in VMware and just set up a proxy in Firefox.

By: Anonymous

Yes, someone, please do this! also add that the DNS requests go to OpenDNS. (suggested by poster below)

By: Anonymous

Here here! That would be great, or if you could make a windows installer instead!!!!! PIMPIN! I would totally settle for a VM! If anybody can make one and megaupload it or something Id be so grateful!

By: crazyfag

Its really great to have a Megaupload automatic filler, thanks!

By: Anonymous

dont forget, your ISP can record your DNS lookups. so be sure to route your DNS to keep it private.

By: David Nielsen

This seems like the perfect use case for appliances.

You should easily be able to make one or several of the required components along with configuration using one of the many appliance tools Linux now offers.

http://www.rpath.com/rbuilder/

http://studio.suse.com

http://fedoraproject.org/wiki/CustomSpins

This would allow you to post a working set of images to go along with the tutorial which have all the required packages and configuration. SuSE Studio here is especially smart since it lets you boot the appliance and customize the configuration before making the final images. This would also add the additional security of encapulating everything so no unneeded components were installed and it would allow you to run everything virtualized.

By: baby

Maybe you could also give tips for n00bs..

Your idea sounds great.

By: Anonymous

Why not just give torvm to every user for their browsing safety? The malware_domains.txt contains what? And, that's an old version of tor, it appears 0.2.0.31 is current.

By: Yaminashi

This is great for noobs that need to get on websites at work or at school, everybody is looking for proxys and this is a great way to get on websites.

By: Anonymous

Set one up in a virtual machine using Debian, but surfing is still quite slow. Any way to speed things up? The problem appears to be latency, not bandwidth.

By: Nick

Hi There,

Seems like a bit of  resource waste running a privoxy process for each tor listener, I would suggest load balancing tor with haproxy would be a less intensive approach :)

 an example haproxy config would be something like.....

listen tor :9100
        mode tcp
        option tcplog
        balance roundrobin

        server tor1 127.0.0.1:9150 check
        server tor2 127.0.0.1:9250 check
        server tor3 127.0.0.1:9350 check
        server tor4 127.0.0.1:9450 check
        server tor5 127.0.0.1:9550 check

By: Aries

Because tor is not a http proxy.

By: Bill

It's a lot easier to just use web proxies  here's a few working ones

Proxy Bypass
Web Anonymous
Hidden Proxy
Web Proxies
IP Hide
Internet Bypass
How to Bypass

By: Anonymous

I am kind of surprised this hasn't been built into an image.  Back in High School I was pretty into linux and looking at this I imagine that one day I would have been able to understand it and implement it fairly easily.  However, those days have passed, and now I am sitting here scratching my head, praying that someone would be kind enough to blow this into an image and post it somewhere...

 Alas...

By: tb0hdan

Hello, All

I'm happy to announce that this article was finally used to create a loosely based LiveCD with most features, described here.

 Source code (GPLv3) is available here: http://github.com/tb0hdan/4n0n

And there's an ISO, too: http://4n0n.org.ua/4n0n-6.1.2.iso

 This can be interesting for linux developers, bash scripters,

and security professionals.

Have phun!

By: Anonymous

Hi I am using ubuntu 11.04

 The domain list updator script would not work. it spits out errors about the decode function. appears there are syntax errors in the script.

 Thank You.

By: us&

prob with squid3,i'm on linux mint & i have /etc/rc0d rc1d etc...

pls a tuto for mint :)

By: chris neumaier

Is there a tutorial to make this work on ubuntu? 

maybe only the scripts and configs need adaption. Does anybody know about this ?

 

By: renne

Since TOR 2.4.0 you can just use multiple SOCKSPort directives in torrc to get multiple SOCKS-ports with separate virtual circuits. ;)

By: Anonymous

very impressive stuff, im gonna try that sometime soon, very interesting read..