nginx reverse proxy with multiple servers

Discussion in 'Server Operation' started by mesiah, Jan 16, 2020.

  1. mesiah

    mesiah New Member

    renewed, see next entry please
     
    Last edited: Mar 23, 2020
  2. Adnan Khaleel

    Adnan Khaleel New Member

    Hi Mesiah,

    Thank you very much for posting this, I've been trying to do something similar with not much success so your post definitely helps.
    Have a few questions though:
    If I already have IP address for my backend servers, for eg:
    (webserver1.domain.com --> 192.168.0.10, webserver2.domain.com --> 192.168.0.11, webserver3.domain.com --> 192.168.0.12)
    What would I change in the "passthru" config above?

    Also, what does the following do? Is this an internal DNS? And assuming I don't have an internal DNS, what should this be?
    upstream support_backend {
    server 192.168.0.1:443; # or DNS Name
    }
    upstream intranet_backend {
    server 192.168.0.2:443; # or DNS Name
    }

    Thanks,
     
  3. mesiah

    mesiah New Member

    Hello Adnan

    I have now a far better structure for your reverse Proxy. Please follow the tutorial shown below as I'm starting from the beginnig:

    I was in a need to place several webservers behind a reverse proxy. But without encrypt and reencrypt and just pass unencrypted traffic from the reverse proxy. So this Tutorial may help for some people.
    issue:
    1. Several Webservers with various applications on each are running behind a FW and responding only on Port 443
    2. The Webservers have a wildcard Certificate, they are IIS Webservers(whoooho very brave), and have public IP addresses on each webserver
    3. It is requested, that all webserver should not be exposed to the Internet and moved to a DMZ
    4. Since IP4 addresses are short these days, it is not possible get more IPs addresses
    5. Nginx should only passthrough the requests. No Certificate break, decrypt, re-encrypt between webserver and reverse proxy or whatsoever.
    Solution:
    1. All websservers should be moved to a "internal" DMZ
    2. A single nginx reverse proxy should handle all requests based on the webservers DNS entries and map them. This will make the public IP4 address needs obsolete. The reverse proxy could be placed on external DMZ
    3. All webservers would get a private IP
    4. A wild certificate would be just fine to handle all aliases for DNS forwarding.
    Steps to be done:

    1. A single nginx RP should be placed on the external-DMZ.

    2. Install nginx: - Install nginx on a fully patched debian with apt-get install nginx. At this Point you'll get Version 1.14 for nginx. Of course you may compile it too

    1. If you have installed nginx by the apt-get way, it will be configured with the following modules, which you will need later: ngx_stream_ssl_preread, ngx_stream_map, and stream. Don't worry, they are already in the package. You may check with nginx -V
    4. external DNS Configuration: - all DNS request from the Internet should point the nginx.

    E.g
    webserver1.domain.com --> nginx
    webserver2.domain.com --> nginx
    webserver3.domain.com --> nginx
    webserver4.domain.com --> nginx


    5. nginx configurations:
    • Create following folders:
      • /etc/nginx/stream/available
      • /etc/nginx/stream/enabled
    • Now we need three portions of conf file placed into ./available
    a) The mappings file

    00_prefix.conf

    Code:
     
    ## this where the mappings are made to each server and port. Everytime you place a new wsebserver, you need to update this file:
    ## mapping of servers to upstream servers ###
    ## In this section, we have to pair frontend and backend servers
    #  everytime we want have them reverse proxied
    ## e.g. <relaserver.domainname>:port   <backendservervariable_port>
    
     map $ssl_preread_server_name:$server_port $upstream {
     #######
     webserver01.domain.com:443 realserver01_backend_443;
     #######
     webserver02.domain.com:443 realserver02_backend_443;
     #######
     webserver03.domain.com:443 realserver03_backend_443;
     #######
     webserver04.domain.com:443 realserver04_backend_443;
     webserver04.domain.com:9098 realserver04_backend_9098;  ### This is some other Port in case you need it
    }


    b) Now the Ports conf file.

    99_suffix.conf

    Code:
    server {
      listen 443;
      proxy_pass $upstream;
      ssl_preread on;
    }


    c) At least the conf file for each upstream

    01_webserver01.domain.com.conf

    Code:
    # 01_webserver01.domain.com.conf
    upstream  realserver02_backend_443 {
      hash $remote_addr consistent;
      server 192.168.1.1:443;  # -> this is the real backend IP of the webserver
    }
    02_webserver02.domain.com.conf

    Code:
    # 02_webserver02.domain.com.conf
    upstream  realserver02_backend_443 {
      hash $remote_addr consistent;
      server 192.168.1.2:443;  # -> this is the real backend IP of the webserver
    }
    03_webserver03.domain.com.conf
    Code:
    # 03_webserver02.domain.com.conf
    upstream  realserver03_backend_443 {
      hash $remote_addr consistent;
      server 192.168.1.3:443;  # -> this is the real backend IP of the webserver
    }

    04_webserver04.domain.com.conf

    Code:
    upstream realserver04_backend_443 {
      hash $remote_addr consistent;
      server 192.168.1.4:443;  # -> this is the real backend IP of the webserver
    }
    
    server {
      listen 9098;  # -> this is the  extra Port if you have other services on the same webserver
      proxy_pass $upstream;
      ssl_preread on;
    }


    The last config is made for a server, which is running a different service extra. So the listen 9098 directive must be added as a additional entry into the 04_webserver04.domain.com.conf

    now make a symlink to the ./stream/enabled/ folder:

    ln -s /etc/nginx/stream/available/*.conf /etc/nginx/stream/enabled/


    6. Add a Port 80 forwarding to the realservers

    For each Port 80 running on the backend server you need to place a conf file here /etc/nginx/sites-available and symlink it to /etc/nginx/sites-enabled

    I show you one example:

    realserver01.conf

    Code:
    server {
      server_name webserver01.domain.com;
     listen 80;
    
      location ~ {
        proxy_pass_header Authorization;
        proxy_pass http://192.168.1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_buffering off;
        client_max_body_size 0;
        proxy_read_timeout 36000s;
        proxy_redirect off;
      }
    }

    7. Modify your nginx.conf


    Code:
    user www-data;
    
    worker_processes auto;
    pid /run/nginx.pid;
    include /etc/nginx/modules-enabled/*.conf;
    
    events {
            # worker_connections 768;
             worker_connections  1024;
             multi_accept on;
             use  epoll;
    }
    
    
    
    http {
            ##
            # Basic Settings
            ##
    
            sendfile on;
            tcp_nopush on;
            tcp_nodelay on;
            keepalive_timeout 65;
            types_hash_max_size 2048;
            server_tokens off;
    
            # server_names_hash_bucket_size 64;
            # server_name_in_redirect off;
    
            include /etc/nginx/mime.types;
            default_type application/octet-stream;
    
            ##
            # SSL Settings
            ##
    
            # ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
            # ssl_protocols TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
            # ssl_prefer_server_ciphers on;
    
            ##
            # Logging Settings
            ##
    
            #access_log /var/log/nginx/access.log;
            #error_log /var/log/nginx/error.log;
    
            ##
            # Gzip Settings
            ##
    
             gzip on;
             gzip_vary on;
             gzip_proxied any;
             gzip_comp_level 6;
            # gzip_buffers 16 8k;
            # gzip_http_version 1.1;
             gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
    
            ##
            # Virtual Host Configs
            ##
    
            include /etc/nginx/conf.d/*.conf;
            include /etc/nginx/sites-enabled/*;
    }
    
            stream {
    
     include /etc/nginx/stream/enabled/*.conf;
    
    }
    
    
    #mail {
    #       # See sample authentication script at:
    #       # http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
    #
    #       # auth_http localhost/auth.php;
    #       # pop3_capabilities "TOP" "USER";
    #       # imap_capabilities "IMAP4rev1" "UIDPLUS";
    #
    #       server {
    #               listen     localhost:110;
    #               protocol   pop3;
    #               proxy      on;
    #       }
    #
    #       server {
    #               listen     localhost:143;
    #               protocol   imap;
    #               proxy      on;
    #       }
    #}



    8. Place a optimize file for nginx into /etc/nginx/conf.d

    optimize.conf

    Code:
            ##
    
            # Advanced Settings
            ##
    
            client_body_buffer_size 10K;
            client_header_buffer_size 1k;
            client_max_body_size 1G;
            large_client_header_buffers 2 1k;
    
            ##
            # Buffering
            ##
    
            proxy_buffering    off;
            proxy_buffer_size  128k;
            proxy_buffers 100  128k;

    9. Unlink the default virtual webserver rm /etc/nginx/sites-enabled/default

    10. Test nginx

    nginx -t and nginx -T

    11. Reload nginx with systemctl reload nginx

    Now to your questions:

    1. Just follow the new tutorial. You have to configure the 00_prefix.conf and add additional upstream server files 0x_webserver0x.domain.conf everytime you have a new webserver.
    2. These are the IP's of your backends. BTW you can use the host file in nginx and map the IP's to the names. No need to have a DNS. Of course you can just use the IP's. But don't forget to change the IP's in the upstream, when you change your realservers IP's
    3. For sure you need a external DNS. Otherwise you cannot handle point 4

    Rgds

    Haydar






     
    Last edited: Apr 17, 2020
    ahrasis and Steini86 like this.
  4. Adnan Khaleel

    Adnan Khaleel New Member

    Hi Haydar,

    Thanks for really cleaning this up and for making it more comprehensive. I've implemented what you've suggested above and it works! Thank you!

    Had one more question. I want to install a Letsencrypt certificate for SSL. In your original text, you had stated:
    Solution.4 A wild certificate would be just fine to handle all aliases for DNS forwarding.
    So should I just do a wildcard certificate on the Nginx (i.e. a certificate for *.domain.com)? Or should I instead install the certificate on the backend server i.e certificate on webserver1.domain.com, and so on, on the other backend servers?

    Not sure what the pros and cons of either approach is, but I thought I'd ask since I'm not entirely clear of where the SSL request terminates. My preference would be to have the entire communication be encrypted from client to server.

    I also noticed that in the nginx.conf, all of the SSL directives are commented out. Don't we need this for SSL?
    ##
    # SSL Settings
    ##

    # ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
    # ssl_protocols TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
    # ssl_prefer_server_ciphers on;

    Adnan

    PS: Noticed a typo in Section 5 above:
    5. nginx configurations:
    • Create following folders:
      • /etc/nginx/stream/stream/available => /etc/nginx/stream/available
      • /etc/nginx/stream/stream/enabled => /etc/nginx/stream/enabled
     
  5. mesiah

    mesiah New Member

    Hello Adnan:

    Yes indeed all ssl directives are commented out. Why? Because I just wanted to act the nginx as a simple reverse forwarder. The SSL decryption must be done on the webservers themself. Of course the presentation of the ssl certificate must be done on the backend servers too. Why? Because of troubleshoot and security issues. I don't want to have the nginx change the decryption and encryption. So the web server admin should not blame me, if things might go wrong. Further I don't want to have the communication between nginx and web server unencrypted.

    Of course if you like the other way, like have nginx decrypt the communication and forward the requests un-encrypted or re-encrypted, sure you may do so.

    As for the wildcard certificate, yes you don't need a wild card certificate, every backend could have its own. My approach was to Implement the ssl certificate on the backend. However, it depends on your needs.

    This Tutorial was generated by an Project of mine. So I thought it might be useful for others too.

    Yes indeed everybody is configuring the nginx the way they like. An this was my way;-) So the configs depend on your needs.

    BTW, Thanks for the typo reminder, I've corrected this section now

    Rgds

    Haydar
     
    Last edited: Mar 23, 2020
    ahrasis likes this.
  6. Adnan Khaleel

    Adnan Khaleel New Member

    Thank you very much. This works perfectly and was exactly what I was looking for.
     
  7. Emile

    Emile New Member

    Hi Messiah,
    Thanks for the extensive tutotrial. I found another typo: 00.prefix.conf instead of 00_prefix.conf.
    I followed the tutorial, but when I test the nginx with sudo nginx -t I get the following error:
    nginx: [emerg] could not build map_hash, you should increase map_hash_bucket_size: 32
    nginx: configuration file /etc/nginx/nginx.conf test failed

    I tried to modify the nginx.conf to avoid the error, but without success. Can you point me in the right direction? Thanks in advance!
     
  8. mesiah

    mesiah New Member

  9. Emile

    Emile New Member

    Hey Mesiah,
    Thanks for your quick response!
    Of course I did google as well; my nginx.conf is a copy of yours above, extended with map_hash_bucket_size 32 just above the Basic settings but still the same error. I tried the google results as well but no success
     
  10. mesiah

    mesiah New Member

    Hello,

    ok, I would disable in the stream/enabled settings and disable the stream directive in the nginx.conf and test the conf. I would check the nginx.conf one by one
     
  11. Emile

    Emile New Member

    Hi Mesiah,
    I disabled the stream settings in the nginx.conf and now the test is successful. So there must be something wrong in the stream settings. I'll doublecheck them
     
  12. mesiah

    mesiah New Member

    BTW I would take out all the #### comments and activate each stream conf one by one, it might be a type or so
     
  13. Emile

    Emile New Member

    Hi Mesiah,
    I don't see a typo in the attached files. Can you please check them? (I renamed the suffix from .conf to .txt otherwise I cannot upload them)
     

    Attached Files:

    Last edited: Apr 17, 2020
  14. mesiah

    mesiah New Member

    Please try to remove the # and comments from all your files and where is 99_ files and your nginx.conf
     
  15. Emile

    Emile New Member

    I did :(. In the attachment is the nginx.conf. The 99_suffix.conf is exactly the same as your post
     

    Attached Files:

  16. mesiah

    mesiah New Member

    include /etc/nginx/stream/enabled/* is this folder existing ? Otherwise I don't see anything weird,

    BTW is your map and stream modul added to nginx installtion

    check it with nginx -V against ngx_stream_ssl_preread, ngx_stream_map, and stream.
     
  17. Emile

    Emile New Member

    That folder exists.
    In the attachment the output for nginx -V
    I'm pretty new to nginx so I don't understand the modules
     

    Attached Files:

  18. mesiah

    mesiah New Member

    these modules are missing:

    ngx_stream_ssl_preread, ngx_stream_map

    This is mandatory, which was explained in my tutorial
     
  19. Emile

    Emile New Member

    Ah now I see. Can you please explain how to install / activate the two missing modules?
     
  20. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Reading message #3, they should be installed if following the tutorial.
    If they are not, I would guess using apt-get install for them does the trick. If my guess is wrong, try Internet Search Engines with
    Code:
    installing ngx_stream_ssl_preread ngx_stream_map
     

Share This Page