Heads up! Long post and lots of head bashing against the wall.
Context:
I have a written an a python app (Django). I have dockerized the deployment and the compose file has three containers, app, nginx and postgres. I'm currently trying to deploy a demo of it in a VPS running Debian 11. Information below has been redacted (IPs, Domain name, etc.)
Problem:
I keep running into 502 errors. Locally things work very well even with nginx (but running on 80). As I try to deploy this I'm trying to configure nginx the best I can and redirecting http traffic to https and ssl certs. The nginx logs simply say "connect() failed (111: Connection refused) while connecting to upstream, client: 1.2.3.4, server: demo.example.com, request: "GET / HTTP/1.1", upstream: "http://192.168.0.2:8020/", host: "demo.example.com"". I have tried just about everything.
What I've tried:
Adding my server block configs to /etc/nginx/conf.d/default.conf
Adding my server block configs to a new file in /etc/nginx/conf.d/app.conf and leaving default at out of box config.
Tried putting the above config (default.conf and app.conf) in sites-available (/etc/nginx/sites-available/* not at the same time tho).
Recreated /etc/nginx/nginx.conf by copy/pasting out of box nginx.conf and then adding server blocks directly in nginx.conf
Running nginx -t inside of the nginx container (Syntax and config were "successful")
Running nginx -T when recreated /etc/nginx/nginx.conf
nginx -T when the server blocks where in /etc/nginx/conf.d/* lead me to think that since there were two server listen 80 blocks that I should ensure only one listen 80 block was being read by the container hence the recreated /etc/nginx/nginx.conf from above
Restarted container each time a change was made.
Changed the user block from nginx (no dice when using nginx as user) to www-data, root and nobody
Deleted my entire docker data and redeployed everything a few times.
Double checked the upstream block 1,000 times
Confirmed upstream block container is running and on the right exposed port
Checked access.log and error.log but they were both empty (not sure why, tried cat and tail)
Probably forgetting more stuff (6 hours deep in the same error loop by now)
How can you help:
Please take a look at the nginx.conf config below and see if you guys can spot a problem, PLEASE! This is my current /etc/nginx/nginx.conf
EDIT: I have also confirmed that both containers are connected to the same docker network (docker network inspect frontend)
EDIT 2: Solved my problem. See my comments to @chaospatterns. TLDR there was an uncaught exception in the app but it didn’t cause a crash with the container. Had to dig deep into logs to find it.
Assume nothing! Test every little assumption and you'll find the problem. Some things to get you started:
Does the "app" domain resolve to the app container's IP from within the nginx container?
Can you proxy_pass to the host:port directly rather than using an upstream definition? If not, what about IP:port?
Can you connect to the app container from outside (if exposed)? What about from inside the nginx container? What about inside the app container?
Is the http(s) connection to the server (demo.example.com) actually going to your nginx instance? Shut it down and see if it changes.
If it works locally on 80, can you get it to work on the VPS on 80?
Are you using the exact same docker-compose.yaml file for this as locally? If not, what's different?
Are you building the image? If so, are you incrementing the version number of the build so it gets updated?
Is there a firewall running on the host OS? If so, is it somehow interfering? Disable it and see.
While not a direct solution to your problem, I no longer manually configure my reverse proxies at all now and use auto-configuring ones instead. The nginx-proxy image is great, along with it's ACME companion image for automatic SSL cert generation with certbot - you'll be up and running in under 30 mins. I used that for a long time and it was great.
I've since moved to using Traefik as it's more flexible and offers more features, but it's a bit more involved to configure (simple, but the additional flexibility means everything requires more config).
That way you just bring up your container and the reverse proxy pulls meta-data from it (e.g. host to map/certbot email) and off it goes.
Great pointers! Some of them I had done and triple checked and other are great future troubleshooting points. There was no way I was going to put hours of troubleshooting and checking on a post so I tried to provide as much information as possible without putting up a giant wall of text.