Sysadmin Mega Thread

[oO.o]

Wonder if 45drives people helping with that

[Trooper_ish]

I saw it on the recent Lawrence systems video, and pretty sure Tom mentioned that.

[cdoublejj]

anyone setup lancache? i can’t tell if ubuntu server is blocking the ports or snapd is doing something weird with docker. but, dns doesn’t seem to work when i point dns to lancache dns

[nx2l]

yes… i made a how-to
with podman…

[cdoublejj]

@nx2l

yes… i made a how-to
with podman…

how long ago?

i disabled ufw and disabled systemd listening on port 53 as per lancahce common TSing. when ever i disabled DNS masq in dd-wrt and set to my ubuntu vm running docker as per my specified in my .env file and yet dns BREAKS, my machines can’t resolve hgtv.com or espn.com etc etc.

said machines can ping ubuntu vm, i do not see any ip conflicts

…what gives…? what am i not seeing and missing.

are there commands i can run to see if my lancache containers can do dnslookups to it’s dns server (8.8.8.8)? is my lancahce dns server starving for dns data?

[nx2l]

I wont be much help with ubuntu, sorry.

You can try to exec into the container to run commands.

Guide - steamcache on Centos7/podman

Preface: I write this guide from POV of the root user I wont cover OS install or drive/disk/filesystem setup I am using a file system mounted at /mnt/cached mkdir -p /mnt/cached/cache/{riot,origin,blizzard,steam}/{data,logs} I set selinux on that directory: semanage fcontext -a -t container_file_t ‘/mnt/cached(/.*)?’ The number of IP addresses you need to use can vary. If only doing steamcache: two Ip addresses should be enough Be sure to use whatever you desire for your upstream dns * Step one: install Centos7 (minimal) Step two: setup static networking vim /etc/sysconfig/network-scripts/ifcfg-eth0 nmcli con reload && nmcli con up eth0 ip add or nmtui nmcli con reload && nmcli con up eth0 ip add Step three: install packages and updates yum install vim bash-completion setroubleshoot epel-release net-snmp && yum upgrade shutdown -r +4 wait for reboot yum install podman Step four: scripts Create three executable scripts from the below text Edit them if you have different IP’s or are using a different folder/mount point. The first two should be all you need, unless you need to cleanup and start over. #!/bin/bash # Steam sudo podman run \ --restart always -d \ --name steam-cache \ -v /mnt/cached/cache/steam/data:/data/cache \ -v /mnt/cached/cache/steam/logs:/data/logs \ -p 192.168.0.50:80:80 \ lancachenet/generic:latest #Origin sudo podman run \ --restart always -d \ --name origin-cache \ -v /mnt/cached/cache/origin/data:/data/cache \ -v /mnt/cached/cache/origin/logs:/data/logs \ -p 192.168.0.51:80:80 \ lancachenet/generic:latest #Blizzard / Battle.net sudo podman run \ --restart always -d \ --name blizzard-cache \ -v /mnt/cached/cache/blizzard/data:/data/cache \ -v /mnt/cached/cache/blizzard/logs:/data/logs \ -p 192.168.0.52:80:80 \ lancachenet/generic:latest #Riot Games sudo podman run \ --restart always -d \ --name riot-cache \ -v /mnt/cached/cache/riot/data:/data/cache \ -v /mnt/cached/cache/riot/logs:/data/logs \ -p …

[2FA]

This is my compose file:

services:
  dns:
    image: lancachenet/lancache-dns:latest
    env_file: .env
    restart: unless-stopped
    ports:
      - ${DNS_BIND_IP}:53:53/udp
  sniproxy:
    image: lancachenet/sniproxy:latest
    env_file: .env
    restart: unless-stopped
    ports:
      - 443:443/tcp
  monolithic:
    image: lancachenet/monolithic:latest
    env_file: .env
    restart: unless-stopped
    ports:
      - 80:80/tcp
    volumes:
      - ${CACHE_ROOT}/cache:/data/cache
      - ${CACHE_ROOT}/logs:/data/logs

And my .env

LANCACHE_IP=10.20.1.34
DNS_BIND_IP=10.20.1.34
UPSTREAM_DNS=10.20.1.35
CACHE_ROOT=/cache
CACHE_DISK_SIZE=10000000m
CACHE_MEM_SIZE=500m
CACHE_MAX_AGE=3650d

I have systemd-resolved set to listen on 127.0.0.53 which I believe is the default. Not sure if you knew but not specifying an IP in the port mapping makes it bind to all IPs. I specifically binded to 10.20.1.34 so that it doesn’t interfere with the DNS resolution of systemd-resolved.

Also the commands docker events and docker logs <container name> are super handy for troubleshooting.

[cotton]

Does anyone have any docs for creating an SMTP system such as the one below:

Basically, There are mulitple applications which need to relay to a single FQDN. Say smtp.example.com. That relay needs to then be able to forward the message to the correct endpoint on the right. The rules are applications connect to the relay using SMTP. To the right of the relay the same endpoint is used — smtp.3rdpartysmtprelay.com.

I’ve done this once already, using postfix and some interesting dns stuff, but I’d like to know if anyone has any alternative suggestions.

[mutation666]

IP KVMs what ones have you used/liked? Looking to update one at work wanted to know if anyone had any opinions on them.

[cdoublejj]

@mutation666

IP KVMs what ones have you used/liked? Looking to update one at work wanted to know if anyone had any opinions on them.

very interested, they are all a few $k USD last i looked, all i can afford is old java windows xp based stuff. it’s almost cheaper to buy an asrock PAUL for each machine.

[rcxb]

If you only need to control a small number of a systems, PiKVM is an option. Perhaps combine it with a (non-IP) KVM with either keyboard based, RS232 or other type of remote control option.

For more systems, some old gear is cheap and still works well…
You can get a 2161DS-2 for under $90. Then the SIP/POD for each computer for $12/ea. If you want to connect more than 16 systems, 0FG697 Expansion modules are fairly cheap as well.

Yeah, to get a web based console on those, you need to use Internet Explorer or something else old that supports java plugins, BUT you can download the stand-alone utility, which still works well on modern versions of Windows and Linux.

BEWARE of other IP-KVMs on eBay like Avocent. You’ll find some selling for $20 but they won’t do anything until you pay for a license which you must renew forever.

[cdoublejj]

@2FA

This is my compose file:

services:
  dns:
    image: lancachenet/lancache-dns:latest
    env_file: .env
    restart: unless-stopped
    ports:
      - ${DNS_BIND_IP}:53:53/udp
  sniproxy:
    image: lancachenet/sniproxy:latest
    env_file: .env
    restart: unless-stopped
    ports:
      - 443:443/tcp
  monolithic:
    image: lancachenet/monolithic:latest
    env_file: .env
    restart: unless-stopped
    ports:
      - 80:80/tcp
    volumes:
      - ${CACHE_ROOT}/cache:/data/cache
      - ${CACHE_ROOT}/logs:/data/logs

And my .env

LANCACHE_IP=10.20.1.34
DNS_BIND_IP=10.20.1.34
UPSTREAM_DNS=10.20.1.35
CACHE_ROOT=/cache
CACHE_DISK_SIZE=10000000m
CACHE_MEM_SIZE=500m
CACHE_MAX_AGE=3650d

I have systemd-resolved set to listen on 127.0.0.53 which I believe is the default. Not sure if you knew but not specifying an IP in the port mapping makes it bind to all IPs. I specifically binded to 10.20.1.34 so that it doesn’t interfere with the DNS resolution of systemd-resolved.

Also the commands docker events and docker logs <container name> are super handy for troubleshooting.

i just disabled the system d thingies as per common issue on lancache site. idk what the system d dns even is needed for?

[2FA]

@cdoublejj

dk what the system d dns even is needed for?

So the local system, in this case your Lancache system, can resolve DNS. If you want Lancache to listen on 0.0.0.0, so all IP addresses on the system, you have to disable resolved. I don’t have to do that because I have mine specifically set to one IP address.

I have mine set that way because I have a couple Windows DNS servers in front of my Lancache for local records. Mine is like Client–>AD DNS–>Lancache–>AdGuard Home. AdGuard forwards to Quad9 over HTTPS. Looking at the documentation for Lancache, apparently you can set your own local records so I might do that instead of the AD DNS as it is kind of overkill lol

[oO.o]

Anyone have experience with Lanner?

Online Store for Lanner's Next-Gen x86 Network Computing Appliances

The network appliances are cool, albeit, mostly out of my price range. Some of them appear to be on ali for much cheaper though.

https://www.alibaba.com/product-detail/1U-High-Performance-x86-Network-Appliance_60669409174.html?spm=a2700.details.0.0.32017ae0agslax

But I’m a little in the dark. I don’t know how an OPMA IPMI slot works or what I need to buy to put in what slot, and exactly which NIC modules to buy or where to buy them.

I kinda want to order one and try and jam things into it until something works or I break it, but that’s still a pretty expensive adventure.

[cotton]

I have a question about my understanding of ephemeral ports, proxies, and linux.

It’s my understanding that Linux, as a client, is limited on the number of concurrent connections to a specific server over a specific port. This is because there is a finite number of ephemeral ports on a linux machine. For example, a linux client can only make about 64,000ish concurrent connections to a specific server’s 443 socket.

If this is true: Assume there is an linux machine running as an HAProxy. This Linux machine is operating as a reverse proxy to a backend service. Further, it is operating as a TLS termination endpoint for client connections to itself. This would mean that when a client connection to the reverse proxy takes place, there is a subsequent connection made from the reverse proxy to the backend. In the second half of that statement, the reverse proxy is operating as a client in regards to it’s interaction “moving” the request to the backend service.

If my understanding is correct, it seems to me there is the ephermal limit of concurrent connections from the reverse proxy to the backend service.

If anyone can weigh in, please let me know if my understanding is correct.

Respectfully,

cotton

[judahnator]

From what I have seen in production systems with a very high number of connections is that you run into issues with file descriptors long before you run into any networking issues.

That isn’t to say that it’s a non-issue, but with HA-Proxy or Nginx or even something like Swoole, you run into other bottlenecks way faster.

[oO.o]

I’ve never run something large enough where that’s an issue, but I found this:

How to handle ephemeral port exhaustion in envoy

[rcxb]

@cotton

It’s my understanding that Linux, as a client, is limited on the number of concurrent connections to a specific server over a specific port. This is because there is a finite number of ephemeral ports on a linux machine.

This in NOT specific to Linux at all. It’s fundamental to TCP/IP. There has to be a SRC_IP:PORT pair and a DEST_IP:PORT pair for every connection. Ports numbers are 0-65535, with those below 1024 set aside for privileged use only.

If that ever becomes a problem for you (almost certainly will not) there are plenty of options:

HTTP Keep-Alive, Pipelining, Multiplexing and Connection Pooling - HAProxy...

[thro]

If you’re hitting those limits you’re likely going to do something like dns round robin or such in front of haproxy.

[ulzeraj]

Anyone having issues with Azure OMSAgent for Linux and its upgrade to the latest version doing dumb shit like deleting and creating the system users but since the new users has a different UID the data folders are not allowed and the upgrade process is broken?