'Too many open files' - SSH tunnel error

Networking
1

Hello,
When doing I/O via SSH tunnel, the tty sends this output:

accept: Too many open files
accept: Too many open files
channel 1019: open failed: administratively prohibited: open failed
channel 795: open failed: administratively prohibited: open failed

Here are my settings:

# /etc/security/limits.conf
#
#Each line describes a limit for a user in the form:
#
#<domain>        <type>  <item>  <value>
#
#Where:
#<domain> can be:
#        - an user name
#        - a group name, with @group syntax
#        - the wildcard *, for default entry
#        - the wildcard %, can be also used with %group syntax,
#                 for maxlogin limit
#        - NOTE: group and wildcard limits are not applied to root.
#          To apply a limit to the root user, <domain> must be
#          the literal username root.
#
#<type> can have the two values:
#        - "soft" for enforcing the soft limits
#        - "hard" for enforcing hard limits
#
#<item> can be one of the following:
#        - core - limits the core file size (KB)
#        - data - max data size (KB)
#        - fsize - maximum filesize (KB)
#        - memlock - max locked-in-memory address space (KB)
#        - nofile - max number of open files
#        - rss - max resident set size (KB)
#        - stack - max stack size (KB)
#        - cpu - max CPU time (MIN)
#        - nproc - max number of processes
#        - as - address space limit (KB)
#        - maxlogins - max number of logins for this user
#        - maxsyslogins - max number of logins on the system
#        - priority - the priority to run user process with
#        - locks - max number of file locks the user can hold
#        - sigpending - max number of pending signals
#        - msgqueue - max memory used by POSIX message queues (bytes)
#        - nice - max nice priority allowed to raise to values: [-20, 19]
#        - rtprio - max realtime priority
#        - chroot - change root to directory (Debian-specific)
#
#<domain>      <type>  <item>         <value>
#

#*               soft    core            0
#root            hard    core            100000
#*               hard    rss             10000
#@student        hard    nproc           20
#@faculty        soft    nproc           20
#@faculty        hard    nproc           50
#ftp             hard    nproc           0
#ftp             -       chroot          /ftp
#@student        -       maxlogins       4

# End of file


file in /etc/sysctl.d
------------------------
vm.swappiness = 10
fs.file-max = 204708


another file
------------------------

# Auto-reboot linux 10 seconds after a kernel panic
kernel.panic = 10
kernel.panic_on_oops = 10
kernel.unknown_nmi_panic = 10
kernel.panic_on_unrecovered_nmi = 10
kernel.panic_on_io_nmi = 10

# Controls whether core dumps will append the PID to the core filename, useful for debugging multi-threaded applications
kernel.core_uses_pid = 1

# Allow for more PIDs
kernel.pid_max = 4194303

# Turn off address space randomization - the servers are behind a firewall
kernel.randomize_va_space = 0

# ------ VM ------

# See http://en.wikipedia.org/wiki/Swappiness
vm.swappiness = 1

# ------ VM ------

fs.file-max = 204708
fs.epoll.max_user_instances = 4096

# ------ NETWORK SECURITY ------

# Protect ICMP attacks
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Turn on protection for bad icmp error messages
net.ipv4.icmp_ignore_bogus_error_responses = 1

# Turn on syncookies for SYN flood attack protection
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_synack_retries = 3

# Log suspicious packets, such as spoofed, source-routed, and redirect
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1

# Disables these ipv4 features, not very legitimate uses
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0

# ------ NETWORK PERFORMANCE ------

# don't cache ssthresh from previous connection
net.ipv4.tcp_no_metrics_save = 1

# Allow reusing sockets in TIME_WAIT state for new connections
net.ipv4.tcp_tw_reuse = 1

# Socket max connections waiting to get accepted; the listen() backlog.
# Default is 128.
net.core.somaxconn = 4096

# Enable receiver autotuning. Receiver autotuning is 'new'; sender autotuning has been around a long time.
# Default is disabled.
net.ipv4.tcp_moderate_rcvbuf = 1

# Reduce TCP retries.
# Default is 15.
net.ipv4.tcp_retries2 = 3

# Tune TCP keepalive.
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl = 30

# Decrease fin timeout. After telling the client we are closing, how long to wait for a FIN, ACK?
# Default is 60.
net.ipv4.tcp_fin_timeout = 10

# Enable TCP FAST_OPEN for client and server. Still rarely used by applications. See https://lwn.net/Articles/508865/. Default from kernel 3.13.
net.ipv4.tcp_fastopen = 3
2

Look at the output of ulimit -a

You may find max open files is 1024.

1 Like
3

@MarcT
Indeed, open files is set to 1024. What do you recommend?

4

Try ulimit -n -H which will show the "hard" limit for open files (-n), for example 4096. Then raise the limit to that value eg: ulimit -n 4096 before invoking your SSH tunnel.

It's a temporary change and only affects the one login session, so would need to be done each time. You could put the command in your login script though.

If it doesn't work, it could be the remote system running out of open files. You'd then have to change the limit in the sshd startup on the remote system.

5

Works well