enforcing low-latency SSH

It would not be obvious just from browsing a few pages, but the web server that hosts this site actually functions as a remote headless “workstation” of sorts. I execute a variety of applications, from instant messaging to HDL simulations within a GNU Screen session for many days at a time.

As visits to this site increase each month, I have been observing my SSH latency increasing to absurd levels, with RTT reaching into the minutes. I even experienced typing half of a document page—and waiting in dire suspense of losing the paragraphs to the Internet abyss.

The primary networking solution would be to configure traffic shaping in my Linux router—but most methods that I have tried in the past involved a harry mixture of custom packet tagging and queuing without much for high-level configuration. One example would be ultimate-tc which never worked correctly for me without considerably reducing my overall bandwidth.

Using shorewall’s more high-level configuration and some experimental networking measurements, I determined an over-specified set of rules to give dedicated precedence to SSH when competing with other traffic.

First, the available bandwidth of the external network interface is significantly over-estimated to prevent the router from ever limiting the network bandwidth.

#/etc/shorewall/tcdevices
#INTERFACE   IN-BANDWITH  OUT-BANDWIDTH
 $NET_IF     10000kbit    500kbit

My actual bandwidth is somewhere near:

#             7000kbit    400kbit

Second, high-frequency protocols are assigned Type of Service (ToS) tags to classify their desired network behavior. The catch-all source, destination, and dual port forms are deliberate as I wish for the rules to apply for any communication of a particular protocol, regardless of client/server or lan/Internet relationships.

#/etc/shorewall/tos
#SRC  DEST PROTO  SRC       DEST
                  PORT      PORT      TOS

# max responsiveness (min latency)
 all  all  tcp    -         ssh       16
 all  all  tcp    ssh       -         16
 all  all  tcp    -         ssh-alt   16
 all  all  tcp    ssh-alt   -         16
 all  all  tcp    rsync     -         16
 all  all  tcp    -         rsync     16
 all  all  tcp    imaps     -         16
 all  all  tcp    -         imaps     16
# max reliability
 all  all  tcp    -         ftp       4
 all  all  tcp    ftp       -         4
 all  all  tcp    https     -         4
 all  all  tcp    -         https     4
# min cost / priority
 all  all  tcp    ftp-data  -         2
 all  all  tcp    -         ftp-data  2
 all  all  tcp    http      -         2
 all  all  tcp    -         http      2

In and of themselves, the ToS tags provide some noticeable network improvements when SSH'ing from friends' computers connecting to the same ISP. I assume the only explanation would be that my ISP's routers observe ToS to a limited extent, which is helpful, but not really adequate as I frequently connect across multiple ISP networks.

Third, to enforce ToS internally, I specify the dedicated and capping bandwidths for each ToS class, TCP ACK packets, and any unclassified traffic (‘default’). Shown previously, the download bandwidth was over-specified by a larger ratio than was the upload bandwidth; this prevents any limiting of the download bandwidth and loosens prioritization while downloading, but more tightly enforces upload priorities.

#/etc/shorewall/tcclasses
#INTERFACE  MARK  RATE      CEIL       PRIORITY  OPTIONS
 $NET_IF    1     full*2/3  full       1         tos=0x10/0x10
 $NET_IF    2     full/2    full       2         tos=0x04/0x04
 $NET_IF    3     full/4    full       3         tcp-ackn
 $NET_IF    4     full/5    full*8/10  4         default
 $NET_IF    5     full/10   full*6/10  5         tos=0x02/0x02

With these settings, download networking latency is slightly reduced and bandwidth is unaffected. The upload latency, the original problem, is drastically reduced. Upload bandwidth is rigidly prioritized between protocols, and SSH will steal up to ~60% of the network responsiveness and bandwidth from other protocols automatically. And when there is no competition, each protocol is still able to consume near 100% of the network upload bandwidth.

Now, quite happily, all of my remote sessions remain responsive, regardless of the number of people (or bots) browsing site content; and without noticeably reducing any portion of the network bandwidth.