Re: limiting outbound packet size

• Date: Fri, 17 Jan 1997 10:28:53 PST
• From:
  Jay Carlson <nop@nop.com>
• cc: cunkel@us.itd.umich.edu (Christopher Unkel), moo-cows@parc.xerox.com
• In-reply-to: Your message of "Fri, 17 Jan 97 07:26:34 PST." <199701171526.AA09925@crl5.crl.com>

>> However, you can probably reconfigure your ethernet interface with a
>> smaller MTU to avoid the problem; there is probably an mtu option on the
>> ifconfig command (I can't find a machine with matching OS version so I
>> can't confirm this.) that could be used to keep your ethernet interface
>> from sending large packets.  Doing so would hurt local-network performance
>> but might solve your other problems. 

> No, this won't help at all.. in fact it would just make things worse
> (Dave, if you're listening, DON'T have SRI do this).  The problem
> here is that Dave didn't adequately describe what's really going on
> which is causing the problem.

> The problem is not due to the MOO's machine sending out ethernet
> packets which are too large, the problem is what happens when it
> sends out _IP_ packets which are too large (larger than the MTU of
> the network (1500)).  When an IP packet which is larger than the MTU
> is sent, the IP layer breaks this packet down into "fragments",
> causing one IP packet to be sent as multiple ethernet (or whatever)
> frames.  The multiple network frames are then reassembled at the
> other end of things into the original IP packet.

> However, the problem we're experiencing appears to be at least
> partially due to a router at our ISP (well, several routers, it
> appears) which are overburdened and are having problems passing
> certain types of traffic without losing bits of it.  These routers
> seem to be able to pass single-packet network frames ok, but when
> they encounter an IP packet which has been fragmented into multiple
> frames, it won't get through (or won't get through complete), and
> the connection just stops while TCP keeps retrying (and keeps
> failing) to send it through.

I wonder if they're trying to reassemble the packet.  That used to be
considered a Bad Thing for routers to do, but I'm way out of touch.

Consider disabling MTU Discovery.  If your upstream routers don't
correctly handle fragmentation, MTU discovery will produce very bad
results.

> This is why I was attempting to reduce the size of the IP packets
> which DU was sending to under 1500 (to keep the IP packet size under
> the MTU of the network, and thus avoid fragmentation).  For obvious
> reasons, therefore, reducing the MTU of the local ethernet adapter
> would only make the situation worse, as it would cause even more
> packets to be fragmented and thus die.

Ah, the tricky bit here is that reducing the MTU on the interface will
also reduce the maximum segment size (MSS) used by TCP.  A requirement
of conforming TCP/IP implementations is that they

    MUST calculate "effective send MSS" correctly:
      min(physical_MTU, remote_MSS+20) - sizeof(tcphdr) - sizeof(ipopts)

according to RFC1122 (paraphrased by the Linux kernel source).  I've
heard remarks to the effect that this can be relaxed in certain
circumstance, but correct implementations must follow the spirit of
this for obvious reasons.

In other words, reducing the MTU on an interface will reduce the size
of the individual TCP packets sent such that each segment will fit in
a single packet.

Certain operating systems allow you to change the MSS on a per-route
basis.  I'm assuming that you're not doing anything silly like that.

> BTW, it is also quite clear to me that the problems DU is
> experiencing are not due to local OS bugs, but due to network
> congestion or router misconfiguration at the ISP, and therefore
> applying various SunOS patches to the machine are unlikely to change
> much (though, admittedly, they are a good thing to do in general)

The problems you're having I believe are aggravated by your local
configuration.  The finger of blame can be pointed at your network
providers, certainly, but there are things you can do to work around
them.  Solaris 2.4 and 2.5 have notorious wide-area TCP performance
problems and I highly recommend installing 2.5 or 2.5.1 with the
recommended patch cluster if your server is connected to lossy
wide-area networks such as the commercial Internet.  Sun announced a
bunch of performance patches for 2.5.1 as the Solaris Internet Server
Supplement (SISS), and you may want to investigate ordering that free
CD as well.
-- 
Jay Carlson    nop@nop.com    nop@kagoona.mitre.org

Flat text is just *never* what you want.   ---stephen p spackman

• Re: limiting outbound packet size
• From: Alex Stewart <riche@crl.com>

• Prev: Re: limiting outbound packet size
• Next: next server release
• Index: MOO-cows
• Thread: MOO-cows