OpenBSD Journal

Updates to ftp(1) and pkg_add

Contributed by merdely on from the stayin-alive dept.

Recently, Marc Espie (espie@) reminded us on ports@ that FTP is broken. "So a file transfer goes like this: send some commands, open a data connection, and transfer the file... during the transfer, the control connection stays silent... that is, totally silent. And then, we come back up, and there is NO control connection left. Over the lousy five minutes it took us to transfer the file," the network expired the control connection.

To attempt to make ftp(1) work better, Marc committed code to:

implement a `keep-alive' option that sends bytes over an inactive connection. The FTP protocol provides us with a NOOP operation that is perfectly suitable for that, and so far servers are happy with it. Sending the command slowly is an idea I borrowed from spamd. No change for people not using the option, so it can't break normal ftp.

A new "-k" option was added to the ftp command:

     -k seconds
             Sends a byte after each seconds period over the control connec-
             tion during long transfers, so that incorrectly configured net-
             work equipment won't agressively drop it.  The FTP protocol sup-
             ports a NOOP command that can be used for that purpose.  This as-
             sumes the FTP server can deal with extra commands coming over the
             control connection during a transfer.  Well-behaved servers queue
             those commands, and process them after the transfer.

As Marc then announced on ports@:

[The -k option] doesn't solve the other ftp problem, that in many cases you may have an intermediate proxy that does not understand telnet correctly, and so your urgent ABORTs won't get through correctly, and it will take forever for pkg_add to close connections early (we solved this in our ftp-proxy a few years ago).
So, from pkg_add, you can try out to set the env variable FTP_KEEPALIVE if you think you have the problem... one common symptom is to have the installation of openoffice hanging at the end.

(Comments are closed)


Comments
  1. By Anonymous Coward (85.97.21.103) on

    Before this fix, I tried to solve the problem by setting FTPMODE to "active" and it worked. This doesn't mean that I don't appreciate this fix. Thank you for making our operating system better each day.

    Comments
    1. By Anonymous Coward (84.56.88.23) on

      > Before this fix, I tried to solve the problem by setting FTPMODE to "active" and it worked. This doesn't mean that I don't appreciate this fix. Thank you for making our operating system better each day.

      You aren't serious about that, are you?

      Comments
      1. By Anonymous Coward (85.102.220.197) on

        > > Before this fix, I tried to solve the problem by setting FTPMODE to "active" and it worked. This doesn't mean that I don't appreciate this fix. Thank you for making our operating system better each day.
        >
        > You aren't serious about that, are you?
        >

        I was serious and I would like to know what I was missing.

        Comments
        1. By Anonymous Coward (202.81.60.133) on

          > > > Before this fix, I tried to solve the problem by setting FTPMODE to "active" and it worked. This doesn't mean that I don't appreciate this fix. Thank you for making our operating system better each day.
          > >
          > > You aren't serious about that, are you?
          > >
          >
          > I was serious and I would like to know what I was missing.
          perhaps your firewall blocks outgoing connection to port >1024 which explains why setting FTPMODE to active solved the problem (as I also have same problem).

  2. By Cabal (Cabal) Cabal on http://www.enginuity.org/

    This short-coming is nothing new to the FTP protocol, what made it crop up now?

    Comments
    1. By Marc Espie (213.41.185.88) espie@openbsd.org on

      > This short-coming is nothing new to the FTP protocol, what made it crop up now?

      Two things:
      - increasing number of issues with big packages and pkg_add. I guess it's due to more people using pkg_add over ftp.
      - very lousy network at my current work, due to people sitting on public IP adresses, nat being overused, and aggressively expiring connections.

      Those are `long term' fixes. In many cases, it's fairly hard to test and fix issues you don't see.

      For instance, this has nothing to do with active ftp. There are at least three distinct issues with ftp in there !

      - active vs. passive ftp
      - proxy not relaying telnet ABORT correctly
      - nat firewalls dropping control connections.

      And maybe more.

      Anyways, this is just a band-aid. If someone wants to *look* at ftp and actually fix the code (probably rewrite it), they're welcome.

      Comments
  3. By Anonymous Coward (208.146.43.174) on

    maybe it's because i haven't finished my morning tea yet, but i'm not seeing where ftp-proxy deals with the telnet commands as you mentioned....

    Comments
    1. By sthen (85.158.45.32) on

      > maybe it's because i haven't finished my morning tea yet, but i'm
      > not seeing where ftp-proxy deals with the telnet commands as you
      > mentioned....

      Unless I'm mistaken it's because /usr/libexec/ftp-proxy uses
      NAT/RDR - the data connections are passed straight through,
      they don't need special handling in the proxy.

      Comments
      1. By sthen (85.158.45.32) on

        > > maybe it's because i haven't finished my morning tea yet, but i'm
        > > not seeing where ftp-proxy deals with the telnet commands as you
        > > mentioned....
        >
        > Unless I'm mistaken it's because /usr/libexec/ftp-proxy uses
        > NAT/RDR - the data connections are passed straight through,
        > they don't need special handling in the proxy.

        ugh, s/libexec/sbin/ of course

      2. By Anonymous Coward (208.146.43.174) on

        > > maybe it's because i haven't finished my morning tea yet, but i'm
        > > not seeing where ftp-proxy deals with the telnet commands as you
        > > mentioned....
        >
        > Unless I'm mistaken it's because /usr/libexec/ftp-proxy uses
        > NAT/RDR - the data connections are passed straight through,
        > they don't need special handling in the proxy.

        those commands are sent over the control connection not the data.

    2. By Marc Espie (213.41.185.88) espie@openbsd.org on

      > maybe it's because i haven't finished my morning tea yet, but i'm not seeing where ftp-proxy deals with the telnet commands as you mentioned....

      This was some fix I did in the *previous* ftp-proxy (read cvs log, you'll see this was completely rewritten).

      I haven't actually looked at the new one... I have no idea whether it deals correctly with urgent stuff (which is what this is all about). I'm bound to update my soekris one of these days, now that I've got a big enough flash available to get away from flashdist, but I don't fancy reinstalling my gateway...

      Comments
      1. By Anonymous Coward (71.216.4.2) on

        > > maybe it's because i haven't finished my morning tea yet, but i'm not seeing where ftp-proxy deals with the telnet commands as you mentioned....
        >
        > This was some fix I did in the *previous* ftp-proxy (read cvs log, you'll see this was completely rewritten).
        >
        > I haven't actually looked at the new one... I have no idea whether it deals correctly with urgent stuff (which is what this is all about). I'm bound to update my soekris one of these days, now that I've got a big enough flash available to get away from flashdist, but I don't fancy reinstalling my gateway...
        >
        >

        ah ok, i see it. thanks.
        well it doesn't appear that the new proxy has any of that.

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]