Welcome!

By registering with us, you'll be able to discuss, share and private message with other members of our community.

SignUp Now!

IFTP problems (broken pipe?)

May
12,846
164
I can get this file OK without IFTP:
Code:
v:\internetregistries> copy ftp://ftp.arin.net/pub/stats/arin/delegated-arin-extended-latest
ftp://ftp.arin.net/pub/stats/arin/delegated-arin-extended-latest => V:\InternetRegistries\delegated-arin-extended-latest
  1 file copied

But if fails when I try it from an IFTP session:
Code:
v:\internetregistries> iftp "ftp://ftp.arin.net/pub/stats"

v:\internetregistries> copy ftp:arin/delegated-arin-extended-latest delegated-arin-latest
ftp://ftp.arin.net/pub/stats/arin/delegated-arin-extended-latest => V:\InternetRegistries\delegated-arin-latest
TCC: FTP protocol error: 451 Transfer aborted. Broken pipe "/pub/stats/arin/delegated-arin-extended-latest"
  0 files copied  1 failed

Here's the verbose response to that command:
Code:
PWD
257 "/pub/stats" is the current directory
PWD
257 "/pub/stats" is the current directory
CWD /pub/stats/arin/delegated-arin-extended-latest
550 /pub/stats/arin/delegated-arin-extended-latest: No such file or directory
PORT 74,79,84,76,240,125
200 PORT command successful
LIST /pub/stats/arin/delegated-arin-extended-latest
150 Opening BINARY mode data connection for file list
226 Transfer complete
ftp://ftp.arin.net/pub/stats/arin/delegated-arin-extended-latest => V:\InternetRegistries\delegated-arin-latest
TYPE I
200 Type set to I
SIZE /pub/stats/arin/delegated-arin-extended-latest
213 7434610
PORT 74,79,84,76,240,126
200 PORT command successful
RETR /pub/stats/arin/delegated-arin-extended-latest
150 Opening BINARY mode data connection for /pub/stats/arin/delegated-arin-extended-latest (7434610 bytes)
451 Transfer aborted. Broken pipe
TCC: FTP protocol error: 451 Transfer aborted. Broken pipe "/pub/stats/arin/delegated-arin-extended-latest"
  0 files copied  1 failed
 
Some antivirus programs, including McAfee, default to blocking FTP. I spent two hours on the phone with McAfee support to find out what it takes to enable the RFC.
 
This has something to do with big files. I can IFTP plugins from lucky.syr.edu just fine. But when I put a copy of the ARIN file on lucky, or put a 7 MB file of my own there, I get the same failure:
Code:
v:\> copy ftp:7meg
ftp://lucky.syr.edu/4plugins/7meg => V:\7meg
TCC: FTP protocol error: 426 Data connection unexpectedly closed, file transfer /4plugins/7meg aborted by client. "/4plugins/7meg"
  0 files copied  1 failed

The same error message appears in the server's log:
Code:
426 Data connection unexpectedly closed, file transfer /4plugins/7meg aborted by client
 
FWIW: COPY via IFTP is OK for a 131072-byte file (128K). It fails, as described above, for a 131073-byte file! I have repeated the test below many times.
Code:
v:\> iftp "ftp://lucky.syr.edu/4plugins"

v:\> rndmfile \\lucky\e$\anonymous\4plugins\128k %@eval[128*2**10]

v:\> copy "ftp:128k" junk.zip
ftp://lucky.syr.edu/4plugins/128k => V:\junk.zip
  1 file copied

v:\> rndmfile \\lucky\e$\anonymous\4plugins\128k %@eval[128*2**10+1]

v:\> copy "ftp:128k" junk.zip
ftp://lucky.syr.edu/4plugins/128k => V:\junk.zip
TCC: FTP protocol error: 426 Data connection unexpectedly closed, file transfer
/4plugins/128k aborted by client. "/4plugins/128k"
  0 files copied  1 failed
 
When I try this on the ftp server itself, with the IFTP connection to "localhost", the difference between a successful copy and an unsuccessful copy is the difference between a 1984-byte file and a 1985-byte file. I have repeated this experiment many times.
Code:
v:\> do i=1984 to 1985 ( rndmfile e:\anonymous\4plugins\test\xxx.txt %i & echo ^nfilesize = %i & copy ftp:xxx.txt & delay 1 )

filesize = 1984
ftp://localhost/4plugins/test/xxx.txt => V:\xxx.txt
  1 file copied

filesize = 1985
ftp://localhost/4plugins/test/xxx.txt => V:\xxx.txt
TCC: FTP protocol error: 426 Data connection unexpectedly closed, file transfer
/4plugins/test/xxx.txt aborted by client. "/4plugins/test/xxx.txt"
  0 files copied  1 failed
 
Not reproducible here with 16.02.49 (tried 20 repetitions with "copy ftp:arin/delegated-arin-extended-latest delegated-arin-latest").
I don't know what to say. It does it on differently on two client computers, and it happens with using two different servers. On one computer, the difference is 131072/131072 bytes and on the other it's 1984/1985 bytes. It would seem like some sort of undefined behavior that hasn't bitten you. In addition, COPY FTP:// works while IFTP doesn't, IFTP works fine in TCC version 15, and none of FTP.EXE, CuteFTP, FireFox, and IE have any problem. On lucky (work machine), it's exactly the same today, after a restart. If you have any ideas for my debugging it I'd like to hear them.

A capture shows two strange items when this error occurs. (1) TCC sends an ACK on the data channel (port 20) which advertises Win=0 (receive window size, I imagine) and (2) that's immediately followed by TCC sending a packet on the control channel (port 21) ... which is odd. During a successful transfer, there's no traffic on the control channel until it's over.
 
In case anyone knows better than I how to read network captures, I've attached two NetMon captures of failed downloads (one from ARIN, one from lucky).
 

Attachments

  • captures.zip
    18.4 KB · Views: 248
I don't know what to say. It does it on differently on two client computers, and it happens with using two different servers. On one computer, the difference is 131072/131072 bytes and on the other it's 1984/1985 bytes. It would seem like some sort of undefined behavior that hasn't bitten you. In addition, COPY FTP:// works while IFTP doesn't, IFTP works fine in TCC version 15, and none of FTP.EXE, CuteFTP, FireFox, and IE have any problem. On lucky (work machine), it's exactly the same today, after a restart. If you have any ideas for my debugging it I'd like to hear them.

A capture shows two strange items when this error occurs. (1) TCC sends an ACK on the data channel (port 20) which advertises Win=0 (receive window size, I imagine) and (2) that's immediately followed by TCC sending a packet on the control channel (port 21) ... which is odd. During a successful transfer, there's no traffic on the control channel until it's over.

Since you're not using 16.02.49, I don't think further tests are meaningful until it's released & you upgrade.
 
I'd love to get a copy of rndmfile please....

FWIW: COPY via IFTP is OK for a 131072-byte file (128K). It fails, as described above, for a 131073-byte file! I have repeated the test below many times.
Code:
v:\> iftp "ftp://lucky.syr.edu/4plugins"

v:\> rndmfile \\lucky\e$\anonymous\4plugins\128k %@eval[128*2**10]

v:\> copy "ftp:128k" junk.zip
ftp://lucky.syr.edu/4plugins/128k => V:\junk.zip
  1 file copied

v:\> rndmfile \\lucky\e$\anonymous\4plugins\128k %@eval[128*2**10+1]

v:\> copy "ftp:128k" junk.zip
ftp://lucky.syr.edu/4plugins/128k => V:\junk.zip
TCC: FTP protocol error: 426 Data connection unexpectedly closed, file transfer
/4plugins/128k aborted by client. "/4plugins/128k"
  0 files copied  1 failed
 
I'd love to get a copy of rndmfile please....
Here it is. It's one of the oldest programs I have around here. I wrote it back in the 90s and it still bears the date stamp 1997-02-09. I was a little surprised to find out (last night) that it would make a gigabyte file. As the name (almost) implies it writes the specified number of random bytes (0~255) to a file. I originally used it to make files that were uncompressible. Even 17 years later, when I zip a 1MB file, 1,048,576 bytes, it comes out 1,049,006 bytes!
 

Attachments

  • rndmfile.zip
    14.8 KB · Views: 267
Code:
 6/06/2014  16:21         <DIR>    .
6/06/2014  16:21         <DIR>    ..
6/06/2014  16:27      16,609,993  1M.7z
6/06/2014  16:17   1,073,741,824  1M.txt
6/06/2014  16:20   1,073,741,970  1m.zip
12/13/1996   0:15          29,184  RNDMFILE.exe

So at least 7-zip does compress the file.

1M.txt is rndmfile 1m.txt 1024**1024....
 
Code:
 6/06/2014  16:21         <DIR>    .
6/06/2014  16:21         <DIR>    ..
6/06/2014  16:27      16,609,993  1M.7z
6/06/2014  16:17   1,073,741,824  1M.txt
6/06/2014  16:20   1,073,741,970  1m.zip
12/13/1996   0:15          29,184  RNDMFILE.exe

So at least 7-zip does compress the file.

1M.txt is rndmfile 1m.txt 1024**1024....
I don't believe that. I think there was a mix-up or 7z failed. Here I get the following (and 7za did compress the big one a tad).
Code:
2014-06-06  17:08  1,067,794,245  1gig.7z
2014-06-06  17:01  1,073,741,824  1gig.bin
2014-06-06  16:59  1,060,701  1meg.7z
2014-06-06  16:59  1,048,576  1meg.bin
 
OK, I believe you. After an upgrade of 7z from v4 to v9, I get results similar to yours.
Code:
2014-06-06 17:28 16,609,268 1gig.7z
2014-06-06 17:01 1,073,741,824 1gig.bin
That shows you the inadequacies of the "C" language built-in random number generator, rand(). Attached is a new version of RNDMFILE which uses the "Mersenne Twister" random number generator. And here's the result of trying to 7z one made with that new version.
Code:
2014-06-06 17:38 1,088,279,243 1gig.7z
2014-06-06 17:32 1,073,741,824 1gig.bin
I added a feature to the new one:
Code:
Syntax: RNDMFILE filename size [/T] ... /T(ext): chars 32-126
 

Attachments

  • rndmfile.zip
    22.6 KB · Views: 240
echo %@verinfo["c:\Program Files\7-Zip\7z.exe"]
9.20

Win7 x64
 
Vince,

A suggestion would be when rndmfile is ran - maybe provide some sample parameters....

Syntax: RNDMFILE filename size [/T] ... /T(ext): chars 32-126
Example: RNDMFILE 1m.bin 1073741824
: RNDMFILE 1m.txt 1073741824 /T
maybe a few more too?? does /T need anything after?
 
Code:
 6/06/2014  19:52         <DIR>    .
6/06/2014  19:52         <DIR>    ..
6/06/2014  19:57   1,088,309,341  1M.7z
6/06/2014  19:46   1,073,741,824  1M.txt
6/06/2014  19:48   1,073,741,970  1m.zip
6/06/2014  16:53          45,056  RNDMFILE.exe
 
I don't believe that PkZzip did better then 7-zip - but those are the results I just got...
 
Vince,

A suggestion would be when rndmfile is ran - maybe provide some sample parameters....

Syntax: RNDMFILE filename size [/T] ... /T(ext): chars 32-126
Example: RNDMFILE 1m.bin 1073741824
: RNDMFILE 1m.txt 1073741824 /T
maybe a few more too?? does /T need anything after?
OK. Your examples should be sufficient. But don't you have 'm' and 'g' confused? :-)
"/T" needs nothing more. Do you think the "/T" option is useful? I can imagine another option, perhaps "/L N" ... output CRLF-terminated lines of N bytes. But I wonder if that would be useful. When I first made RNDMFILE, I also made separately RNDMTEXT (bytes 32 - 126) and RNDM7BIT (bytes 0 - 127), but over many years, RNDMFILE is the only one I've used.
 
... or remove the "/T" option and add

Code:
Syntax:  RNDMFILE filename size [M N]
M N ... use bytes N to M, inclusive; default 0 255
As I said, the simple RNDMFILE has served me well for many years. I'm not inclined to make it too fancy.
 
... or remove the "/T" option and add

Code:
Syntax:  RNDMFILE filename size [M N]
M N ... use bytes N to M, inclusive; default 0 255
As I said, the simple RNDMFILE has served me well for many years. I'm not inclined to make it too fancy.

or if you want to keep the /T, then add a /7 for the RNDM7BIT.exe - that way the other old files would never be needed...
 
At the moment, I have this.
Code:
Syntax:  RNDMFILE filename size [M N]

M N - use only bytes M to N (inclusive); default: 0 255

Examples: RNDMFILE 1meg.bin 1048576  (random bytes)
  RNDMFILE 1meg.101 1048576 48 49  ('0's and '1's)
  RNDMFILE 1meg.A2Z 1048576 65 90  ('A' to 'Z')
  RNDMFILE 1meg.txt 1048576 32 126 (English text characters)

Interesting: I made a 1GB file using "rmdmfile 101.bin %@eval[2**30] 0 1" (all bytes 0 or 1, ~22 seconds) and zipped it with 7za.exe and with the internal ZIP. I wasn't timing carefully, but 7za.exe took about 45 minutes and ZIP took about 2 minutes ... with the following results.
Code:
2014-06-06  22:03  145,498,857  101.7z
2014-06-06  21:36  1,073,741,824  101.bin
2014-06-06  22:05  184,051,302  101.zip
 
I have 5 crossword puzzles to do (4 London Times, 1 NY Times) so I'm through for the night. Here's the latest as described in my last post. Of course, "0 127" will be the equivalent of my RNDM7BIT.EXE (which has a last access date in 1999).
 

Attachments

  • rndmfile.zip
    22.7 KB · Views: 222

Similar threads

Back
Top