ffind hangs on large file

[David Marcus]

The command

ffind /t"xyz" Test.txt

hangs while using maximum cpu. The file Test.txt is 12,278,005 bytes, has only one line, and the string "xyz" does not occur in the file.

This problem is similar to

ffind /e hangs

but that problem involved a regular expression. Does /t have a limit on the length of lines?

TCC 26.00.31 x64 Windows 10 [Version 10.0.18363.752]

 

[rconn]

Same as /E. If you do a /T that is not a reverse search, it is converted to a /E search (because it's 10x faster).

 

[David Marcus]

It may be 10x faster in many cases, but in my case, it is infinitely slower. In your Dec. 30, 2019, post, you recommended using /T for this use case; now my btm file is broken again. Hanging is not user friendly; an error message would be better. (I'll rewrite my btm.)

 

[vefatica]

So what's the bottom line here? Should FFIND find be able to fail to find text in a single-line 12,000,000 MB file? Should it work or not? My attempt has been running for over 10 minutes, using all of one CPU; I can't interrupt it with Ctrl-C.

 

[Charles G]

@David Marcus

Does ffind work if you tried TCC 25 ?

 

[David Marcus]

It was working fine in TCC 25, as suggested in ffind /e hangs . It appears that this was an intentional, undocumented, breaking change (it isn't mentioned in the What's New).

 

[rconn]

There is no line length limit in FFIND; but there is a 16Mb buffer size limit (which is to prevent pipes from taking forever to load when the piped input is very large).

I cannot reproduce any problem with a 12Mb file & FFIND. Please zip your test file and send it to [email protected] and I will try again to reproduce it here.

 

[rconn]

I tried your file with v26:

[D:\TakeCommand26\TCConsole]timer & ffind /m/b/t"Page Not Found" playerlist13.htm & timer
Timer 1 on:  9:51:52a
Timer 1 off:  9:51:52a  Elapsed: 0:00:00.028

[D:\TakeCommand26\TCConsole]ver

TCC  26.00.32 x64   Windows 10 [Version 10.0.19592.1001]

[D:\TakeCommand26\TCConsole]

FFIND and LIST (which share most of their code) have been deprecated for some time now other than for simple cases. They were both written for 4DOS with a focus on small code, not fast code, and I will not be adding any new features to them (ever).

LIST was replaced with VIEW years ago; for any heavy / complex use TPIPE is a better solution than FFIND. I will probably write a new (much faster) replacement for FFIND for a future version of TCC.

 

[Kachupp]

i seem to remember that we were told by you that ffind should be used like so /m /b /t"text" some file , I dont see any spaces between the optional switches. I know it still works but some consistency along syntax forming should be followed by the author no?

 

[rconn]

I was duplicating the syntax that the OP said he was using (in a separate email).

 

[David Marcus]

My email used spaces between the switches. The problem is 100% reproducible for me, so I don't know why Rex can't reproduce it. Vince said he can reproduce it. Regardless, I've switched to Cygwin's grep, which does not have the problem with long lines that FFIND has when using /T"xx" and /E"xx".

 

[vefatica]

I can no longer reproduce it with build 32. I've tried a 12 MB file (test.txt) of random ASCII printables (32-126) and a 12 MB concatenation of 538 copies of my ApplicationPrivateSettings.xml (big.xml), both files having no newlines.

When I ask it to find non-existent test, I get results like Rex's.

c:\users\vefatica\appdata\local\microsoft\visualstudio\16.0_6b6f020f> timer & ffind /m/b/t"Page Not Found" big.xml & timer
Timer 1 on: 11:55:40
Timer 1 off: 11:55:40  Elapsed: 0:00:00.021

v:\> timer & ffind /m/b/t"Page Not Found" test.txt & timer
Timer 1 on: 11:56:05
Timer 1 off: 11:56:05  Elapsed: 0:00:00.021

When I ask it to find test which does exist and with the "/b" option, it finds it quickly.

v:\> timer & ffind /m/b/t"X" test.txt & timer
Timer 1 on: 11:58:56
V:\test.txt
Timer 1 off: 11:58:56  Elapsed: 0:00:00.016

And for existing test without "/b" it immediately spits out the one line (all 12 MB).

My only gripe is that it takes 3+ minutes to spit out that line (TYPE takes 1.7 second) and it can't be interrrupted.

Except for the speed of output and the fact that it can't be interrupted, it seems to be working correctly.

 

[vefatica]

In another test with existing text and without "/b", where the text is near the end of the file, it's fast but only spits out the last 6934 bytes of the file. That might be WAD ... beats rewinding to find out where the line began (?).

v:\> timer & ffind /m/t"n7g=17Sy)Bb4" test.txt | wc & timer
Timer 1 on: 12:13:10
      3      86    6934
Timer 1 off: 12:13:10  Elapsed: 0:00:00.193

The two extra lines reported by wc.exe are an empty line and the header "---- V:\test.txt".

 

[David Marcus]

In build 32, it works for my 12 megabyte file. But, it hangs for a file of size 25 megabytes.

If there are unexpected limitations like this, it would save time (and forum posts) if the Help mentioned it.

 

[vefatica]

Same as /E. If you do a /T that is not a reverse search, it is converted to a /E search (because it's 10x faster).

If you're going to do that you might massage the regex (search string) a bit (more) so Oniguruma doesn't choke on it (first below) or do the wrong thing (second below).

v:\> ffind /m/t"d:\tc26\1" sv.btm
TCC: invalid backref number/name "(?i)d:\tc26\1"

d:\tc26> ffind /m/t"d:\tc26" tcmd.ini

d:\tc26> ffind /m/t"d:\\tc26" tcmd.ini

---- D:\tc26\tcmd.ini
TCStartPath=d:\TC26

 

[rconn]

In build 32, it works for my 12 megabyte file. But, it hangs for a file of size 25 megabytes.

If there are unexpected limitations like this, it would save time (and forum posts) if the Help mentioned it.

OK. FFIND will also not work with files > 128 yottabytes.

And regarding your file - it's not a 25Mb file that's a problem, it's a (somewhat ridiculous) 25Mb single line.

 

[David Marcus]

That's a ridiculous thing to say. Were you trying to be funny?

 

[rconn]

I have reverted FFIND /T in 26.01.33 to the (vastly) slower v25 behavior.

 

[David Marcus]

Thank you, but I've already switched to Cygwin's grep for this, so please don't do it on my account. A note in the Help that ffind expects line endings every so often would suffice. (Although, it would be friendlier if /e and /t errored when you exceed the buffer limit rather than hanging.)