Welcome!

By registering with us, you'll be able to discuss, share and private message with other members of our community.

SignUp Now!

ffind hangs on large file

Jun
768
6
The command

ffind /t"xyz" Test.txt

hangs while using maximum cpu. The file Test.txt is 12,278,005 bytes, has only one line, and the string "xyz" does not occur in the file.

This problem is similar to

ffind /e hangs

but that problem involved a regular expression. Does /t have a limit on the length of lines?

TCC 26.00.31 x64 Windows 10 [Version 10.0.18363.752]
 
It may be 10x faster in many cases, but in my case, it is infinitely slower. In your Dec. 30, 2019, post, you recommended using /T for this use case; now my btm file is broken again. Hanging is not user friendly; an error message would be better. (I'll rewrite my btm.)
 
So what's the bottom line here? Should FFIND find be able to fail to find text in a single-line 12,000,000 MB file? Should it work or not? My attempt has been running for over 10 minutes, using all of one CPU; I can't interrupt it with Ctrl-C.
 
I tried your file with v26:

Code:
[D:\TakeCommand26\TCConsole]timer & ffind /m/b/t"Page Not Found" playerlist13.htm & timer
Timer 1 on:  9:51:52a
Timer 1 off:  9:51:52a  Elapsed: 0:00:00.028

[D:\TakeCommand26\TCConsole]ver

TCC  26.00.32 x64   Windows 10 [Version 10.0.19592.1001]

[D:\TakeCommand26\TCConsole]

FFIND and LIST (which share most of their code) have been deprecated for some time now other than for simple cases. They were both written for 4DOS with a focus on small code, not fast code, and I will not be adding any new features to them (ever).

LIST was replaced with VIEW years ago; for any heavy / complex use TPIPE is a better solution than FFIND. I will probably write a new (much faster) replacement for FFIND for a future version of TCC.
 
i seem to remember that we were told by you that ffind should be used like so /m /b /t"text" some file , I dont see any spaces between the optional switches. I know it still works but some consistency along syntax forming should be followed by the author no?
 
My email used spaces between the switches. The problem is 100% reproducible for me, so I don't know why Rex can't reproduce it. Vince said he can reproduce it. Regardless, I've switched to Cygwin's grep, which does not have the problem with long lines that FFIND has when using /T"xx" and /E"xx".
 
I can no longer reproduce it with build 32. I've tried a 12 MB file (test.txt) of random ASCII printables (32-126) and a 12 MB concatenation of 538 copies of my ApplicationPrivateSettings.xml (big.xml), both files having no newlines.

When I ask it to find non-existent test, I get results like Rex's.

Code:
c:\users\vefatica\appdata\local\microsoft\visualstudio\16.0_6b6f020f> timer & ffind /m/b/t"Page Not Found" big.xml & timer
Timer 1 on: 11:55:40
Timer 1 off: 11:55:40  Elapsed: 0:00:00.021

v:\> timer & ffind /m/b/t"Page Not Found" test.txt & timer
Timer 1 on: 11:56:05
Timer 1 off: 11:56:05  Elapsed: 0:00:00.021

When I ask it to find test which does exist and with the "/b" option, it finds it quickly.

Code:
v:\> timer & ffind /m/b/t"X" test.txt & timer
Timer 1 on: 11:58:56
V:\test.txt
Timer 1 off: 11:58:56  Elapsed: 0:00:00.016

And for existing test without "/b" it immediately spits out the one line (all 12 MB).

My only gripe is that it takes 3+ minutes to spit out that line (TYPE takes 1.7 second) and it can't be interrrupted.

Except for the speed of output and the fact that it can't be interrupted, it seems to be working correctly.
 
In another test with existing text and without "/b", where the text is near the end of the file, it's fast but only spits out the last 6934 bytes of the file. That might be WAD ... beats rewinding to find out where the line began (?).

Code:
v:\> timer & ffind /m/t"n7g=17Sy)Bb4" test.txt | wc & timer
Timer 1 on: 12:13:10
      3      86    6934
Timer 1 off: 12:13:10  Elapsed: 0:00:00.193

The two extra lines reported by wc.exe are an empty line and the header "---- V:\test.txt".
 
In build 32, it works for my 12 megabyte file. But, it hangs for a file of size 25 megabytes.

If there are unexpected limitations like this, it would save time (and forum posts) if the Help mentioned it.
 
Same as /E. If you do a /T that is not a reverse search, it is converted to a /E search (because it's 10x faster).
If you're going to do that you might massage the regex (search string) a bit (more) so Oniguruma doesn't choke on it (first below) or do the wrong thing (second below).

Code:
v:\> ffind /m/t"d:\tc26\1" sv.btm
TCC: invalid backref number/name "(?i)d:\tc26\1"

Code:
d:\tc26> ffind /m/t"d:\tc26" tcmd.ini

d:\tc26> ffind /m/t"d:\\tc26" tcmd.ini

---- D:\tc26\tcmd.ini
TCStartPath=d:\TC26
 
In build 32, it works for my 12 megabyte file. But, it hangs for a file of size 25 megabytes.

If there are unexpected limitations like this, it would save time (and forum posts) if the Help mentioned it.

OK. FFIND will also not work with files > 128 yottabytes.

And regarding your file - it's not a 25Mb file that's a problem, it's a (somewhat ridiculous) 25Mb single line.
 
Thank you, but I've already switched to Cygwin's grep for this, so please don't do it on my account. A note in the Help that ffind expects line endings every so often would suffice. (Although, it would be friendlier if /e and /t errored when you exceed the buffer limit rather than hanging.)
 

Similar threads

Back
Top