Very long startup times - solved for me

[Lippolt]

Hello !

I know there is alreay a thread about that theme, but my report has a slightly different direction. I also had increasing startup times (both for TCC and 4NT, version 10/61), with more than 5 minutes to wait, and no obvious extended disk activitiy. So I began to look in another places than loading big directory trees.

For me disabling the history files for command history (and directory history) solved the problem - I have now an instant start again.

Is there a known problem in loading back the history files into memory? Can other users also reduce their startup times by turning history files off?

Best regards from Hannover in Germany,

Peter

 

[rconn]

Lippolt wrote:

> Hello !
>
> I know there is alreay a thread about that theme, but my reports has a slightly different direction. I also had increasing startup times (both for TCC and 4NT, version 10/61), with more than 5 minutes to wait, and no obvious extended disk activitiy. So I began to look in another places than loading big directory trees.
>
> For me disabling the history files for command history (and directory history) solved the problem - I have now an instant start again.
>
> Is there a known problem in loading back the history files into memory? Can other users also reduce their startup times by turning history files off?

This should only be a problem if (1) you have REALLY big history files,
and (2) you have small history lists, so that TCC has to continually
delete the old entries and shuffle the list.

Rex Conn
JP Software

 

[dim]

On 2009-03-25 01:29, Lippolt wrote:

> Is there a known problem in loading back the history files into memory? Can other users also reduce their startup times by turning history files off?

Yes, it has also been excruciatingly slow for me. I have turned off
history loading a long time ago, since it didn't seem to get any better
with subsequent releases...

 

[Lippolt]

Lippolt wrote:



This should only be a problem if (1) you have REALLY big history files,
and (2) you have small history lists, so that TCC has to continually
delete the old entries and shuffle the list.

Rex Conn
JP Software

The history file was about 450 kbyte. Where can I set the size of the history list? Where can I limit the size of the history file?

What is your suggestion for reasonable settings? And where have they to be set?

Thanks for your fast reply,

Peter

 

[rconn]

Lippolt wrote:

> The history file was about 450 kbyte. Where can I set the size of the history list? Where can I limit the size of the history file?
>
> What is your suggestion for reasonable settings? And where have they to be set?

Depends on whether you're using global or local lists.

If they're local:

OPTION / Command Line / History Buffer Sizes.

If they're global, they're set to the value of the History and
DirHistory directives in the first TCC instance (default of 128K
characters for the history list and 4K for the directory history).

If you're doing a HISTORY /R, it adds the lines to the history one at a
time, which means it has to check HistoryExclude for each line, and (if
you have the HistDups option set) checking the entire history list for
matching lines. *Plus* deleting all of the old entries and shuffling
the list when you have a larger history file than history buffer.
(Plus, it has to lock & unlock the list for each line to prevent any
other process from accessing it while it's being updated.)

Rex Conn
JP Software

 

[vefatica]

On Tue, 24 Mar 2009 21:01:27 -0500, rconn <> wrote:

|If they're global, they're set to the value of the History and
|DirHistory directives in the first TCC instance (default of 128K
|characters for the history list and 4K for the directory history).

That's what I thought, and I have "History=131072" in my INI file. But OPTION
shows 133120. Another user reported a history file of over 400KB (over 200
characters). What's the limit? I couldn't find it in the help.
--
- Vince

 

[rconn]

vefatica wrote:

> On Tue, 24 Mar 2009 21:01:27 -0500, rconn <> wrote:
>
> |If they're global, they're set to the value of the History and
> |DirHistory directives in the first TCC instance (default of 128K
> |characters for the history list and 4K for the directory history).
>
> That's what I thought, and I have "History=131072" in my INI file. But OPTION
> shows 133120. Another user reported a history file of over 400KB (over 200
> characters). What's the limit? I couldn't find it in the help.

The history list sizes are always padded slightly in order to store
internal info. The maximum command history size is 500K, and the
maximum directory history size is 50K. (But IMO anybody who uses a
history that size is nuts!)

Rex Conn
JP Software

 

[Steve Fabian]

rconn wrote:
| Lippolt wrote:
|
|
|
| ---Quote---
|| The history file was about 450 kbyte. Where can I set the size of
|| the history list? Where can I limit the size of the history file?
||
|| What is your suggestion for reasonable settings? And where have they
|| to be set?
| ---End Quote---
| Depends on whether you're using global or local lists.
|
| If they're local:
|
| OPTION / Command Line / History Buffer Sizes.
|
| If they're global, they're set to the value of the History and
| DirHistory directives in the first TCC instance (default of 128K
| characters for the history list and 4K for the directory history).
|
| If you're doing a HISTORY /R, it adds the lines to the history one at
| a time, which means it has to check HistoryExclude for each line, and
| (if
| you have the HistDups option set) checking the entire history list for
| matching lines. *Plus* deleting all of the old entries and shuffling
| the list when you have a larger history file than history buffer.
| (Plus, it has to lock & unlock the list for each line to prevent any
| other process from accessing it while it's being updated.)

I suggest suboption for HISTORY /R to load the specified file without
checking for duplicates. This being an option it would be up to the user
whether or not eliminating duplicates from a history file are significant.

Another possible subpotion would be to keep the list locked continuously
until the HISTORY /R command is completed. This would certainly be harmless
when local lists are used. For those who use a global list this would only
affect those concurrent TCC instances where 1/ you manually execute commands
while another instance executes HISTORY /R or 2/ you concurrently execute
HISTORY/R. In both these situations the much reduced delay of not unlocking
and relocking for each data line would be generally worth the slightly
increased total delay.

In my own use, HISTORY/R is executed by a special, transient instance of
TCC, invoked when I log in as a user, and thereafter only manually. Each of
the suggested suboptions would speed up the login process, without any
aftereffects. Any further use of HISTORY /R is strictly manual.
--
Steve

 

[dim]

On 2009-03-25 03:43, rconn wrote:

> The history list sizes are always padded slightly in order to store
> internal info. The maximum command history size is 500K, and the
> maximum directory history size is 50K. (But IMO anybody who uses a
> history that size is nuts!)

Come on, 500 kiB is nothing these days; the average "soccer mom"
consumer now buys a computer at Best Buy with 1 or 2 GiB of RAM. You
are thinking in DOS-day terms. :)

Also, if you type commands for a few days, the accumulated history is
easily a few hundred kilobytes. Why should it throw away this old
history?

I have always logged all commands I typed, and this is just a few
megabytes per year, completely irrelevant with today's memory and
harddisk sizes.

 

[rconn]

dim wrote:

> On 2009-03-25 03:43, rconn wrote:
>
>
> ---Quote---
>> The history list sizes are always padded slightly in order to store
>> internal info. The maximum command history size is 500K, and the
>> maximum directory history size is 50K. (But IMO anybody who uses a
>> history that size is nuts!)
> ---End Quote---
> Come on, 500 kiB is nothing these days; the average "soccer mom"
> consumer now buys a computer at Best Buy with 1 or 2 GiB of RAM. You
> are thinking in DOS-day terms. :)

The amount of RAM available is irrelevant; the problem is when somebody
wants to have a large history and also have TCC do extensive processing
on it. (And then complain that processing their large list takes time!)

> Also, if you type commands for a few days, the accumulated history is
> easily a few hundred kilobytes. Why should it throw away this old
> history?

Why keep all of it when you'll never look at 99% of it again?

Rex Conn
JP Software

 

[rconn]

Steve Fábián wrote:

> I suggest suboption for HISTORY /R to load the specified file without
> checking for duplicates. This being an option it would be up to the user
> whether or not eliminating duplicates from a history file are significant.

Definitely NITV, as it would require a complete rewrite of HISTORY and
DIRHISTORY. (It would also result in a lot of code duplication, as
HISTORY and DIRHISTORY wouldn't be able to use the common routines for
history saves.)

> Another possible subpotion would be to keep the list locked continuously
> until the HISTORY /R command is completed.

This also requires a rewrite.

Rex Conn
JP Software

 

[Steve Fabian]

rconn wrote:
| Steve Fábián wrote:
|
|
|
| ---Quote---
|| I suggest suboption for HISTORY /R to load the specified file without
|| checking for duplicates. This being an option it would be up to the
|| user whether or not eliminating duplicates from a history file are
|| significant.
| ---End Quote---
| Definitely NITV, as it would require a complete rewrite of HISTORY and
| DIRHISTORY. (It would also result in a lot of code duplication, as
| HISTORY and DIRHISTORY wouldn't be able to use the common routines for
| history saves.)

Why would this be a complete rewrite? This would just temporarily imitate
the HistDups=OFF setting.


| ---Quote---
|| Another possible subpotion would be to keep the list locked
|| continuously until the HISTORY /R command is completed.
| ---End Quote---
| This also requires a rewrite.

I definitely would not expect such changes before a new version. OTOH,
wouldn't the same rules be applicable for both HISTORY/R and DIRHISTORY/R so
that code sharing could continue?
--
Steve

 

[Jay Sage]

I used to have code in TCEXIT.BTM that saved the entire command history
and directory history to files. Code in TCSTART.BTM would load them
using the /R option. Now that I understand how this slows down the
launch of TCMD, I now save only a specified small number of lines from
the ends of the histories. TCMD now starts up much faster.

It would be nice though (albeit in a future version) if there were a way
to load files directly into the command history and directory history
with no processing at all. My saved history files, after all, do not
need any additional processing; they were saved right out of the
existing history.

Alternatively, perhaps there could be separate facilities for saving
histories on exit from TCC and reloading them for new TCC tasks. It is
nice to be able to shut down TCMD or TCC tasks and later have it/them
come up with some of the state restored.

-- Jay

 

[Steve Fabian]

Jay Sage wrote:
| I used to have code in TCEXIT.BTM that saved the entire command
| history
| and directory history to files. Code in TCSTART.BTM would load them
| using the /R option. Now that I understand how this slows down the
| launch of TCMD, I now save only a specified small number of lines from
| the ends of the histories. TCMD now starts up much faster.
|
| It would be nice though (albeit in a future version) if there were a
| way
| to load files directly into the command history and directory history
| with no processing at all. My saved history files, after all, do not
| need any additional processing; they were saved right out of the
| existing history.
|
| Alternatively, perhaps there could be separate facilities for saving
| histories on exit from TCC and reloading them for new TCC tasks. It is
| nice to be able to shut down TCMD or TCC tasks and later have it/them
| come up with some of the state restored.

I save command and directory history only through the automatic mechanism
built into SHRALIAS.EXE in recent versions, i.e., when SHRALIAS.EXE is
terminated, all global tables are saved. I do use all four tables as
globals. Whenever I log in into a WinXP session, a shortcut in
%allusers\start menu\programs\startup starts a transient TCC instance, which
loads all four global tables from the files where they were saved. Once
that's done, SHRALIAS.EXE keeps them alive, so no new TCC instances ever
need to load them again. In this manner the slow behavior of the HISTORY/R
command occurs only on rare occasions (averaging less than once a day). Note
that the same instance of TCC could instead be retained until logout, and
used to run the various MONITOR commands (that is, be the "master thread"
for them). It could also serve to dispatch other tasks either based on time
of day, or on the occurrence of other events.
--
Steve

 

[Jay Sage]

> I save command and directory history only through the
> automatic mechanism built into SHRALIAS.EXE

Steve, thank you so much for that information! Perhaps I was once aware
of that SHRALIAS feature (I have defined a SHRALIAS_SAVE_PATH variable),
but I had forgotten. I've now reworked my TCSTART and TCEXIT scripts to
omit the loading and saving if SHRALIAS is running.

-- Jay

 

[drwtsn32]

Wild... 5 minutes or more to start TCC? My history is 32K and TCC starts instantly (with loading the history in TCSTART.BTM). I can't imagine it taking 5+ minutes even if your history is 15 times the size of mine.