1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Fixed TYPE http://... incomplete data

Discussion in 'Support' started by vefatica, Apr 7, 2014.

  1. vefatica

    Joined:
    May 20, 2008
    Messages:
    8,066
    Likes Received:
    30
    On my computer at work, with TCCD16, I get this (below). It's quite incomplete, both at the beginning and at the end. View the page in a browser to see all that I should be getting (a lot more). It works correctly on the same computer with TCC v15, v14, v13, ...
    Code:
    v:\> type http://w1.weather.gov/data/obhistory/KSYR.html
     size="2" noshade></td>  </tr><tr valign="top"><td class="gray" align="left">National Weather Service<br>Southern Region Headquarters<br>
      Fort Worth, Texas<br><ahref="http://weather.gov/disclaimer">Disclaimer</a><br></td>
      <td><form><input type="button" value="Back to previous page" onClick="history.back()"></form></td><td align="right" class="gray">
      Last Modified: Febuary,7 2012<br><a href="http://weather.gov/privacy">Privacy Policy</a></td>
      </tr></table></body></html>
     
  2. vefatica

    Joined:
    May 20, 2008
    Messages:
    8,066
    Likes Received:
    30
    Here's a PNG showing the same command twice in a row ... with different and incomplete results. It works fine with earlier versions of TCC. weather.png
     
  3. vefatica

    Joined:
    May 20, 2008
    Messages:
    8,066
    Likes Received:
    30
    Here's another example showing that "TYPE http://..." is undependable.
    Code:
    v:\> type http://www.jpsoft.com | wc -c
      23327
    
    v:\> type http://www.jpsoft.com | wc -c
      2041
    
    v:\> type http://www.jpsoft.com | wc -c
      23327
    
    v:\> type http://www.jpsoft.com | wc -c
      23327
    
    v:\> type http://www.jpsoft.com | wc -c
      2049
     
  4. Steve Fabian

    Joined:
    May 20, 2008
    Messages:
    3,520
    Likes Received:
    4
    I presume wc is the POSIX "word count" with the option -c instructing it to report the character count of the file, at the report sometimes returns the actual file size, and sometimes a value just over 2000 characters. If this is the correct interpretation of your post, it is exactly the same issue - well, exactly the same symptoms - that I have been struggling with "copy http://jpsoft.com/downloads/v16/tcmdx64.exe" (and also tcmd.exe). However, I wonder that the "root" of the website is unchanged between the tests?
     
  5. Charles G

    Joined:
    Apr 2, 2011
    Messages:
    1,037
    Likes Received:
    0
    Steve are your results the same if you specify www.jpsoft.com and not just jpsoft.com ?
     
  6. Steve Fabian

    Joined:
    May 20, 2008
    Messages:
    3,520
    Likes Received:
    4
    No, I tried both ways (with and without www) in v14, v15 and v16, all the same size, but 6 different CRC32s. I suspect (not being a web maven) that the root has a code (current time? viewer count?) that is not displayed, which would explain this. What is not explained is when instead of 23,327 characters only a bit more than 2,000 characters showed in Vince's OP. One would need to create a known, unalterable web page, and downloaded many times to validate the issue. Sorry, no can do, but would like to learn the results.
     
  7. vefatica

    Joined:
    May 20, 2008
    Messages:
    8,066
    Likes Received:
    30
    Here's another, missing data at both ends. V15 gets it right every time.
    Code:
    v:\> type http://www.google.com
    |0<=a.indexOf("#q="))&&(a=a.substring(1),-1==a.indexOf("#"))){for(var d=0;d<a.le
    ngth;){var b=d;"&"==a.charAt(b)&&++b;var c=a.indexOf("&",b);-1==c&&(c=a.length);
    b=a.substring(b,c);if(0==b.indexOf("fp="))a=a.substring(0,d)+a.substring(c,a.len
    gth),c=d;else if("cad=h"==b)return 0;d=c}_gjwl.href="/search?"+a+"&cad=h";return
    1}return 0}
    function _gjh(){!_gjuc()&&window.google&&google.x&&google.x({id:"GJH"},function(
    ){google.nav&&google.nav.gjh&&google.nav.gjh()})};
    window._gjh&&_gjh();</script><style>#gbar,#guser{font-size:13px;padding-top:1px
    !important;}#gbar{height:22px}#guser{padding-bottom:7px !important;text-align:ri
    ght}.gbh,.gbd{border-top:1px solid #c9d7f1;font-size:1px}.gbh{height:0;position:
    absolute;top:24px;width:100%}@media all{.gb1{height:22px;margin-right:.5em;verti
    cal-align:top}#gbar{float:left}}a.gb1,a.gb4{text-decoration:underline !important
    }a.gb1,a.gb4{color:#00c !important}.gbi .gb4{color:#dd8e27 !important}.gbf .gb4{
    color:#900 !important}</style><style>body,td,a,p,.h{font-family:arial,sans-serif
    }body{margin:0;overflow-y:scroll}#gog{padding:3px 8px 0}td{line-height:.8em}.gac
    _m td{line-height:17px}form{margin-bottom:20px}.h{color:#36c}.q{color:#00c}.ts t
    d{padding:0}.ts{border-collapse:collapse}em{font-weight:bold;font-style:normal}.
    lst{height:25px;width:496px}.gsfi,.lst{font:18px arial,sans-serif}.gsfs{font:17p
    x arial,sans-serif}.ds{display:inline-box;display:inline-block;margin:3px 0 4px;
    margin-left:4px}input{font-family:inherit}a.gb1,a.gb2,a.gb3,a.gb4{color:#11c !im
    portant}body{background:#fff;color:black}a{color:#11c;text-decoration:none}a:hov
    er,a:active{text-decoration:underline}.fl a{color:#36c}a:visited{color:#551a8b}a
    .gb1,a.gb4{text-decoration:underline}a.gb3:hover{text-decoration:none}#ghead a.g
    b2:hover{color:#fff !important}.sblc{padding-top:5px}.sblc a{display:block;margi
    n:2px 0;margin-left:13px;font-size:11px}.lsbb{background:#eee;border:solid 1px;b
    order-color:#ccc #999 #999 #ccc;height:30px}.lsbb{display:block}.ftl,#fll a{disp
    lay:inline-block;margin:0 12px}.lsb{background:url(/images/srpr/
    v:\>
     
  8. vefatica

    Joined:
    May 20, 2008
    Messages:
    8,066
    Likes Received:
    30
    FWIW, when v16 uses the ipw9 DLLs from the v15 distribution, there are no errors.
    And when v15 uses the ipw9 DLLs from the v16 distribution, the errors mentioned in my earlier posts occur.
     
  9. vefatica

    Joined:
    May 20, 2008
    Messages:
    8,066
    Likes Received:
    30
    You could try http://lucky.syr.edu (an HTTP front-end for the FTP server). I don't think it changes. Here, the v16/v15 difference is striking:
    Code:
    d:\tc16> ver & do i=1 to 10 (type http://lucky.syr.edu | wc -c)
    
    TCC  16.01.45  Windows 7 [Version 6.1.7601]
      2049
      2049
      2049
      2049
      2049
      2049
      2049
      2049
      2049
      2049
    Code:
    d:\tc15> ver & do i=1 to 10 (type http://lucky.syr.edu | wc -c)
    
    TCC  15.01.57  Windows 7 [Version 6.1.7601]
      37861
      37861
      37861
      37861
      37861
      37861
      37861
      37861
      37861
      37861
     
  10. Charles Dye

    Charles Dye Super Moderator
    Staff Member

    Joined:
    May 20, 2008
    Messages:
    3,355
    Likes Received:
    39
    Less consistent here:

    Code:
    C:\>ver & do i=1 to 10 (type http://lucky.syr.edu | wc -c)
    
    TCC  16.01.45  Windows XP [Version 5.1.2600]
      37861
      37861
      2048
      37861
      2048
      2048
      37861
      1986
      2048
      37861
    
    C:\>
    
    I think Steve's right. This isn't an issue with TYPE, it's an IPWorks issue which also affects COPY, and possibly other commands as well.
     
  11. nickles

    Joined:
    Jun 24, 2008
    Messages:
    220
    Likes Received:
    0
    I recently encountered a similar issue with ftp:. Copying in some RAR archive from our file server left me with an unreadable file (using ftp.exe worked with no problems).
    Seems there's a bug somewhere.
     
  12. Charles Dye

    Charles Dye Super Moderator
    Staff Member

    Joined:
    May 20, 2008
    Messages:
    3,355
    Likes Received:
    39
    LIST seems more reliable, but after trying list /w http://lucky.syr.edu numerous times, I have gotten a partial file on a few transfers. Oddly, the start of the file seems to be lost at least as often as the end.
     
  13. Charles G

    Joined:
    Apr 2, 2011
    Messages:
    1,037
    Likes Received:
    0
    I'd like to try the test above but don't have anything similar to wc - so would someone please send me a copy....
     
  14. JohnQSmith

    Joined:
    Jan 19, 2011
    Messages:
    559
    Likes Received:
    7
    Install Cygwin. It's part of the base load.
     
  15. vefatica

    Joined:
    May 20, 2008
    Messages:
    8,066
    Likes Received:
    30
    I prefer UnxUtils. In the absence of either, do it with your bare hands!
    Code:
    v:\> ver & do i=1 to 5 (copy /q http://lucky.syr.edu tmp.tmp & echo %@filesize[tmp.tmp]) & del /q tmp.tmp
    
    TCC  16.01.45  Windows 7 [Version 6.1.7601]
    2048
    2048
    2048
    2048
    2048
    Code:
    d:\tc15> ver & do i=1 to 5 (copy /q http://lucky.syr.edu tmp.tmp & echo %@filesize[tmp.tmp]) & del /q tmp.tmp
    
    TCC  15.01.57  Windows 7 [Version 6.1.7601]
    37861
    37861
    37861
    37861
    37861
     
  16. rconn

    rconn Administrator
    Staff Member

    Joined:
    May 14, 2008
    Messages:
    10,029
    Likes Received:
    84
    It's a bug in the IPWorks dll. I contacted the developers and they have sent me a fix. I will upload a new version in the next day or two (after running some regression tests).
     
  17. JohnQSmith

    Joined:
    Jan 19, 2011
    Messages:
    559
    Likes Received:
    7
    I find the UnxUtils to be several versions out of date.
    I tried doing that using the @LEN[] function, but I guess I need a little assistance figuring that out. I tried things like
    Code:
    echo %@LEN[type http://lucky.syr.edu]
    which gave me "25" and then
    Code:
    echo %@LEN[%@EXEC[type http://lucky.syr.edu]]
    which spewed out the source of the website, but then returned "1" since it returned the length of the return value from the @EXEC function instead of the length of the site code.

    There's gotta be a TCC command/function I'm missing.
     
  18. vefatica

    Joined:
    May 20, 2008
    Messages:
    8,066
    Likes Received:
    30
    I think you're thinking of @LEN[@EXECSTR[ ... but that @EXECSTR gives only one line of the command's output (and @EXEC gives only the command's return value).
     
  19. JohnQSmith

    Joined:
    Jan 19, 2011
    Messages:
    559
    Likes Received:
    7
    I tried @EXECSTR[] too. But, as you pointed out, it only returns the first line, which doesn't include the whole site.
     
  20. JohnQSmith

    Joined:
    Jan 19, 2011
    Messages:
    559
    Likes Received:
    7
    Looking through the TCC documentation, I found in the Reference > Limits... then under "Miscellaneous Limits (characters)"... "character count in any function" = "20,000". Does that mean the @LEN function would fail anyway since the website is 37k? Or is that talking about the length of user defined functions rather than the input to functions?

    Edit: Answered my own question. The @LEN function would fail.
    Code:
    C:\temp> echo %@LEN[%@REPEAT[x,20000]]
    20000
    
    C:\temp> echo %@LEN[%@REPEAT[x,20010]]
    TCC: (Sys) The parameter is incorrect.
    "%@REPEAT[x,20010]"
    20003 characters seems to work, but it fails at 20004 or higher.

    Edit 2: Actually, it's the @REPEAT function that's failing above 20003. The @LEN function seems to be able to handle it. I'm not sure if it will work in the end since I got the following
    Code:
    C:\temp> echo %@LEN[%@REPEAT[x,20000] %@REPEAT[y,20000]]
    TCC: Command line too long
    TCC has a 32k command line length limit.
     
    #20 JohnQSmith, Apr 9, 2014
    Last edited: Apr 9, 2014
  21. vefatica

    Joined:
    May 20, 2008
    Messages:
    8,066
    Likes Received:
    30
    @REPEAT has a documented count-limit of 32K. So something ain't right!

    That's a bug in either the function or the documentation. You ought to start a new "bug" thread about it.
     
  22. vefatica

    Joined:
    May 20, 2008
    Messages:
    8,066
    Likes Received:
    30
    P.S. It seems OK in v14 (but not in v15).
     
  23. Steve Fabian

    Joined:
    May 20, 2008
    Messages:
    3,520
    Likes Received:
    4
    Maybe someone will use "feedback" to request that @FILESIZE[] support HTTP? Not I!
     
  24. JohnQSmith

    Joined:
    Jan 19, 2011
    Messages:
    559
    Likes Received:
    7
    But, Rex loves you! He always enjoys exchanging pleasantries with you about your ideas.
     
  25. Charles Dye

    Charles Dye Super Moderator
    Staff Member

    Joined:
    May 20, 2008
    Messages:
    3,355
    Likes Received:
    39
    Don't even need to ask. HTTP doesn't provide any way to report file sizes.
     
  26. Steve Fabian

    Joined:
    May 20, 2008
    Messages:
    3,520
    Likes Received:
    4
    But it is possible to copy the whole file just to find out its size and immediately discard the copy... One can even go fancier and just count the bytes without saving them as you download them. Of course, unless one has infinite bandwidth it is a lot of wasted effort. What would be totally impossible is to obtain file age. After all, HyperText Transfer Protocol is designed for text, not files. Any tricks to verify content, for example, installer integrity and software updating. is non-standard, custom grafted onto the standard itself.
     
  27. vefatica

    Joined:
    May 20, 2008
    Messages:
    8,066
    Likes Received:
    30
    I'm not so sure about that. I found a few examples (none in C/C++) that uses Wininet to get the size of **something** (maybe it's the file size). They involve:
    hInet = InternetOpen(szMyName, INTERNET_OPEN_TYPE_PRECONFIG, ...)
    hConnect = InternetConnect(szServerName, INTERNET_DEFAULT_HTTP_PORT, ...)
    hRequest = HttpOpenRequest(hInternet, L"HEAD", szResourceName, ...)
    HttpSendRequest(hRequest, ...)
    HttpQueryInfo(hRequest, HTTP_QUERY_CONTENT_LENGTH, pResult, ...)

    I'll play with it a bit.
     
  28. Charles Dye

    Charles Dye Super Moderator
    Staff Member

    Joined:
    May 20, 2008
    Messages:
    3,355
    Likes Received:
    39
    I look forward to being proven wrong....
     
  29. vefatica

    Joined:
    May 20, 2008
    Messages:
    8,066
    Likes Received:
    30
    OK! :-)
    The very sketchy test code (far) below (used in place of 4UTILS's _DESK function) tells me that the size of "http://jpsoft.com/downloads/tcmd.exe" is ...
    Code:
    l:\projects\4utils\release> timer echo %_desk
    Timer 1 on: 22:24:38
    21534064
    Timer 1 off: 22:24:38  Elapsed: 0:00:00.00
    
    And that's correct. And I'm pretty sure it didn't download the file in that brief time (I was watching with a sniffer anyway ... and it didn't). I won't have any time soon to try to make a robust plugin out of this.
    Code:
    #include <wininet.h>
    INT WINAPI _DESK ( WCHAR *psz )
    {
       HINTERNET hInet = InternetOpen(L"JPUser", INTERNET_OPEN_TYPE_PRECONFIG, NULL, NULL, 0);
       HINTERNET hConnect = InternetConnect(hInet, L"jpsoft.com", INTERNET_DEFAULT_HTTP_PORT, NULL, NULL, INTERNET_SERVICE_HTTP, 0, 0);
       HINTERNET hRequest = HttpOpenRequest(hConnect, L"HEAD", L"/downloads/v16/tcmd.exe", NULL, NULL, NULL, 0, 0);
       HttpSendRequest(hRequest, NULL, 0, NULL, 0);
       WCHAR Buffer[32];
       DWORD Len = 32 * sizeof(WCHAR), Index = 0;
       HttpQueryInfo(hRequest, HTTP_QUERY_CONTENT_LENGTH, Buffer, &Len, &Index);
       Sprintf(psz, L"%s\r\n", Buffer);
       return 0;
    }
    
     
  30. vefatica

    Joined:
    May 20, 2008
    Messages:
    8,066
    Likes Received:
    30
    That is apparently limited to **files**. I don't think it will tell you how much data would be sent if you "TYPE http://www.jpsoft.com".
    And (obviously) there's no error checking in that code. And I didn't close any of the HINTERNETs. And I didn't need the "\r\n" in the result (it was left over from a Printf()).
     
    #30 vefatica, Apr 9, 2014
    Last edited: Apr 9, 2014

Share This Page