[M3devel] what to do about file sizes being 32bits?

Fri Jan 8 02:18:51 CET 2010

[truncated]

To repeat myself, basically every system supports 64bit file

sizes and they are easy to use from C.
In my opinion, a "systems" language is something you

can do "anything" in.
Now, granted, the language isn't the issue here.
But conflation of libraries and languages is common

and not entirely invalid.

 - Jay

From: jay.krell at cornell.edu
To: hosking at cs.purdue.edu
CC: m3devel at elegosoft.com
Subject: RE: [M3devel] what to do about file sizes being 32bits?
Date: Fri, 8 Jan 2010 01:07:23 +0000

 > As you are already aware, some systems have only 32-bit addressable files.

Really?!
I don't think any system that we currently support or almost any that we would forseably lacks ability to use files larger than 4G.
4G has not been a large file size for quite a long time now.
FreeBSD, NetBSD, NT, Linux, Solaris, AIX, Irix, VMS, HP-UX. 64bit file sizes in 32bit code.
Maybe not MS-DOS, maybe not Win9x, but maybe esp. via remote file systems.

 > If we go the route you suggest then LONGINT will escape from a few isolated uses into the rest of the system.

I can build "std" now. It does escape pretty far, but it is actually fairly contained.
It would help a little of an INTEGER could be assigned to a LONGINT.
Then the "porting" work for Rd/Wr users is just to add ORD on Index/Length calls.
Currently they also need VAL(foo, LONGINT) also for Seek calls, which could be eliminated if..

 > You are conflating OS "files" with Modula-3 Rd/Wr.T.  

Isn't that pretty natural?
Aren't "file" one of the main things used with Rd/Wr?
Besides that, if you consider "streams", going beyond 4G seems even more limiting.
You know, I didn't change Read/Write/Get/whatever to use LONGINT. They require an in-memory buffer and therefore must use INTEGER or CARDINAL.

I agree that dealing with such large files can be difficult, depending on the algorithms.
But the "base" of the system shouldn't remove the ability for higher parts to do so.

And resorting to C seems unnecessary.
The operating system interfaces are essentially identical on Posix, as long you #define whatever.
They vary at the ABI level for compatibility.

Should I create a branch, or can we review the diffs and decide?

 - Jay

From: hosking at cs.purdue.edu
Date: Thu, 7 Jan 2010 16:33:41 -0500
To: jay.krell at cornell.edu
CC: m3devel at elegosoft.com
Subject: Re: [M3devel] what to do about file sizes being 32bits?

What do you mean by "files"?  You are conflating OS "files" with Modula-3 Rd/Wr.T.  They are not necessarily the same.  Modula-3 is designed to be OS-independent.  As you are already aware, some systems have only 32-bit addressable files.

If we go the route you suggest then LONGINT will escape from a few isolated uses into the rest of the system.  This seems overkill.  With Rd/Wr.Length typed as INTEGER then on 64-bit systems you will have 64-bit addressable files.  Elsewhere, 32-bit.

I am really starting to think the addition of LONGINT was a big mistake.

On 7 Jan 2010, at 15:31, Jay K wrote:

Given that files are in fact larger than 4GB, why should we impose such a limit?
Doesn't it make for a pretty lame system?

Pretty much no existing 32bit C system for many years now any such limit and
C started going past these limits around 15 years ago.

It turns out changes I sent were pretty nearly done. I can now build all of "std"
with LONGINT for file sizes. It's not just Rd/Wr, though that is most of it, it is also "File".

 - Jay

Subject: Re: [M3devel] what to do about file sizes being 32bits?
From: hosking at cs.purdue.edu
Date: Thu, 7 Jan 2010 14:17:40 -0500
CC: jay.krell at cornell.edu; m3devel at elegosoft.com
To: hosking at cs.purdue.edu

I guess what I am really saying is that it does not seem unreasonable to me that Modula-3 (as a language) should be able to restrict the Rd.T and Wr.T instances to have a length that is always expressible as INTEGER.  That was the point of my question about eliminating LONGINT.  With the C interfaces abstracted, LONGINT would no longer be necessary.

Antony Hosking | Associate Professor | Computer Science | Purdue University

305 N. University Street | West Lafayette | IN 47907 | USA
Office     +1 765 494 6001  +1 765 494 6001    +1 765 494 6001  +1 765 494 6001      +1 765 494 6001  +1 765 494 6001    +1 765 494 6001  +1 765 494 6001        +1 765 494 6001  +1 765 494 6001    +1 765 494 6001  +1 765 494 6001      +1 765 494 6001  +1 765 494 6001    +1 765 494 6001  +1 765 494 6001    | Mobile     +1 765 427 5484  +1 765 427 5484    +1 765 427 5484  +1 765 427 5484      +1 765 427 5484  +1 765 427 5484    +1 765 427 5484  +1 765 427 5484        +1 765 427 5484  +1 765 427 5484    +1 765 427 5484  +1 765 427 5484      +1 765 427 5484  +1 765 427 5484    +1 765 427 5484  +1 765 427 5484  

On 7 Jan 2010, at 14:07, Tony Hosking wrote:

Jay,

I am *very* concerned that these changes are damaging the clarity of the language and its libraries in bad ways.  What is so unreasonable about having the Modula-3 library interfaces place restrictions that continue to impose INTEGER file sizes.  Just because the underlying OS allows files larger than that, if a program creates and manipulates files through the standard interfaces then they cannot ever see file sizes bigger than INTEGER.  The interfaces support failure of Put operations on Wr.T that permit us to fail when exceeding the restricted file size.  Please do not make these changes in the mainline trunk until further discussion.  If you want to propose a set of changes I strongly suggest that you place them in a branch for evaluation by others!

-- Tony

On 7 Jan 2010, at 12:26, Jay K wrote:

> Not in the current release

Agreed.

I think I'll have this done in the next few days, with the
major caveat that it does break a lot of code. I'll fix the cm3 tree.

The breakage is roughly:

rd/wr users:
before:
  a := Rd.Length(b);
  c := Rd.Index(b);
  Rd.Seek(b, d);

after:
  a := ORD(Rd.Length(b));
  c := ORD(Rd.Index(b));
  Rd.Seek(b, VAL(d, LONGINT));

Though I think the last line should just work without change.

rd/wr implementers:
 more substantial changes, but still similar, lots of ORD/VAL, and "L".

One more compatible alternative might be to add LengthL, IndexL, SeekL?
Maybe only break rd/wr implementers but not users?

The reason I don't like that though is that it is even more of a no-op.
Nobody will switch to the new functions.
Similar to how "today" everybody will just ORD/VAL over the difference.

To be clear though, the time for this change was 10 or 20 years ago.
Now, 32bit systems are going away and with them this problem.
(Amazingly, some 64bit systems still have 32bit time_t, like I think OpenBSD..)

 - Jay

> Date: Thu, 7 Jan 2010 14:52:10 +0100
> From: wagner at elegosoft.com
> To: m3devel at elegosoft.com
> Subject: Re: [M3devel] what to do about file sizes being 32bits?
> 
> Quoting hendrik at topoi.pooq.com:
> 
> > On Thu, Jan 07, 2010 at 06:59:31AM +0000, Jay K wrote:
> >>
> >> File.i3:
> >>
> >>
> >> Status = RECORD
> >> type: Type;
> >> modificationTime: Time.T;
> >> size: CARDINAL (* oops... *)
> >> END;
> >>
> >>
> >> What to do?
> >> [0.. higher than 7FFFFFFF] doesn't "just work".
> >> higher than 7FFFFFFFF is not legal on 32bit, unless you put "L" 
> >> on the end,
> >> which presumably has some relationship to turning it into a 
> >> LONGINT, which
> >> causes users to fail to compile
> >
> > In any case, is the proper type for file offsets [0..7fffffffffffffff]
> > or [0..ffffffffffffffff]? I suspect the latter. It might take some
> > effort to make that legal in Modula 3.
> 
> Well, I don't think that should be any practical problem right now,
> shouldn't it? But 32 bit offsets have been overcome for years even
> on 32 bit systems, so I definitely think we should keep the LONGINT
> type and even try to incompatibly change the internal file length
> type (with due care and consideration of course).
> 
> And please not for the still unfinished release, but for the next
> one.
> 
> Olaf
> -- 
> Olaf Wagner -- elego Software Solutions GmbH
> Gustav-Meyer-Allee 25 / Gebäude 12, 13355 Berlin, Germany
> phone:     +49 30 23 45 86 96  +49 30 23 45 86 96    +49 30 23 45 86 96  +49 30 23 45 86 96      +49 30 23 45 86 96  +49 30 23 45 86 96    +49 30 23 45 86 96  +49 30 23 45 86 96        +49 30 23 45 86 96  +49 30 23 45 86 96    +49 30 23 45 86 96  +49 30 23 45 86 96      +49 30 23 45 86 96  +49 30 23 45 86 96    +49 30 23 45 86 96  +49 30 23 45 86 96    mobile:     +49 177 2345 869  +49 177 2345 869    +49 177 2345 869  +49 177 2345 869      +49 177 2345 869  +49 177 2345 869    +49 177 2345 869  +49 177 2345 869       +49 177 2345 869  +49 177 2345 869    +49 177 2345 869  +49 177 2345 869      +49 177 2345 869  +49 177 2345 869    +49 177 2345 869  +49 177 2345 869    fax: +49 30 23 45 86 95
> http://www.elegosoft.com | Geschäftsführer: Olaf Wagner | Sitz: Berlin
> Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://m3lists.elegosoft.com/pipermail/m3devel/attachments/20100108/733c2790/attachment-0002.html>