[M3devel] TextLiteral.MaxBytes again (unable to compile 64bit target on 32bit host)

[Rodney M. Bates]

On 06/03/2015 06:18 PM, Jay K wrote:
> Should we just be a little lax and say minus 64?
> Or pick some other number like 256MB?
>

Sure, just to allow for unexpected changes.  My principle here was to make
it as large as possible, subject to accommodating both native word sizes,
in pickling, compiling (including cross compiling), etc.  Plus, if it
changes, then pickles need to be adapted, so I hope to get it to something
we can keep.

Do subtract a number that is one less than a multiple of 4, so MaxBytes comes
out a multiple of 4.  This is pretty confusing.  What is really wanted is
MaxBytes = (2^31) DIV BITSIZE (Byte) - <someMultipleOf4>,
but I'm not sure compile-time DIV for this overflow case is well-defined, so
16_7FFFFFFF DIV BITSIZE(Byte) + 1 gives the max multiple of 4 bytes whose bit
size is <= LAST(Int32).  Then subtract another multiple of 4.  So maybe
16_7FFFFFFF DIV BITSIZE(Byte) + 1 - 64 is easier to understand.

> Can we declare there is no bound?
>

No, we're writing code that always ignores the type's bound anyway and uses the
cnt field instead.  We just want the type's bound as high as possible, to
minimize the coding trouble of avoiding the compiler's checks.  This is just
the kind of low-level coding that the language's safe subset is designed to
disallow.

> Thank you,
>   - Jay
>
>
>  > Date: Wed, 3 Jun 2015 14:23:07 -0500
>  > From: rodney_bates at lcwb.coop
>  > To: m3devel at elegosoft.com; jay.krell at cornell.edu
>  > Subject: Re: [M3devel] TextLiteral.MaxBytes again (unable to compile 64bit target on 32bit host)
>  >
>  >
>  >
>  > On 06/02/2015 09:51 PM, Jay K wrote:
>  > > Ideally:
>  > > 1 32bit host could target 64bit target
>  > > 2 32bit texts could be as large as the address space/OS allows.
>  > > 3 ditto 64bit texts -- larger than 4GB
>  > > 4 be unpicklable between 32bit and 64bit, as long as they fit into the address space.
>  > > Currently only #4 is true.
>  > > TEXTs for any target can only be about 500MB.
>  > >
>  > >
>  > > To fix 2 and 3 requires m3front to use TInt more.
>  > >
>  > > This fixes #1 and likely breaks #4. Cutting the limit just a little.
>  > > Can we special case somehow?
>  > > Is there some construct we can use that has no stated limit, like 0..cnt - 1?
>  > > Should/can we introduce one?
>  > >
>  > > jbook2:src jay$ git diff "../src/text/TextLiteral.i3"
>  > > diff --git a/m3-libs/m3core/src/text/TextLiteral.i3 b/m3-libs/m3core/src/text/TextLiteral.i3
>  > > index fa72589..4d16a44 100644
>  > > --- a/m3-libs/m3core/src/text/TextLiteral.i3
>  > > +++ b/m3-libs/m3core/src/text/TextLiteral.i3
>  > > @@ -14,7 +14,7 @@ IMPORT RTHooks, TextClass;
>  > > CONST
>  > > (* DIV BITSIZE should not be here! *)
>  > > (* MaxBytes = LAST (INTEGER) DIV BITSIZE (Byte) - 7 - 8 * ORD(BITSIZE(INTEGER) = 64); *)
>  > > - MaxBytes = 16_7FFFFFFF DIV BITSIZE (Byte) - 7;
>  > > + MaxBytes = 16_7FFFFFFF DIV BITSIZE (Byte) - 15;
>  >
>  > I'd say go ahead and change this. We really do want it to cross-compile
>  > sooner than anybody will be able to put TInt in all over the place.
>  >
>  > I did not build a cross compiler, but did what I think should be a realistic
>  > simulation of it by changing the cnt:INTEGER field to two INTEGER fields and
>  > compiling with a 32-bit compiler. That would be the same size as a 64-bit
>  > INTEGER. It requires MaxBytes <= 16_7FFFFFFF DIV BITSIZE (Byte) - 11 to
>  > compile, with bit size having space for 31 more bits before it overflows.
>  > This byte count of buf is a multiple of 4, which we want for unicode-range
>  > WIDECHAR. Making it -10 overflows, presumably because the additional byte is
>  > padded out to a multiple of 32 bits.
>  >
>  > I would have expected that somewhere, a size including the one-word heap header
>  > would be computed, but these are never actually heap allocated, and it seems to
>  > work. But to hedge against things, I suggest using -19. This would allow space
>  > to count it, cross-compiling 32->64.
>  >
>  > This will allow a text literal of 268435436 ISO characters, and a wide text literal
>  > of 1/4 that, which is not likely to be a serious limitation, considering that it
>  > all has to be on a single line of source code.
>  >
>  > As for pickles, the horse is already out of the barn on that, as pickles could
>  > already have been written with the current fingerprint, and we can't change the
>  > type in any way without the fingerprint changing. It is not hard to add
>  > recognition of the old fingerprint to pickles. Right off hand, I don't remember
>  > the process for finding actual fingerprint values, but I've done it before and
>  > can rediscover it.
>  >
>  > > (* - 8 * ORD(BITSIZE(INTEGER) = 64) *)
>  > > (* Do not adjust this for INTEGER size. It makes T have different
>  > > fingerprints on 32- and 64-bit machines, which undermines pickling/
>  > >
>  > >
>  > >
>  > > otherwise we have:
>  > >
>  > >
>  > > new source -> compiling TextLiteral.i3
>  > > "../src/text/TextLiteral.i3", line 27: CM3 restriction: record or object type is too large
>  > > 1 error encountered
>  > >
>  > >
>  > > Ok?
>  > > - Jay
>  >
>  > --
>  > Rodney Bates
>  > rodney.m.bates at acm.org

-- 
Rodney Bates
rodney.m.bates at acm.org