[M3devel] TextLiteral.MaxBytes again (unable to compile 64bit target on 32bit host)

[Rodney M. Bates]

On 06/02/2015 09:51 PM, Jay K wrote:
>   Ideally:
>    1 32bit host could target 64bit target
>    2 32bit texts could be as large as the address space/OS allows.
>    3 ditto 64bit texts -- larger than 4GB
>    4 be unpicklable between 32bit and 64bit, as long as they fit into the address space.
>   Currently only #4 is true.
>   TEXTs for any target can only be about 500MB.
>
>
>   To fix 2 and 3 requires m3front to use TInt more.
>
>   This fixes #1 and likely breaks #4. Cutting the limit just a little.
>   Can we special case somehow?
>   Is there some construct we can use that has no stated limit, like 0..cnt - 1?
>   Should/can we introduce one?
>
> jbook2:src jay$ git diff  "../src/text/TextLiteral.i3"
> diff --git a/m3-libs/m3core/src/text/TextLiteral.i3 b/m3-libs/m3core/src/text/TextLiteral.i3
> index fa72589..4d16a44 100644
> --- a/m3-libs/m3core/src/text/TextLiteral.i3
> +++ b/m3-libs/m3core/src/text/TextLiteral.i3
> @@ -14,7 +14,7 @@ IMPORT RTHooks, TextClass;
>   CONST
>    (* DIV BITSIZE should not be here! *)
>    (* MaxBytes = LAST (INTEGER) DIV BITSIZE (Byte) - 7 - 8 * ORD(BITSIZE(INTEGER) = 64); *)
> - MaxBytes = 16_7FFFFFFF DIV BITSIZE (Byte) - 7;
> + MaxBytes = 16_7FFFFFFF DIV BITSIZE (Byte) - 15;

I'd say go ahead and change this.  We really do want it to cross-compile
sooner than anybody will be able to put TInt in all over the place.

I did not build a cross compiler, but did what I think should be a realistic
simulation of it by changing the cnt:INTEGER field to two INTEGER fields and
compiling with a 32-bit compiler.  That would be the same size as a 64-bit
INTEGER.  It requires MaxBytes <= 16_7FFFFFFF DIV BITSIZE (Byte) - 11 to
compile, with bit size having space for 31 more bits before it overflows.
This byte count of buf is a multiple of 4, which we want for unicode-range
WIDECHAR.  Making it -10 overflows, presumably because the additional byte is
padded out to a multiple of 32 bits.

I would have expected that somewhere, a size including the one-word heap header
would be computed, but these are never actually heap allocated, and it seems to
work.  But to hedge against things, I suggest using -19.  This would allow space
to count it, cross-compiling 32->64.

This will allow a text literal of 268435436 ISO characters, and a wide text literal
of 1/4 that, which is not likely to be a serious limitation, considering that it
all has to be on a single line of source code.

As for pickles, the horse is already out of the barn on that, as pickles could
already have been written with the current fingerprint, and we can't change the
type in any way without the fingerprint changing.  It is not hard to add
recognition of the old fingerprint to pickles.  Right off hand, I don't remember
the process for finding actual fingerprint values, but I've done it before and
can rediscover it.

>       (* - 8 * ORD(BITSIZE(INTEGER) = 64) *)
>       (* Do not adjust this for INTEGER size.  It makes T have different
>          fingerprints on 32- and 64-bit machines, which undermines pickling/
>
>
>
> otherwise we have:
>
>
> new source -> compiling TextLiteral.i3
> "../src/text/TextLiteral.i3", line 27: CM3 restriction: record or object type is too large
> 1 error encountered
>
>
>   Ok?
>    - Jay

-- 
Rodney Bates
rodney.m.bates at acm.org