[M3devel] 64bit INTEGERs, WIDECHAR: language specified or configuration/target dependent?

[Rodney M. Bates]

On 06/04/2015 12:43 PM, Elmar Stellnberger wrote:
> Thread forked from: Re: [M3devel] 32bit host 64bit target TextLiteral recurring problem
>
> Am 04.06.15 um 18:35 schrieb Rodney M. Bates:
>>
>>
>> On 06/04/2015 09:01 AM, Elmar Stellnberger wrote:
>>> Am 03.06.15 um 03:58 schrieb Jay K:
>>>> We cannot cross from 32bit host to 64bit target.
>>>>
>>>>
>>>> Ok to commit this?
>>>>
>>>>
>>>
>>> Compatibility between the 32bit and the 64bit version of the same program would be the minimum I expected from a pickle.
>>> To me this is just another argument for a language specified value range of INTEGER.
>>> It makes certain things easier.
>>>
>>> My suggestion was:
>>> TYPE OFFSET = BITS BITSIZE(ADDRESS) FOR INTEGER; (and INTEGER = BITS 32 FOR INTEGER for 32 and 64bit targets)
>>> while also allowing BITS 16 FOR INTEGER (if that works for WIDECHAR I believe there is no reason to forbid it for INTEGER)
>>> BITS 64 FOR INTEGER would only work on x64 systems.
>>>
>>
>> What you want is already pretty much here.  Just declare your desired type yourself:
>>
>> TYPE Int32T = [-16_7FFFFFFF-1 .. 16_7FFFFFFF];
>    This should be good news. However my suggestion was to make Int32.T the default
> as this will be somewhat faster for the average application which is data intensive (just
> think of image processing) which will never be rewritten to use Int32.T instead of INT.
> All of our current and old programs being prepared to run on a 32bit as well as a 64bit
> target should compile and run without any problem. There is one single exception
> where the compiler would throw an error: when using INTEGER for address arithmetics
> in unsafe modules (error is: can only LOOPHOLE between types of the same size, or
> some way similar as I have already tested this). There we would need the so called
> targetint/offset.

This would be a very biased thing to do.  The only people who would benefit would be those
who know for certain that they want *every* integer variable to have 32-bit range, on both
32-bit and 64-bit machines.  Without debating the wisdom or likelihood of this, a
quite easy string search/replace on "INTEGER" would accomplish the same thing.  But
everybody else would have to vet their integers one by one, to see which they wanted
always 32 and which they wanted native size.  It would break lots of code for the
unlucky (majority, I would guess), while providing only speed improvements for the
favored.  I have worked on lots of code in the M3 distribution itself that would break.

>    If I understand things right the following two things would need to get implemented
> to allow a default int size of 32 for existing applications (be it the default or just an
> additional compiler option as the size of WIDECHAR already is):
> * automatic range adaption for packed ints (i.e. those with BITS .. FOR)

No, your are still trying to misuse BITS to control the value range.  Subranges do this just
fine.  Use the mechanism the language already provides.  BITS is for manual control of
record/object/array layout only, not value range.

> * allowing non 32bit alignment of INTs in memory (as is already the case for WIDE[CHAR]s)

INTEGERs (and subranges thereof) are currently no different from WIDECHARS, in regard to alignment.
They get the same treatment as anything else.  For BITS types inside a record/object/array,
the language lets the compiler choose what it will accept, but I am reasonably sure all existing
M3 compiler variations, for all targets, will lay things out as you ask (including misaligned)
and will correctly access them, as long as no item crosses a native word boundary.
Outside records/objects/arrays, the compiler just aligns as needed.

>>
>> It will always have this range and occupy 32 bits, regardless of the native word size.  Pickles
>> will always save and reread it with exactly this range, even between machines of different
>> word size.
> So the target size of the pickle is already determined by the respective INT subrange, right?

Defining what pickles do/should do is fraught with pitfalls.  Here's what we now have:
For (almost) all types, pickles preserve the properties of the M3 type.  The bounds of a subrange
type are static constant numbers, and are evaluated at compile time on the compiling machine.
So [0..16_3FFFFFFF] is the same type on either size of machine, but [0..LAST(INTEGER) DIV 2] is not.
On a 32-bit machine, [0..LAST(INTEGER) DIV 2] evaluates to, and thus means [0..16_3FFFFFFF], but
on a 64-bit machine, it means [0..16_3FFFFFFFFFFFFFFF], a different type.  So, for example, you
could exchange pickled values between 32-bit and 64-bit machines if the field/element of the containing
record/object/array type were written as [0..16_3FFFFFFF] on both machines, or if it was written as
[0..LAST(INTEGER) DIV 2] on the 32-bit machine, and [0..16_3FFFFFFF] on the 64, but not if both had
coded the type as [0..LAST(INTEGER) DIV 2].

INTEGER, CARDINAL, LONGINT, and LONGCARD are exceptions.  Each of these is treated as the same
M3 type, regardless of native word size and upper bound.  So pickles will copy a field/element
of any these types with a size change, when written and read on different sized machines.  The size
change does the more-or-less obvious sign-extend one way and raises an exception if the value
won't fit, when going the other way.

The floating types are not handled by pickles across machines with different floating formats.
If the format is the same, (and one pickles knows exists), they will work, otherwise, they will
raise an exception.  It would be nice to implement this, but apparently, nobody has had a
sufficient need.  I am no numerical analyst, but I think defining good usable rules for this
would call for the services of someone who is.

> Something that should to my believe be well documented as one could easily like to extend
> an enum with 255 members to more or equal than 256 an then wonder why the new an the
> old file formats become incompatible. To circumvent such a problem we would likely need
> to allow BITS 8*anything FOR INTEGER and take the bitcount specified in BITS for in order to
> determine the target size of an object instead of semi-automatically readjusting it all the
> time.

This one really gets into a tar pit.  Pickles encode the type of a value as a "fingerprint",
which is a 64-bit has of the M3 type structure.  A copy of this goes into the pickle for
every type it contains.  Meanwhile, the compiler puts copies of the fingerprints of all
types in the program into static global readonly data.  Pickle read uses this to look or
a type defined in the reading program with the same signature, which means it must be
exactly the same type, for the read to succeed.

Allowing similar but unequal types would very difficult to define even what differences
would be tolerated.  This get very complicated very fast.  Remember, there is no guarantee
that the reading program is the same program as the writing program, or has much similarity
to it at all.  Pickles requires only that the reading program have a defined reference
type that is structurally identical to that of the object had in the writing program.

Allowing to write a field of type {Red, Green} in a pickle and read it into a program that has
{Red, Green, Blue} would get very messy.  Remember that pickles does not handle scalar values,
just heap objects.  So we would have to be defining some recursive rule about record types
whose corresponding field types were each sufficiently "similar", etc, etc.

And to implement, the pickle format would have to be completely reworked to contain full
type structural descriptions, not just signatures.

>>
>> The only thing that will differ from 32- to 64-bit machine is the size of intermediate results in
>> expressions.  If you are already _not_ being careful about avoiding overflows, there are probably
>> cases where the results of silent overflows would differ with machine word size.  Not sure what
>> you would really want here.  The unsigned arithmetic operations in Word are explicitly defined
>> to silently do binary twos-complement wrap-around when overflows happen.  The language does not
>> define this for the builtin signed operators.  In code where I care about overflows, I never assume
>> this.  It is probably machine-dependent.
> Yes, I believe it would be good like this. No real 32bit INT with 32bit arithmetics but just restricting
> the range for Int32.T when it gets written into memory.
>>
>> Or, you can import it: Int32.T. is already declared, in m3core, with this range.
>>
>> However, Int32.T specifies BITS 32 FOR ...  I strongly suggest this is almost never what you really want.
>> The only place the BITS specification makes any difference is when you declare a record or object
>> field of this type.  In all other situations, BITS 32 FOR .. has _no effect_.  The only purpose of
>> BITS is so you can force the layout of a record.  In that case, since the range needs exactly 32
>> bits anyway, even if you omit the BITS, a field of this type will still always occupy 32 bits.
> Nonetheless it is currently not possible to declare a variable of a 32bit type globally:
>
>   TYPE Pixel = BITS 32 FOR INTEGER;
>
> unluckily does not work as we can have no global variable of type Pixel.

No, you have misunderstood.  Any type BITS n OF T can be used anywhere T can be used,
including global variables, local variables, parameters, etc.  It's just that, except
for a field/element, the addition of BITS... will have no effect, in other words, it
won't change that number of bits or alignment the compiler actually allocates, from
what the compiler would do if the type were given as just T.  It will not make the
declaration illegal.

My recommendation not to use BITS in a named TYPE declaration was only because:
1) Outside of a record/object/array, it will make no difference anyway, and
2) Inside of a record/object/array, the effect it has can and often will be different
    every time, so you have to think about each such field/element and separately judge
    whether the BITS specification is right for that particular case.  And that depends
    on what comes before it in the record/object/array.

Also, there is full assignability between BITS n FOR T and T.  So you don't have to put
explicit conversions in.  The one case there is not such assignability is between
BITS n FOR T and BITS m FOR T, with n#m.  So copying between two record fields of the
same base type but different explicit BITS specifications of different sizes would
require explicit conversion.


> This is not only un-orthogonal but also a very practical problem; Suggesting that we call
> procedures with a parameter of type pixel we will never be able to change that type f.i. to
> a packed record with members red, green, blue alpha if there are places in our main
> program where we need to declare VAR pix1, pix2 : INTEGER just because Pixel is a packed
> type being refused as global variable.
>
>>
>> But, with the BITS specification, the compiler is not allowed to insert alignment padding, (if it
>> were, you could not fully specify the layout) and if the field comes at a place that is not 32-bit
>> aligned, you will either get a compile-time error or maybe a compiler will accept it and handle
>> unaligned field access (it is not required to--ours does not), neither of which is probably what
>> you want.  Without BITS, the compiler will pad as needed.  With BITS, you will have to insert explicit
>> padding for alignment yourself as needed, which can be different for every field of this type of every
>> record/object.  And later adding/removing/changing any prior field can upset it and require you to
>> rework the padding.  Moreover, I am sure there are simple cases where the required padding differs
>> between 32-bit and 64-bit machines.
> On the long term I would personally even advocate an ALIGN directive:
>   f.i. TYPE SSEdata = ALIGN 128 FOR RECORD ... END;
>
> It would be very handy to have this for tapping the full power of the vectorization units
> of current processors. Manually re-aligning heap allocated objects in an unsafe module
> is not a very satisfying alternative (no local/global/stack allocation, memory waste and
> thus a certain performance loss if there are many wholes). However this would possibly
> also make changes to our garbage collector necessary as it currently only guarantees
> word alignment. Finally it would also be possible to write ALIGN BITSIZE(TARGETINT) ....
>
>
>>
>> Having been bitten by this more than once, I now only put BITS..FOR as the anonymous type of a specific
>> field, never in a named type to be used in multiple places.
> A less bitchy and more orthogonal Modula-3 compiler supporting integer globals and
> locals of non word size and alignment would be a real pleasure, do you consent Rodney?
>>
>> (Actually, BITS does the same thing when used as the type of an array element, but cases where this
>> matters are rare.  If the bit count evenly divides the word size, there won't be any alignment padding
>> needed, and BITS won't matter.  Otherwise, the compiler is likely to refuse, unless the entire array
>> fits in a single machine word.  I do have a couple of such cases.)
>>
>>> The value range - bitsize correlation problem could be solved easily by automatically extending and reducing the value
>>> range of an integer type to what its binary representation allows as long as no explicit range has ever been specified for
>>> any of its supertypes (i.e. BITS 16 FOR INTEGER = BITS 16 FOR [-32768..+32767]).
>>>
>>> Finally it needs to be said that the argument that target size arithmetics is always the fastest which was often used in
>>> here can be very wrong. It does not hold for the 64bit systems of today because the CPU is no more the bottleneck. In
>>> a fact now the memory hierarchy has become the new bottleneck (which means that using more memory for the same
>>> tends to be much slower).
>>>
>>> Regards,
>>> Elmar
>>>
>>
>
>

-- 
Rodney Bates
rodney.m.bates at acm.org