[M3devel] suggest fixed size integers in m3cg_binrd/binwr instead of compact encoding, for trivial decoding/looping?

[Jay K]

BinRd/BinWr "redesign".


How about BinRd/BinWr is "redesigned" with the following in mind:


instead of most compact encoding, instead use least-decoding-needed
encoding, which is also least compact


in particular, store all integers as native-endian 8 byte integers


include at the start of the stream, like:


 UINT8 file_endian;
 UINT64 integer_map_offset;
 UINT64 integer_map_size;


reader would mmap the file (mmap has its pluses and minuses).
The data would be presumed to fit in address space -- even if
it tends to be accessed sequentially, random access would be possible.


if file_endian != host_endian, it would jump ahead to the
positions indicated by integer_map_offset and swap them.


writer would note all these offsets while writing



If I made this change, then I'd be much more likely to just
use this "pre-"existing" format as an in-memory format that
I loop over multiple times.


Given that these are temporary files that don't usually make it to disk,
optimizing for size isn't clearly valuable.


Given this change, walking the m3cg data multiple times would 
end up just being bumping a pointer through an array of variably-sized
records, the sizes of which would be trivially looked up based
on the first byte (with the exception of case_jump).
This would likely obvious my ongoing little M3CG_MultiPass tedious work.
Something I believe is desired/needed here for M3C.m3. This isn't meant
to be a pointless excercise (though it is *somewhat* speculative).


m3front/m3cg/m3cg_ops would be unchanged.
The change would be in M3CG_BinRd.m3/M3CG_BinWr.m3/parse.c.


Thoughts?


I've omitted the obvious detail -- floats/doubles.
There'd actually be an array of MultiByteValueMaps.
They'd include a size. There'd be "4" for floats, and "8"
for INTEGERs and doubles. And hypothetically 10 or 16 for "extended",
but those don't really exist.



 - Jay

 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://m3lists.elegosoft.com/pipermail/m3devel/attachments/20120822/6edcfefb/attachment-0001.html>