[M3devel] pixmap problem

Sat May 10 12:48:08 CEST 2008

Randy, 3.6 is still available for download on the web, and it still works.

I know exactly what my fix does and why it wasn't working before in the immediate past -- some set operations are dependent on data in m3core, in hand.c, and importing data from a .dll requires dereferencing a pointer to get the data. Whereas getting data from a static .lib does not. The pointer for data "foo" is usually called "__imp__foo", though that is just a convention implemented by the linker. So, let's say foo is a function. The generated code to call foo can be, and in Modula-3 always is, merely "call foo", or maybe "call _foo". x86 Windows tends to put a leading underscore on everything. That's a different matter. Now, again, foo is a function let's say, and the code says "call foo". Now, if foo ends up being imported from a .dll, what the linker does is generate a one instruction function like so:

foo:
  jmp dword ptr [__imp__foo]

And then __imp__foo is also special. __imp__foo is created such that at runtime, at load time, it is overwritten with the address of the actual location of the actual foo function in another .dll. All the __imp__ symbols are adjacent, so these writes aren't scattered around and written pages are kept to a minimum, at least in this regard.

Now, if you KNOW you are going to import foo, you can give the compiler a small optimization hint, and the compiler can "call dword ptr [__imp__foo]" directly, skipping the one instruction -- said instruction also having poor locality. As well, the Visual C++ compiler is smart enough to prefetch the instruction, and if you make repeated calls to an imported function, even keep it around in a register. There is no way to get this in Modula-3 today, and really it doesn't seem like a big deal to me. See, you know, if you don't know if foo will be dynamically linked or statically linked, "call foo" is reasonable efficient generic code. If you think it will be imported, you can generate "call dword ptr [__imp__foo]" instead, but then if you are wrong, either a) you get a link error, or b) you can go ahead and generate the pointer that points to the function, and you will pay an extra pointer dereference in the function calls, but at least not an extra instruction. As I wildly guess, calls to statically linked functions are more common than calls to dynamically linked functions, so erring toward making the static call more efficient seems better. On the other hand, it appears to me that Windows is alone here. My brief look into the Linux and Darwin calling conventions indicates they use big inefficient methods, worse than the "worst case" Windows uses, when you take either of the deoptimizations I describe. Granted, everyone else tends to get actually position independent code, and Windows does not. Windows get relocatable code, but not position independent, and the cost of the relocation can be high (every page written, loss of physical memory sharing, dirty pages backed by pagefile instead of the original .exe/.dll).
AMD64 Windows has mostly position independent code via the new RIP-relative addressing mode, but not fully. What often bites you is pointers in your data, like vtables. I do wish Windows had fully position independent code and I don't know if that would result in exactly the inefficient sequences I see on Linux and Darwin or not.

Ok now. That only mentioned functions. There was no mention of data.
You must pay close attention to particular details. In particular, the code always says "call foo" and if foo is imported, the linker generates a single instruction function, named foo, just like expected, that indirects through a pointer. You must indirect through a pointer for dynamic linking. And since there is a function call involved, a function can be generated to fit.
There's some saying here, like, you know that "any problem can be solved by an extra level of indirection", well, there's some corralary like "give me a function call and I can add indirection afterward".

But if you think about data, compiler generates something like "mov eax, foo" -- there's no way for the linker to intercede there -- there is no function call. It actually should fail to link. That is a different related matter. If you make the kind of error in C code that we have in Modula-3, you would get a link error. I'll show you in a sec. What happens in Modula-3, is, well, something, like mklib, doesn't know foo is data instead of a function, and goes ahead and generates foo that jmps through __imp__foo. That function foo is what these data references end up using, with fairly "random" results.

What my fix does is statically link in foo (_highbits, _lowbits). So the regular "mov eax, foo" works. No pointer derefence is needed for static linking, no __imp__ foo. I should go and fix the compiler to reference __imp__foo, and if we end up statically linked, like for standalone, there can be an extra pointer __imp_foo = &foo, and standalone will pay an extra, unnecessary, pointer deref.

Here is how you can see this problem in C.

  dll.c  
  __declspec(dllexport) int foo;  

  exe.c  
  extern int foo;   
  int main()  
 {  
   printf("%d\n", foo);    
  return 0;     }  

 cl -LD dll.c -link -noentry -nodefaultlib 
 rem really that's all it takes to make a .dll 
 cl exe.c dll.lib 
 rem really, that's it takes to build an .exe that uses the .dll 

 >> exe.obj : error LNK2019: unresolved external symbol foo referenced in function main 

But if you change it to:

  __declspec(dllimport) extern int foo;  

then it works. I don't fully understand this -- something somewhere knows that foo is data and doesn't generate the function for it. If you use a .def file, you mark the symbol as "data".

The GNU tools, ld --help, suggests some workaround here. That they let you generate the "wrong" code and the linker somehow makes it work. I don't know what it is..the only thing I can think of might not work and would look really ugly. I think link -dump crashes on ld output so I stopped looking..

I will look into 4.1 to determine what changed and why it worked.

The result in 5.x is actually not guaranteeably wrong, just very very difficult to predict the outcome of.
Instead of referencing the correct data, some fairly arbitrary data is used. But of course not every bit matters.
Could be the 4.1 code doesn't use the set operations here even, avoiding this problem.
We'll see. "There is no need to speculate; just debug it and know." :)
That's my own saying. I see so much unnecessary speculation and metaphorical throwing of hands in the air, and not enough debugging. :)  (Granted, something is screwy on my XP AMD64 machine and I'm more likely to reinstall than debug it...and maybe if it recurs then I'll debug some...can't do much without source and symbols though...that's where my debugging skills rapidly fall off...(which is a complaint I have about Modula-3, there's a certain lack of symbols for global data I believe, probably should be fixed...)) 

 - Jay

Date: Sat, 10 May 2008 06:05:02 -0400From: rcoleburn at scires.comTo: m3devel at elegosoft.comSubject: Re: [M3devel] pixmap problem

Jay:

I get correct behavior now for TestPixmap on d5.7 when built either way, with or without standalone option.
THANKS!

On 4.1, TestPixmap also works both ways.  The problem was occurring before your fix on d5.7 (and on earlier releases after 4.1).  Not sure what your fix does or what changed from 4.1 to d5.7 that made your fix necessary.

I am using your NT386 stuff for cm3.cfg now.

I don't have 3.6.

I will need to run tests now with d5.7 against my programs to see if they are working correctly before I can abandon 4.1.  Also, we will need to drive a stake in the ground at some point soon to create an official 5.7 release.  My company will want to be able to point to a given development environment for future support of the product I'm working on.  They won't accept a "d5.7" release that is in a state of flux.

Also, the other issue I'll have to work is getting all the GUI changes incorporated into this edition.  We will need the custom look&feel that CMass put together for us.  I'll check into it.  If I am successful, I can create a branch on CVS with this edition in case anyone else wants it.

Regards,
Randy>>> Jay <jayk123 at hotmail.com> 5/10/2008 5:38 AM >>>Randy, clarification, the bug occured when NOT building standalone.You must know that, since you reproed it recently.Just try both ways, both should work. (You don't have to take my cm3.cfg, but you do need NT386.common, and then either copy NT386 to cm3.cfg, or take NT386 and have cm3.cfg be just one line, include("NT386")) Any reason left for you to stick with 4.1? :)Something with serialio? At least one question does remain -- the TestPixmap behavior and why on 4.1.I'll look into that shortly.3.6 isn't relevant, since it didn't have .dlls. Thanks, - Jay

Date: Sat, 10 May 2008 05:23:35 -0400From: rcoleburn at scires.comTo: m3devel at elegosoft.comSubject: Re: [M3devel] pixmap problem
Jay:

I've performed the following and it seems to have fixed the problem with the TestPixmap program when built standalone:

1.  performed CVS update
2.  updated cm3.cfg and NT386* in \cm3\bin based on sources in cminstall\config
3.  scripts\win\upgrade
4.  scripts\win\do-cm3-std buildship

I checked the .lst file and I am now seeing "hand.obj"

Regards,
Randy>>> "Randy Coleburn" <rcoleburn at scires.com> 5/10/2008 4:51 AM >>>
Jay:

I do not have python installed on my computer, so I can't run your scripts.

I used the scripts in scripts\win to upgrade a few days ago.

I am not seeing hand.obj in the .lst file.

I will do a CVS update and try the upgrade again followed by do-cm3-standard.

Regards,
Randy>>> Jay <jayk123 at hotmail.com> 5/9/2008 5:10 AM >>>
 > Also, more bad news, the pixmap problem is not solved. Randy, this DOES actually fix it, but you have to rebuild more. Try:   cd scripts\python    .\do-cm3-std.py realclean    .\upgrade.py   (probably not needed)    do-cm3-std.py buildship   Make sure "hand.obj" occurs in the various NT386\foo.lst files. I think the fix should be restated to have the code reference *__imp__highbits instead of highbits, and go back to dynamic linking the data.But ok for now. The std distribution I put yesterday should work, though it is missing the RTArgs fix.I'll see about making another.   - Jay

Date: Mon, 5 May 2008 23:28:10 -0400From: rcoleburn at scires.comTo: m3devel at elegosoft.com; jayk123 at hotmail.comSubject: RE: [M3devel] pixmap problem
Jay:

Ok, I've copied your NT386 config stuff into the bin folder and I've copied NT386 to bin\cm3.cfg.

The good news is that my -gui mode programs now link and run.  The bad news is that now I am getting a command window opening up in addition to the GUI window when the program runs.  This behavior does not occur on v4.1.  The command window seems to be the stdout/stderr of the GUI program.

Also, more bad news, the pixmap problem is not solved.

And more bad news, your config stuff breaks the CM3-IDE program which needs to find certain definitions in cm3.cfg like PKG_USE, etc.

Regards,
Randy>>> Jay <jayk123 at hotmail.com> 5/5/2008 10:47 PM >>>Randy, Yes.The NT386 config files are all checked in.You can use them verbatim. You don't need cminstall, I never use it, and you shouldn't need to edit the config files at all, any edits I make, I check in, and should be usable on all machines.As long as you set %PATH%/%INCLUDE%/%LIB%.If I'm wrong, I'd like to know why and try to fix. Consider the config files just part of cm3.exe.Not generally to be changed locally. Unless, well, unless you intend to, and you should strive to make one form that is usable by all.  - Jay

Date: Mon, 5 May 2008 22:42:30 -0400From: rcoleburn at scires.comTo: jayk123 at hotmail.comSubject: RE: [M3devel] pixmap problem
Jay,
When you say "took the updated config file", what do you mean?  Are you saying I need a different cm3.cfg file?
Regards,
Randy>>> Jay <jayk123 at hotmail.com> 5/5/2008 9:54 PM >>>And you took the updated config file?The way you can tell is that, roughly speaking, hand.obj should be in NT386\_m3responsefile0.txt.Or _m3responsefile1.txt, or, well, somewhere.Anyway, I think I saw the attachment, I can check later..and with 4.1 as I said.. I can see about making the fix not depend on config file changes...but it's not terrible asis, and I'd really like to merge more of the quake code into cm3.exe..  - Jay

Date: Mon, 5 May 2008 21:18:12 -0400From: rcoleburn at scires.comTo: m3devel at elegosoft.comSubject: Re: [M3devel] pixmap problem
Jay:

I've updated my sandbox via CVS and I've rebuilt everything.

I rebuilt the TestPixmap program, but it still does not work properly when the buildstandalone() directive is used.

Regards,
Randy>>> Jay <jayk123 at hotmail.com> 5/5/2008 7:04 PM >>>Thanks. I'll look into this tonight probably, as well as with 4.1 assuming I can find my binaries and source for it (probably).COULD BE the pixmap problem is something else and 4.1 also has the problem I fixed. We'll see. No need to speculate (unless I can't find my 4.1). - Jay

Date: Mon, 5 May 2008 10:59:27 -0400From: rcoleburn at scires.comTo: jayk123 at hotmail.comCC: m3devel at elegosoft.comSubject: RE: [M3devel] pixmap problem
Jay:

The TestPixmap.zip is attached.

I will attempt to update all my sources and try again.

Not sure what you mean when you say that I must use your config files or edit what I am using.  Please elaborate.

The pixmap problem does not occur on 4.1.

Regards,
Randy>>> Jay <jayk123 at hotmail.com> 5/4/2008 11:46 AM >>>Randy, Please try with /current/ source, as of this morning Sunday May 4 -- the changes are ALL in NT386.common, so you MUST use MY config files, which are checked in verbatim, or you should be able edit whatever config files you are using. (It's not really a config file issue, don't blame the general confusion people have with them.) I'll try to try later, but I don't have the source right now, gotta look on other machines, and I think it's on a machine on loan.  Or send it to me again.I'd also like to know if this worked with 4.1, or with any release that offered dynamic linking to m3core.dll, on Windows.  I recall a claim that it did. If today's change fixes it, I should be able to investigate 4.1.3.6 did not have such linking.If today's change does not help, well, I still plan on debugging it at some point.. If you aren't current and getting so is difficult, just send the attachment again, it's easy for me to test. Thanks, - Jay

From: jayk123 at hotmail.comTo: hosking at cs.purdue.edu; rcoleburn at scires.comDate: Sat, 19 Jan 2008 08:52:08 +0000CC: m3devel at elegosoft.comSubject: Re: [M3devel] pixmap problem

I can repro the two different behaviors on XP.I was going to try on PPC_DARWIN but I guess Tony's info suffices. I tried rolling hand.c back to before I changed (months/year ago), since it is involved in bit twiddling, no change, phew. Debugging here is a big humungous pain.NT386 has good stacks and line numbers, but no locals or type info, I'd really sometime like to get Codeview info into those .objs...I think it is documented well enough, but still not necessarily easy, or maybe have a C generating backend partly for this reason (and to bring up more targets, using native tools), and function calls in disassembly I think don't resolve; NT386GNU can't yet build this, though it will soon, but I'm not familiar with gdb and there appears to be no type info. I'm building m3gdb right now, I don't know if it will work, I hope it has type info, and I'm not familiar with the relevant code. In gdb I struggle just to view bytes in memory, add numbers to addresses and see those, etc. I think for MY problem, I'm going to link in C code to dump the structs. There is tracing code and I enabled it, but even it crashes in Text_Length. At least MY problem crashes, hopefully near the problem, whereas your problem goes all the way through to completion with no stopping to hint where the problem is. Darn it. :) on the mock rudeness and :( on the lack of a crash. You write the bundle out and it matches, so that doesn't rule out bundles as being the problem? (darn)And bundles look pretty simple, a bundle is just a generated source file with a constant in it.Maybe newline problems? That wouldn't make sense. I just downloaded some program that might let me separately view the ppm, maybe it'll reveal something. The bad behavior has like regularly spaced solid color veritical bars a few pixels side across the whole picture, and regularly spaced vertical stripes from the correct picture alternating with that. Perhaps someone with ppm/bitmap/graphics experience has seen this behavior and it results from some common pixmap mismanagement we could look for? I have to say this is very surprising. I always wondered, and now I pretty much assume -- Modula-3 never imports/exports data in its dynamic linking, right? It alway imports/exports functions, right?On Windows, functions have an optional optimization, but data must be handled differently at compile time if it is going to be dynamically imported.For functions, if you don't apply the optimization at compile time, the linker will just generate a single instruction thunk instead. That is, given a symbol Foo, there is a symbol __imp__Foo that is a pointer to the actual foo in another .dll/.so.The loader only patches pointers, one per import if you build your .dll correctly, it doesn't patch instructions or code strewn throughout a .dll, just one contiguous array of pointers. MAYBE they don't have to be contiguous for each .dl you import from. That is, if call msvcrt!fopen, msvcrt!fclose, kernel32!Sleep, my .dll/.exe typically has an array: pointer to msvcrt!fopenpointer to msvcrt!fclosenullpointer to kernel32!Sleep But perhaps kernel32.dll's pointers and msvcrt.dll's pointers need not be adjacent. So, going back, my point/question is that, if the compiler knows the function is imported, it can call through __imp__Foo instead of calling Foo.But if it calls Foo and you import Foo, the linker will create the function Foo that just jmps through __imp__Foo.That doesn't work for data though. There's no ability to intervenve like that.So for imported data, the compiler must generated a read and dereference of the pointer __imp__Foo for accesses to Foo, if Foo is to be imported.Data imports may be faster when appropriate, but for this reason, I'd say they aren't worth it.And for this reason, I HOPE Modula-3 never imports/exports data. Oh, you could, for data, always reference the pointer __imp_Foo, and if non materializes, create one.However that would suck perf in order to make an uncommon case work.I hope Modula-3 doesn't do that either. :)Though I guess since this global data in question...maybe it is rare??? Later, - Jay

> From: hosking at cs.purdue.edu> Date: Sat, 19 Jan 2008 03:09:47 -0500> To: rcoleburn at scires.com> CC: m3devel at elegosoft.com> Subject: Re: [M3devel] pixmap problem> > Looks fine to me on I386_DARWIN.> > On Jan 19, 2008, at 12:31 AM, Randy Coleburn wrote:> > > <TestPixmap.zip>> 

Need to know the score, the latest news, or you need your Hotmail®-get your "fix" Check it out. 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://m3lists.elegosoft.com/pipermail/m3devel/attachments/20080510/22d26b1b/attachment-0002.html>