From wagner at elegosoft.com Sat Mar 1 15:43:32 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Sat, 01 Mar 2008 15:43:32 +0100 Subject: [M3devel] state of m3gdb on LINUXLIBC6? In-Reply-To: <47C83637.6010602@wichita.edu> References: <1204226001.13580.64.camel@hias.m3w.org> <47C749B8.1060906@wichita.edu> <20080229011936.fs7gutizgggg8s44@mail.elegosoft.com> <47C83637.6010602@wichita.edu> Message-ID: <20080301154332.jk9jnfgzcw48g04s@mail.elegosoft.com> Quoting "Rodney M. Bates" : > Olaf Wagner wrote: >> Quoting "Rodney M. Bates" : >> >>> As for other platforms, I don't have any around, and I'm not aware >>> of anybody >>> trying it. >> >> It compiles without problems on FreeBSD 6.3 and works as long as >> programs are linked with libthr.so (not with libpthread.so). > > Any more information on what the difference in these libraries is? AFAIK, there are three different pthreads implementations on FreeBSD: libc_r, libpthread, libthr. libc_r is completely user-level, while pthread and thr use system level threads (with support and callbacks from the system scheduler). In FreeBSD 6.3, pthread seems to be the default, while in 7.0 (which has been released just a few days ago) they have switched to thr as default. thr is a 1:1 mapping from user to system level threads AFAIK. I switched the CM3 thread default for FreeBSD from POSIX to Antony's PTHREADS some months ago; I think we should go with the libthr as default now (since this is what the FreeBSD folks do themselves). (BTW. they promise dramatic performance increases in FreeBSD 7.0 wrt. concurrency.) In FreeBSD, dynamic library mapping for threads can be switched easily via /etc/libmap.conf. >> I haven't used it much yet, and some things seemed to be strange; >> thread stack switching or viewing didn't work at all. > > m3gdb needs work on threads. I have never done anything on it. > Unless Tony did some when he was working on it several years ago, > I think it is pretty much as it came from SRC, and that was fairly > primitive. Plus, the new pthreads implementation is not supported. > Once I glanced at the code and it looked immediately like what was > there would not work at all. This has been on my list a while. If, as the others say, stock gdb has good support for pthreads, then it is probably a FreeBSD specific problem. The BSD folks often do customizations of the GNU tools on their own, that the FSF does not accept / maintain. One would probably need to base a m3gdb for FreeBSD on the FreeBSD gdb sources :-/ >> If you want to try anything on FreeBSD, let me know, I can provide >> a remote login. > > Yes, that would be helpful. I'm currently not at home, but I'll setup a login on my FreeBSD system for you at the start of next week. I assume your ssh key is already on birch.elegosoft.com (if not, just send it to me). Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From dragisha at m3w.org Sun Mar 2 00:41:19 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 02 Mar 2008 00:41:19 +0100 Subject: [M3devel] random-acces I/O In-Reply-To: <20080229224126.GA21397@topoi.pooq.com> References: <20080229224126.GA21397@topoi.pooq.com> Message-ID: <1204414879.10030.8.camel@hias.m3w.org> On Fri, 2008-02-29 at 17:41 -0500, hendrik at topoi.pooq.com wrote: > I need to do direct-access I/O from a Modula 3 program with a file > format that is externally specified. It is necessary to both read and > write in random fashion during a run of the program, prefereably without > having to close or open the file each time. > > I expect the answers to the following questions are "Yes", but I'd > appreciate confirmation: > > ? If I have a File.T (obtained from FS.OpenFile) can I have both a > Rd.T and a Wr.T active on it simultaeously? Yes > > ? Will things written using the Wr.T be immediately available > for reading with the Rd.T, with no buffereing problems? Most probably you'll have to use unbuffered IO here. > > ? Are there methods for locking the file -- or portions of it -- > against simutaneous access by other programs, including ones written > in other languages? There are. But you have to make some rules, and follow them from all users of your file. > > -- hendrik -- Dragi?a Duri? From mika at async.caltech.edu Sat Mar 1 23:49:23 2008 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 01 Mar 2008 14:49:23 -0800 Subject: [M3devel] random-acces I/O In-Reply-To: Your message of "Fri, 29 Feb 2008 17:41:26 EST." <20080229224126.GA21397@topoi.pooq.com> Message-ID: <200803012249.m21MnOnF018386@camembert.async.caltech.edu> Hi Hendrik, The answer is, "it depends." As a matter of fact this was my biggest area of trouble when porting a program from PM3 on Unix to PM3/Klagenfurt for NT386GNU. On NT386GNU, the default is for Modula-3 to avail itself of Windows's mandatory file locking, whereas POSIX has no such thing. This means that a Modula-3 program on NT386GNU can't even open a file for reading if you are "lessing" it for instance. I am sure it can be worked around, but I never bothered to figure out how, sorry (because the behavior I needed was in fact strict locking of this kind, which I had to go to special lengths to get on Unix!) On Unix, things are a bit more as you describe them, but not exactly. You can definitely have an Rd.T and a Wr.T on a single file, but I do not believe that the Rd.T and Wr.T will be aware of each other's existence (I could be wrong). However, there is a package called rdwr(?) that I think may have the sort of functionality you're looking for. Maybe. File locking is strictly an OS issue as far as I know and that is not something that Rd and Wr concern themselves with. If you need this kind of low-level access I have a feeling Rd and Wr are too high-level for you. I would suspect you want to build something with rdwr (or some other means of getting an Rd out of a Wr or vice versa) and then extend it with routines for maintaining a faithful disk-based copy of what you want... using lower-level access methods. Oh yes Rd and Wr are documented in detail in chapter 6 of the Green Book. Mika hendrik at topoi.pooq.com writes: >I need to do direct-access I/O from a Modula 3 program with a file >format that is externally specified. It is necessary to both read and >write in random fashion during a run of the program, prefereably without >having to close or open the file each time. > >I expect the answers to the following questions are "Yes", but I'd >appreciate confirmation: > > ? If I have a File.T (obtained from FS.OpenFile) can I have both a > Rd.T and a Wr.T active on it simultaeously? > > ? Will things written using the Wr.T be immediately available > for reading with the Rd.T, with no buffereing problems? > > ? Are there methods for locking the file -- or portions of it -- > against simutaneous access by other programs, including ones written > in other languages? > >-- hendrik From dragisha at m3w.org Sun Mar 2 01:12:44 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 02 Mar 2008 01:12:44 +0100 Subject: [M3devel] random-acces I/O In-Reply-To: <200803012249.m21MnOnF018386@camembert.async.caltech.edu> References: <200803012249.m21MnOnF018386@camembert.async.caltech.edu> Message-ID: <1204416764.10030.12.camel@hias.m3w.org> rdwr is package by Blair MacIntyre and it's not related to anything File.* except indirectly, ie if you make File Wr part of TeeWr. On Sat, 2008-03-01 at 14:49 -0800, Mika Nystrom wrote: > Hi Hendrik, > > The answer is, "it depends." As a matter of fact this was my biggest > area of trouble when porting a program from PM3 on Unix to > PM3/Klagenfurt for NT386GNU. On NT386GNU, the default is for > Modula-3 to avail itself of Windows's mandatory file locking, whereas > POSIX has no such thing. This means that a Modula-3 program on > NT386GNU can't even open a file for reading if you are "lessing" > it for instance. I am sure it can be worked around, but I never > bothered to figure out how, sorry (because the behavior I needed > was in fact strict locking of this kind, which I had to go to special > lengths to get on Unix!) > > On Unix, things are a bit more as you describe them, but not exactly. > You can definitely have an Rd.T and a Wr.T on a single file, but I > do not believe that the Rd.T and Wr.T will be aware of each other's > existence (I could be wrong). However, there is a package called > rdwr(?) that I think may have the sort of functionality you're > looking for. Maybe. > > File locking is strictly an OS issue as far as I know and that is not > something that Rd and Wr concern themselves with. > > If you need this kind of low-level access I have a feeling Rd and Wr > are too high-level for you. I would suspect you want to build something > with rdwr (or some other means of getting an Rd out of a Wr or vice > versa) and then extend it with routines for maintaining a faithful > disk-based copy of what you want... using lower-level access methods. > > Oh yes Rd and Wr are documented in detail in chapter 6 of the Green Book. > > Mika > > hendrik at topoi.pooq.com writes: > >I need to do direct-access I/O from a Modula 3 program with a file > >format that is externally specified. It is necessary to both read and > >write in random fashion during a run of the program, prefereably without > >having to close or open the file each time. > > > >I expect the answers to the following questions are "Yes", but I'd > >appreciate confirmation: > > > > ? If I have a File.T (obtained from FS.OpenFile) can I have both a > > Rd.T and a Wr.T active on it simultaeously? > > > > ? Will things written using the Wr.T be immediately available > > for reading with the Rd.T, with no buffereing problems? > > > > ? Are there methods for locking the file -- or portions of it -- > > against simutaneous access by other programs, including ones written > > in other languages? > > > >-- hendrik -- Dragi?a Duri? From mika at async.caltech.edu Sun Mar 2 03:24:02 2008 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 01 Mar 2008 18:24:02 -0800 Subject: [M3devel] random-acces I/O In-Reply-To: Your message of "Sun, 02 Mar 2008 01:12:44 +0100." <1204416764.10030.12.camel@hias.m3w.org> Message-ID: <200803020224.m222O3bP023511@camembert.async.caltech.edu> Yes, I am aware of that. My point was simply that if you want a reader and writer accessing the same data it's probably easiest to connect together the reader and writer directly, somehow, and write some sort of "watcher" that flushes modifications to disk "on the side" of the interaction between the reader and writer. Btw, I do have some code that does this sort of thing (not entirely random access), and to run both on Unix and NT386GNU, yes, I do close the file every time I "turn it around" from reading to writing or vice versa. Mika =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: >rdwr is package by Blair MacIntyre and it's not related to anything >File.* except indirectly, ie if you make File Wr part of TeeWr. > >On Sat, 2008-03-01 at 14:49 -0800, Mika Nystrom wrote: >> Hi Hendrik, >> >> The answer is, "it depends." As a matter of fact this was my biggest >> area of trouble when porting a program from PM3 on Unix to >> PM3/Klagenfurt for NT386GNU. On NT386GNU, the default is for >> Modula-3 to avail itself of Windows's mandatory file locking, whereas >> POSIX has no such thing. This means that a Modula-3 program on >> NT386GNU can't even open a file for reading if you are "lessing" >> it for instance. I am sure it can be worked around, but I never >> bothered to figure out how, sorry (because the behavior I needed >> was in fact strict locking of this kind, which I had to go to special >> lengths to get on Unix!) >> >> On Unix, things are a bit more as you describe them, but not exactly. >> You can definitely have an Rd.T and a Wr.T on a single file, but I >> do not believe that the Rd.T and Wr.T will be aware of each other's >> existence (I could be wrong). However, there is a package called >> rdwr(?) that I think may have the sort of functionality you're >> looking for. Maybe. >> >> File locking is strictly an OS issue as far as I know and that is not >> something that Rd and Wr concern themselves with. >> >> If you need this kind of low-level access I have a feeling Rd and Wr >> are too high-level for you. I would suspect you want to build something > with rdwr (or some other means of getting an Rd out of a Wr or vice >> versa) and then extend it with routines for maintaining a faithful >> disk-based copy of what you want... using lower-level access methods. >> >> Oh yes Rd and Wr are documented in detail in chapter 6 of the Green Book. >> >> Mika >> >> hendrik at topoi.pooq.com writes: >> >I need to do direct-access I/O from a Modula 3 program with a file >> >format that is externally specified. It is necessary to both read and >> >write in random fashion during a run of the program, prefereably without >> >having to close or open the file each time. >> > >> >I expect the answers to the following questions are "Yes", but I'd >> >appreciate confirmation: >> > >> > ? If I have a File.T (obtained from FS.OpenFile) can I have both a >> > Rd.T and a Wr.T active on it simultaeously? >> > >> > ? Will things written using the Wr.T be immediately available >> > for reading with the Rd.T, with no buffereing problems? >> > >> > ? Are there methods for locking the file -- or portions of it -- >> > against simutaneous access by other programs, including ones written >> > in other languages? >> > >> >-- hendrik >-- >Dragi??a Duri?? From wagner at elegosoft.com Sun Mar 2 11:53:57 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Sun, 02 Mar 2008 11:53:57 +0100 Subject: [M3devel] random-acces I/O In-Reply-To: <1204414879.10030.8.camel@hias.m3w.org> References: <20080229224126.GA21397@topoi.pooq.com> <1204414879.10030.8.camel@hias.m3w.org> Message-ID: <20080302115357.2mv96dyas4so04o0@mail.elegosoft.com> Quoting Dragi?a Duri? : > On Fri, 2008-02-29 at 17:41 -0500, hendrik at topoi.pooq.com wrote: >> ? Are there methods for locking the file -- or portions of it -- >> against simutaneous access by other programs, including ones written >> in other languages? > > There are. But you have to make some rules, and follow them from all > users of your file. File.T provides lock() and unlock() methods. You must be aware though that file locking is inherently depending on operating system and file system, and as far as I know there is not the slightest chance of a general applicable specification of locks. In CM3, the POSIX target use the POSIX advisory file locking, as far as I know, which means that only those interested in heeding the lock will see it. Nobody will prevent others from reading or writing the file. On Windows, the default behaviour for NTFS is completely different; on FAT file systems there are probably no locks at all. If your files are located on an NFS share, locking depends on the implementation of the rpc.lockd on the server and on the quality of the NFS client; for SMBFS/SAMBA, it depends both on the onderlying file system and on the configuration of SAMBA. If anybody knows of a commonly available locking abstraction, I'd be interested in it, too. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From hendrik at topoi.pooq.com Sun Mar 2 14:12:47 2008 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Sun, 2 Mar 2008 08:12:47 -0500 Subject: [M3devel] random-acces I/O In-Reply-To: <20080302115357.2mv96dyas4so04o0@mail.elegosoft.com> References: <20080229224126.GA21397@topoi.pooq.com> <1204414879.10030.8.camel@hias.m3w.org> <20080302115357.2mv96dyas4so04o0@mail.elegosoft.com> Message-ID: <20080302131247.GA26816@topoi.pooq.com> On Sun, Mar 02, 2008 at 11:53:57AM +0100, Olaf Wagner wrote: > Quoting Dragi?a Duri? : > > >On Fri, 2008-02-29 at 17:41 -0500, hendrik at topoi.pooq.com wrote: > >> ? Are there methods for locking the file -- or portions of it -- > >> against simutaneous access by other programs, including ones written > >> in other languages? > > > > There are. But you have to make some rules, and follow them from all > >users of your file. > > File.T provides lock() and unlock() methods. You must be aware though > that file locking is inherently depending on operating system and > file system, and as far as I know there is not the slightest chance > of a general applicable specification of locks. In CM3, the POSIX > target use the POSIX advisory file locking, as far as I know, which > means that only those interested in heeding the lock will see it. > Nobody will prevent others from reading or writing the file. Advisory is fine. > > On Windows, the default behaviour for NTFS is completely different; > on FAT file systems there are probably no locks at all. > > If your files are located on an NFS share, locking depends on > the implementation of the rpc.lockd on the server and on the quality > of the NFS client; for SMBFS/SAMBA, it depends both on the onderlying > file system and on the configuration of SAMBA. In Linux, locking via fcntl is NFS-safe. locking via flock is apparently not. lockf is implemented via fcntl, so it is presumably NFS-safe, too, although the spoecifications don't actually require it to be implemented via fcntl. When I had NFS misconfigured (and I could find no information indicating how reconfigure it) one program (monotone) could not lock any NFS files, knwew it, and refused to operate as a result. That's the safe way to deal with NFS misconfiguration. I was annoyed, but overall I was quite impressed. I suspect it used fcntl, though maybe I should ask. The NFS configuration problem went away on its own during upgrades of etch (which at the time was Debian testing). I have no idea what the problem was. What primitive does Modula 3 use on Linux systems to perform locking? Let me hope for fcntl. Or lockf. But not flock. -- hendrik From wagner at elegosoft.com Sun Mar 2 15:11:42 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Sun, 02 Mar 2008 15:11:42 +0100 Subject: [M3devel] random-acces I/O In-Reply-To: <20080302131247.GA26816@topoi.pooq.com> References: <20080229224126.GA21397@topoi.pooq.com> <1204414879.10030.8.camel@hias.m3w.org> <20080302115357.2mv96dyas4so04o0@mail.elegosoft.com> <20080302131247.GA26816@topoi.pooq.com> Message-ID: <20080302151142.qes8lz26qs8gwo44@mail.elegosoft.com> Quoting hendrik at topoi.pooq.com: > On Sun, Mar 02, 2008 at 11:53:57AM +0100, Olaf Wagner wrote: >> Quoting Dragi?a Duri? : > In Linux, locking via fcntl is NFS-safe. locking via flock is > apparently not. lockf is implemented via fcntl, so it is presumably > NFS-safe, too, although the spoecifications don't actually require it to > be implemented via fcntl. > > When I had NFS misconfigured (and I could find no information indicating > how reconfigure it) one program (monotone) could not lock any NFS files, > knwew it, and refused to operate as a result. That's the safe way > to deal with NFS misconfiguration. I was annoyed, but overall I was > quite impressed. I suspect it used fcntl, though maybe I should ask. > The NFS configuration problem went away on its own during upgrades of > etch (which at the time was Debian testing). I have no idea what the > problem was. > > What primitive does Modula 3 use on Linux systems to perform locking? > Let me hope for fcntl. Or lockf. But not flock. fcntl. See http://www.opencm3.net/doc/help/gen_html/libm3/src/os/POSIX/FilePosix.m3.html Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jayk123 at hotmail.com Sun Mar 2 15:54:02 2008 From: jayk123 at hotmail.com (Jay) Date: Sun, 2 Mar 2008 14:54:02 +0000 Subject: [M3devel] full paths to source files in debug info In-Reply-To: <200802292119.m1TLJhcB080697@camembert.async.caltech.edu> References: Your message of "Fri, 29 Feb 2008 11:11:35 CST." <47C83CC7.9060905@wichita.edu> <200802292119.m1TLJhcB080697@camembert.async.caltech.edu> Message-ID: I probably only originally tested changes involving only parse.c, and then the final parse.c change with the m3front changes. I probably did not previously test only changing m3front. I have now tested only changing m3front and the behavior is as I desire. I put back parse.c and left m3front asis. And I believe has a good chance of satisfying everyone else. The assembly source does not have the full path to any FILE, but it does have the current working directory, or directory of the input *c file -- these are always the same and so it is ambiguous, and I didn't track down the code or try where they differ. The two m3front implementation in question, are, given: ../src/Foo.m3 ../src/POSIX/FooPosix.m3 should gcc be given the paths as show above or Foo.m3 FooPosix.m3 Historically the answer was Foo.m3 and FooPosix.m3. This does not lead to what I expect for an "easier" debugging experience (oxymoron?) Changing to the ../src path does. As well I once tested setting a breakpoint by file and line number in gdb. With ../src/foo.m3, you can still break on Foo.m3:123. ok? I did not check that given Foo.m3, if the symbols still have that directory path. It is POSSIBLE that "wrecking" the source file path causes gcc to omit the directory name, but I would be surprised. It is definitely possible to have gdb always check for "../src", however this isn't general and I'm sure would not be accepted "upstream", and doesn't handle m3core, etc. - Jay > CC: m3devel at elegosoft.com> Date: Fri, 29 Feb 2008 13:19:43 -0800> From: mika at async.caltech.edu> Subject: Re: [M3devel] FW: put full paths to source files in debug info]> > "Rodney M. Bates" writes:> ...> >People can and often do move executables to different machines and also> >sometimes move source trees around on the original machine. Using simple> >file names within an executable (of which there is always exactly one)> >is invariant to all this, and a lot shorter to type/read.> ...> > Arbitrary paths are also not trustworthy on many Unix systems that run> automounters. The path you get from getcwd is likely not to be permanently> accessible on those. (You access the files via a different path.)> > Mika _________________________________________________________________ Climb to the top of the charts!?Play the word scramble challenge with star power. http://club.live.com/star_shuffle.aspx?icid=starshuffle_wlmailtextlink_jan -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Sun Mar 2 17:06:07 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 02 Mar 2008 17:06:07 +0100 Subject: [M3devel] random-acces I/O In-Reply-To: <200803020224.m222O3bP023511@camembert.async.caltech.edu> References: <200803020224.m222O3bP023511@camembert.async.caltech.edu> Message-ID: <1204473967.10030.32.camel@hias.m3w.org> On Sat, 2008-03-01 at 18:24 -0800, Mika Nystrom wrote: > Yes, I am aware of that. My point was simply that if you want > a reader and writer accessing the same data it's probably easiest > to connect together the reader and writer directly, somehow, and write > some sort of "watcher" that flushes modifications to disk "on the side" > of the interaction between the reader and writer. But not back to back, reader to writer, as in RdWrPipe. > Btw, I do have some code that does this sort of thing (not entirely > random access), and to run both on Unix and NT386GNU, yes, I do > close the file every time I "turn it around" from reading to writing > or vice versa. I have database library doint exactly that, with (of course) OO, transactions, multinode access and much more. Locking mechanisms are old hat, there is lot of options. Skip buffers, even readers/writers, and you don't have to close/reopen file. Point is, in Hendrik's case, he must define usage patterns for both Modula-3 and non-Modula-3 code, and follow them. > > Mika > > > =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: > >rdwr is package by Blair MacIntyre and it's not related to anything > >File.* except indirectly, ie if you make File Wr part of TeeWr. > > > >On Sat, 2008-03-01 at 14:49 -0800, Mika Nystrom wrote: > >> Hi Hendrik, > >> > >> The answer is, "it depends." As a matter of fact this was my biggest > >> area of trouble when porting a program from PM3 on Unix to > >> PM3/Klagenfurt for NT386GNU. On NT386GNU, the default is for > >> Modula-3 to avail itself of Windows's mandatory file locking, whereas > >> POSIX has no such thing. This means that a Modula-3 program on > >> NT386GNU can't even open a file for reading if you are "lessing" > >> it for instance. I am sure it can be worked around, but I never > >> bothered to figure out how, sorry (because the behavior I needed > >> was in fact strict locking of this kind, which I had to go to special > >> lengths to get on Unix!) > >> > >> On Unix, things are a bit more as you describe them, but not exactly. > >> You can definitely have an Rd.T and a Wr.T on a single file, but I > >> do not believe that the Rd.T and Wr.T will be aware of each other's > >> existence (I could be wrong). However, there is a package called > >> rdwr(?) that I think may have the sort of functionality you're > >> looking for. Maybe. > >> > >> File locking is strictly an OS issue as far as I know and that is not > >> something that Rd and Wr concern themselves with. > >> > >> If you need this kind of low-level access I have a feeling Rd and Wr > >> are too high-level for you. I would suspect you want to build something > > with rdwr (or some other means of getting an Rd out of a Wr or vice > >> versa) and then extend it with routines for maintaining a faithful > >> disk-based copy of what you want... using lower-level access methods. > >> > >> Oh yes Rd and Wr are documented in detail in chapter 6 of the Green Book. > >> > >> Mika > >> > >> hendrik at topoi.pooq.com writes: > >> >I need to do direct-access I/O from a Modula 3 program with a file > >> >format that is externally specified. It is necessary to both read and > >> >write in random fashion during a run of the program, prefereably without > >> >having to close or open the file each time. > >> > > >> >I expect the answers to the following questions are "Yes", but I'd > >> >appreciate confirmation: > >> > > >> > ? If I have a File.T (obtained from FS.OpenFile) can I have both a > >> > Rd.T and a Wr.T active on it simultaeously? > >> > > >> > ? Will things written using the Wr.T be immediately available > >> > for reading with the Rd.T, with no buffereing problems? > >> > > >> > ? Are there methods for locking the file -- or portions of it -- > >> > against simutaneous access by other programs, including ones written > >> > in other languages? > >> > > >> >-- hendrik > >-- > >Dragi?a Duri? -- Dragi?a Duri? From dragisha at m3w.org Sun Mar 2 22:12:39 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 02 Mar 2008 22:12:39 +0100 Subject: [M3devel] DEBUG pragma Message-ID: <1204492359.10030.40.camel@hias.m3w.org> I saw this old post by Mr. Hosking about DEBUG pragma and a way to turn it off. Is there more detailed info on it? Argument to cm3 to turn it off? Runtime switch? TIA, dd -- Dragi?a Duri? From jayk123 at hotmail.com Mon Mar 3 01:16:22 2008 From: jayk123 at hotmail.com (Jay) Date: Mon, 3 Mar 2008 00:16:22 +0000 Subject: [M3devel] full paths to source files in debug info In-Reply-To: References: Your message of "Fri, 29 Feb 2008 11:11:35 CST." <47C83CC7.9060905@wichita.edu> <200802292119.m1TLJhcB080697@camembert.async.caltech.edu> Message-ID: You do now get like: #0 Thread__Fork (M3_EMTrVz_closure=0x7ff9ef58) at ..\src\thread\WIN32\ThreadWin32.m3:653#1 0x643cc0d5 in FileBrowserVBT_M3 (M3_AcxOUs_mode=1) at ..\src\lego\FileBrowserVBT.m3:942#2 0x6bcd840e in RTLinker__RunMainBody (M3_DjPxE3_m=0x64455c60) at ../src/runtime/common/RTLinker.m3:399#3 0x6bcd82d6 in RTLinker__RunMainBody (M3_DjPxE3_m=0x6d840e20) at ../src/runtime/common/RTLinker.m3:379#4 0x6bcd82d6 in RTLinker__RunMainBody (M3_DjPxE3_m=0x6d845480) at ../src/runtime/common/RTLinker.m3:379#5 0x6bcd82d6 in RTLinker__RunMainBody (M3_DjPxE3_m=0x6d841d60) at ../src/runtime/common/RTLinker.m3:379#6 0x6bcd82d6 in RTLinker__RunMainBody (M3_DjPxE3_m=0x4614e0) at ../src/runtime/common/RTLinker.m3:379#7 0x6bcd7939 in RTLinker__AddUnitI (M3_DjPxE3_m=0x4614e0) at ../src/runtime/common/RTLinker.m3:113#8 0x6bcd79c0 in RTLinker__AddUnit (M3_DjPxE5_b=0x44a4d3)whereas presumably before you would get: #0 Thread__Fork (M3_EMTrVz_closure=0x7ff9ef58) at ThreadWin32.m3:653#1 0x643cc0d5 in FileBrowserVBT_M3 (M3_AcxOUs_mode=1) at FileBrowserVBT.m3:942#2 0x6bcd840e in RTLinker__RunMainBody (M3_DjPxE3_m=0x64455c60) at RTLinker.m3:399#3 0x6bcd82d6 in RTLinker__RunMainBody (M3_DjPxE3_m=0x6d840e20) at RTLinker.m3:379#4 0x6bcd82d6 in RTLinker__RunMainBody (M3_DjPxE3_m=0x6d845480) at RTLinker.m3:379#5 0x6bcd82d6 in RTLinker__RunMainBody (M3_DjPxE3_m=0x6d841d60) at RTLinker.m3:379#6 0x6bcd82d6 in RTLinker__RunMainBody (M3_DjPxE3_m=0x4614e0) at RTLinker.m3:379#7 0x6bcd7939 in RTLinker__AddUnitI (M3_DjPxE3_m=0x4614e0) at RTLinker.m3:113#8 0x6bcd79c0 in RTLinker__AddUnit (M3_DjPxE5_b=0x44a4d3) ok? I agree this medium level of verbosity in the ui is not useful. The gain is that gdb finds the source files automatically. - Jay From: jayk123 at hotmail.comTo: m3devel at elegosoft.comDate: Sun, 2 Mar 2008 14:54:02 +0000Subject: [M3devel] full paths to source files in debug info I probably only originally tested changes involving only parse.c, and then the final parse.c change with the m3front changes. I probably did not previously test only changing m3front. I have now tested only changing m3front and the behavior is as I desire.I put back parse.c and left m3front asis. And I believe has a good chance of satisfying everyone else. The assembly source does not have the full path to any FILE, but it does have the current working directory, or directory of the input *c file -- these are always the same and so it is ambiguous, and I didn't track down the code or try where they differ. The two m3front implementation in question, are, given: ../src/Foo.m3 ../src/POSIX/FooPosix.m3 should gcc be given the paths as show above or Foo.m3 FooPosix.m3 Historically the answer was Foo.m3 and FooPosix.m3.This does not lead to what I expect for an "easier" debugging experience (oxymoron?)Changing to the ../src path does. As well I once tested setting a breakpoint by file and line number in gdb.With ../src/foo.m3, you can still break on Foo.m3:123. ok? I did not check that given Foo.m3, if the symbols still have that directory path.It is POSSIBLE that "wrecking" the source file path causes gcc to omit the directory name,but I would be surprised. It is definitely possible to have gdb always check for "../src", however this isn't generaland I'm sure would not be accepted "upstream", and doesn't handle m3core, etc. - Jay > CC: m3devel at elegosoft.com> Date: Fri, 29 Feb 2008 13:19:43 -0800> From: mika at async.caltech.edu> Subject: Re: [M3devel] FW: put full paths to source files in debug info]> > "Rodney M. Bates" writes:> ...> >People can and often do move executables to different machines and also> >sometimes move source trees around on the original machine. Using simple> >file names within an executable (of which there is always exactly one)> >is invariant to all this, and a lot shorter to type/read.> ...> > Arbitrary paths are also not trustworthy on many Unix systems that run> automounters. The path you get from getcwd is likely not to be permanently> accessible on those. (You access the files via a different path.)> > Mika Climb to the top of the charts! Play the word scramble challenge with star power. Play now! _________________________________________________________________ Climb to the top of the charts!?Play the word scramble challenge with star power. http://club.live.com/star_shuffle.aspx?icid=starshuffle_wlmailtextlink_jan -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Mon Mar 3 01:23:28 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 2 Mar 2008 19:23:28 -0500 Subject: [M3devel] DEBUG pragma In-Reply-To: <1204492359.10030.40.camel@hias.m3w.org> References: <1204492359.10030.40.camel@hias.m3w.org> Message-ID: <2C2B845D-9A4E-439B-A81D-3F6E6F67380D@cs.purdue.edu> M3_OPTIONS("-NoDebug") should do it I think. On Mar 2, 2008, at 4:12 PM, Dragi?a Duri? wrote: > I saw this old post by Mr. Hosking about DEBUG pragma and a way to > turn > it off. Is there more detailed info on it? Argument to cm3 to turn it > off? Runtime switch? > > TIA, > dd > > -- > Dragi?a Duri? From mika at async.caltech.edu Mon Mar 3 03:34:41 2008 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 02 Mar 2008 18:34:41 -0800 Subject: [M3devel] [M3commit] tint In-Reply-To: Your message of "Mon, 03 Mar 2008 02:16:46 GMT." Message-ID: <200803030234.m232YfW8058063@camembert.async.caltech.edu> Language spec section 2.6.8, Green Book p. 52 Jay writes: >--_1ea12ebb-f3a7-4477-abce-de694783b438_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > >Tony, it is definitely confusing to claim the array has NUMBER(x) elements,= > and then initialize only some of them "correctly". >Trailing zeros I can see as maybe clear for positive values, but definitely= > confusing for negative values. > > - Jay From dragisha at m3w.org Mon Mar 3 08:56:03 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Mon, 03 Mar 2008 08:56:03 +0100 Subject: [M3devel] cminstall floating point exception Message-ID: <1204530963.10030.51.camel@hias.m3w.org> When trying to use cminstall on Fedora Core 4 box, I am getting this: Floating point exception./cminstall -q $INSTALLROOT Is this known problem with some older glibc? Something I can work around? dd -- Dragi?a Duri? From wagner at elegosoft.com Mon Mar 3 12:29:04 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Mon, 03 Mar 2008 12:29:04 +0100 Subject: [M3devel] Regression tests Message-ID: <20080303122904.qt7o6w6j8k4c04wg@mail.elegosoft.com> After noticing that some more external test results begin to appear in the CM3 tinderbox, I'd like to point out again the differences between the `lastok build' and `release build' tests. `lastok build' tests include simply building the complete standard set of packages with an already installed CM3. They indicate that o all standard packages compile o no imcompatibilities have been introduced since the last test (which would require a more sophisticated compiler bootstrap) `release build' tests include the following: o a compiler upgrade based on the last release (currently 5.4.0) including (a) two stage bootstrap and (b) forced configuration upgrade via cminstall o running of the m3-sys/m3tests test suite (compiler and runtime) --> shipping of the test report to www.opencm3.net o complete build of all packages (based on scripts/pkginfo.txt) and running of package tests --> shipping of build and test report to www.opencm3.net o building of binary distribution archives and their --> shipping to www.opencm3.net o generating the complete package documentation with m3tohtml `release build' will be extended as needed with further tests and archive generation etc. Personally, I don't think we need to run `lastok build' on all target platforms, but we should run `release build' on as many platforms as possible in order to have all the different test results listed above available. The explaining texts on the tinderbox page should probably be improved, too :-) Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From rodney.bates at wichita.edu Tue Mar 4 22:48:51 2008 From: rodney.bates at wichita.edu (Rodney M. Bates) Date: Tue, 04 Mar 2008 15:48:51 -0600 Subject: [M3devel] CG error messages on subranges of LONGINT Message-ID: <47CDC3C3.601@wichita.edu> I wrote the attached test program for testing compiler/debugger subrange bounds communication. It gets CG errors from the compiler: "../src/Subranges.m3", line 34: ** INTERNAL CG ERROR *** unaligned store_indirect type=6 s/a=32/32 "../src/Subranges.m3", line 35: ** INTERNAL CG ERROR *** unaligned store_indirect type=6 s/a=32/32 "../src/Subranges.m3", line 43: ** INTERNAL CG ERROR *** unaligned store_indirect type=6 s/a=32/32 Attempts to remove seemingly irrelevant declarations change the behavior. -- ------------------------------------------------------------- Rodney M. Bates, retired assistant professor Dept. of Computer Science, Wichita State University Wichita, KS 67260-0083 316-978-3922 rodney.bates at wichita.edu -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: Subranges.m3 URL: From hosking at cs.purdue.edu Wed Mar 5 02:45:12 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 4 Mar 2008 20:45:12 -0500 Subject: [M3devel] CG error messages on subranges of LONGINT In-Reply-To: <47CDC3C3.601@wichita.edu> References: <47CDC3C3.601@wichita.edu> Message-ID: <734E2A5D-AEA2-4C2F-AC03-B13632EB597C@cs.purdue.edu> Looks like alignment of LONGINT is not getting handled properly. I'll try to look into it. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 4, 2008, at 4:48 PM, Rodney M. Bates wrote: > I wrote the attached test program for testing compiler/debugger > subrange bounds communication. It gets CG errors from the compiler: > > "../src/Subranges.m3", line 34: ** INTERNAL CG ERROR *** unaligned > store_indirect type=6 s/a=32/32 > "../src/Subranges.m3", line 35: ** INTERNAL CG ERROR *** unaligned > store_indirect type=6 s/a=32/32 > "../src/Subranges.m3", line 43: ** INTERNAL CG ERROR *** unaligned > store_indirect type=6 s/a=32/32 > > > Attempts to remove seemingly irrelevant declarations change the > behavior. > > > -- > ------------------------------------------------------------- > Rodney M. Bates, retired assistant professor > Dept. of Computer Science, Wichita State University > Wichita, KS 67260-0083 > 316-978-3922 > rodney.bates at wichita.edu > MODULE Subranges EXPORTS Main > > ; TYPE U1 = [ 0 .. 16_7F ] > ; TYPE S1 = [ 0 .. 16_FF ] > ; TYPE U2 = [ 0 .. 16_7FFF ] > ; TYPE S2 = [ 0 .. 16_FFFF ] > ; TYPE U3 = [ 0 .. 16_7FFFFF ] > ; TYPE S3 = [ 0 .. 16_FFFFFF ] > ; TYPE U4 = [ 0 .. 16_7FFFFFFF ] > ; TYPE S4 = [ 16_FFFFFFFF .. 16_FFFFFFFF ] > > ; TYPE LU4 = [ 0L .. 16_7FFFFFFFL ] > ; TYPE LS4 = [ 0L .. 16_FFFFFFFFL ] > ; TYPE LU5 = [ 0L .. 16_7FFFFFFFFFL ] > ; TYPE LS5 = [ 0L .. 16_FFFFFFFFFFL ] > ; TYPE LU6 = [ 0L .. 16_7FFFFFFFFFFFL ] > ; TYPE LS6 = [ 0L .. 16_FFFFFFFFFFFFL ] > ; TYPE LU7 = [ 0L .. 16_7FFFFFFFFFFFFFL ] > ; TYPE LS7 = [ 0L .. 16_FFFFFFFFFFFFFFL ] > ; TYPE LU8 = [ 0L .. 16_7FFFFFFFFFFFFFFFL ] > ; TYPE LS8 = [ 0L .. 16_FFFFFFFFFFFFFFFFL ] > > ; PROCEDURE Work ( ) > > = VAR VU1 : U1 := 0 > ; VAR VS1 : S1 := 0 > ; VAR VU2 : U2 := 0 > ; VAR VS2 : S2 := 0 > ; VAR VU3 : U3 := 0 > ; VAR VS3 : S3 := 0 > ; VAR VU4 : U4 := 0 > ; VAR VS4 : S4 := 16_FFFFFFFF > > ; VAR VLU4 : LU4 := 0L > ; VAR VLS4 : LS4 := 0L > ; VAR VLU5 : LU5 := 0L > ; VAR VLS5 : LS5 := 0L > ; VAR VLU6 : LU6 := 0L > ; VAR VLS6 : LS6 := 0L > ; VAR VLU7 : LU7 := 0L > ; VAR VLS7 : LS7 := 0L > ; VAR VLU8 : LU8 := 0L > ; VAR VLS8 : LS8 := 0L > ; VAR X : INTEGER > > ; BEGIN > X := VU1 > END Work > > ; BEGIN > Work () > END Subranges > . -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Wed Mar 5 19:35:20 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Wed, 05 Mar 2008 19:35:20 +0100 Subject: [M3devel] CG error messages on subranges of LONGINT In-Reply-To: <47CDC3C3.601@wichita.edu> References: <47CDC3C3.601@wichita.edu> Message-ID: <20080305193520.3ypns4kao4gco0ss@mail.elegosoft.com> Quoting "Rodney M. Bates" : > I wrote the attached test program for testing compiler/debugger > subrange bounds communication. It gets CG errors from the compiler: > > "../src/Subranges.m3", line 34: ** INTERNAL CG ERROR *** unaligned > store_indirect type=6 s/a=32/32 > "../src/Subranges.m3", line 35: ** INTERNAL CG ERROR *** unaligned > store_indirect type=6 s/a=32/32 > "../src/Subranges.m3", line 43: ** INTERNAL CG ERROR *** unaligned > store_indirect type=6 s/a=32/32 > > Attempts to remove seemingly irrelevant declarations change the > behavior. I've added that test program as p207 to m3-sys/m3tests. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From wagner at elegosoft.com Wed Mar 5 19:39:47 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Wed, 05 Mar 2008 19:39:47 +0100 Subject: [M3devel] cminstall floating point exception In-Reply-To: <1204530963.10030.51.camel@hias.m3w.org> References: <1204530963.10030.51.camel@hias.m3w.org> Message-ID: <20080305193947.0bxs0my3wowgogow@mail.elegosoft.com> Quoting Dragi?a Duri? : > When trying to use cminstall on Fedora Core 4 box, I am getting this: > > Floating point exception./cminstall -q $INSTALLROOT > > Is this known problem with some older glibc? Something I can work > around? It's nothing that I remember and not documented on the Known Problems page. Can you get a stack backtrace with gcc and show against which libraries the program is linked (ldd)? Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From dragisha at m3w.org Wed Mar 5 20:47:15 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Wed, 05 Mar 2008 20:47:15 +0100 Subject: [M3devel] Promotion, anyone? Message-ID: <1204746435.8782.2.camel@hias.m3w.org> http://shootout.alioth.debian.org/ -- Dragi?a Duri? From dragisha at m3w.org Thu Mar 6 16:50:59 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 06 Mar 2008 16:50:59 +0100 Subject: [M3devel] What is this??? How's it possible? Message-ID: <1204818659.8782.16.camel@hias.m3w.org> *** *** runtime error: *** A compile-time type is missing. *** file "../src/runtime/common/RTLinker.i3" *** -- Dragi?a Duri? From hosking at cs.purdue.edu Thu Mar 6 17:15:30 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 6 Mar 2008 11:15:30 -0500 Subject: [M3devel] What is this??? How's it possible? In-Reply-To: <1204818659.8782.16.camel@hias.m3w.org> References: <1204818659.8782.16.camel@hias.m3w.org> Message-ID: <4AB9C403-1CF8-4621-8E90-9B5BB1F8B4B4@cs.purdue.edu> This usually means you did not rebuild a clean set of binaries. i.e., some old binaries were lying around that made old assumptions about type ids. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 6, 2008, at 10:50 AM, Dragi?a Duri? wrote: > *** > *** runtime error: > *** A compile-time type is missing. > *** file "../src/runtime/common/RTLinker.i3" > *** > > -- > Dragi?a Duri? -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Thu Mar 6 18:23:19 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Thu, 06 Mar 2008 18:23:19 +0100 Subject: [M3devel] What is this??? How's it possible? In-Reply-To: <4AB9C403-1CF8-4621-8E90-9B5BB1F8B4B4@cs.purdue.edu> References: <1204818659.8782.16.camel@hias.m3w.org> <4AB9C403-1CF8-4621-8E90-9B5BB1F8B4B4@cs.purdue.edu> Message-ID: <20080306182319.u2flyt1uskg0owkg@mail.elegosoft.com> Quoting Tony Hosking : > This usually means you did not rebuild a clean set of binaries. i.e., > some old binaries were lying around that made old assumptions about > type ids. > On Mar 6, 2008, at 10:50 AM, Dragi?a Duri? wrote: > >> *** >> *** runtime error: >> *** A compile-time type is missing. >> *** file "../src/runtime/common/RTLinker.i3" >> *** Well, we'd have to admit it _is_ a little bit cryptic and may not be very helpful and encouraging to the casual user :-) In theory, CM3 should be able to detect such version conflicts even at link time (using fingerprints for compiled units etc.). I seem to remember that older M3 versions (i.e. DEC SRC M3) were more reliable in detecting such things; but my memory may be wrong or the environment completely different (no shared objects for example). I'd like to better understand this kind of error: does it only show up in connection with old libraries, only in connection with old runtime modules (some special rules here), only when using shared libraries and so on. Conceptually, it's a kind of version error and should be notified as early as possible. Then we can think of ways to (a) detect it earlier (if possible) and (b) improve the user notification ;-) Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From dragisha at m3w.org Thu Mar 6 18:35:07 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 06 Mar 2008 18:35:07 +0100 Subject: [M3devel] What is this??? How's it possible? In-Reply-To: <4AB9C403-1CF8-4621-8E90-9B5BB1F8B4B4@cs.purdue.edu> References: <1204818659.8782.16.camel@hias.m3w.org> <4AB9C403-1CF8-4621-8E90-9B5BB1F8B4B4@cs.purdue.edu> Message-ID: <1204824907.8782.20.camel@hias.m3w.org> It was my mistake, of course. Dynamically loaded libraries were not up to date. And day was too hard on me :). dd On Thu, 2008-03-06 at 11:15 -0500, Tony Hosking wrote: > This usually means you did not rebuild a clean set of binaries. i.e., > some old binaries were lying around that made old assumptions about > type ids. > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > On Mar 6, 2008, at 10:50 AM, Dragi?a Duri? wrote: > > > *** > > *** runtime error: > > *** A compile-time type is missing. > > *** file "../src/runtime/common/RTLinker.i3" > > *** > > > > -- > > Dragi?a Duri? > > > -- Dragi?a Duri? From dragisha at m3w.org Thu Mar 6 19:30:33 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 06 Mar 2008 19:30:33 +0100 Subject: [M3devel] What is this??? How's it possible? In-Reply-To: <20080306182319.u2flyt1uskg0owkg@mail.elegosoft.com> References: <1204818659.8782.16.camel@hias.m3w.org> <4AB9C403-1CF8-4621-8E90-9B5BB1F8B4B4@cs.purdue.edu> <20080306182319.u2flyt1uskg0owkg@mail.elegosoft.com> Message-ID: <1204828233.8782.34.camel@hias.m3w.org> My situation was like this: In hm3-m3lib0 library I have two types: Killer.T Killer.Client That library is linked to xli binary, dynamically. hm3-web is one of libraries I am loading dynamically at runtime (using dlopen and co). Its FastCGI and WebServer modules are using above mentioned two types. I've changed hm3-m3lib0, and hm3-web remained unchanged. So, when I invoke this code: dl:=NEW(DL.T).init(pIn, global); FOR i:=1 TO r.size() DO RTLinker.AddUnit(LOOPHOLE(dl.sym(NARROW(r.get(i-1), TEXT) & "_M3").addr, RT0.Binder)); (* IO.Put("Adding: " & NARROW(r.get(i-1), TEXT) & "\n"); *) END; RTLinker.AddUnit fails. Message is one I've sent. When this error happens during "normal" initialization, runtime error is obviuosly normal behaviour. You can't ignore one of linked libraries and expect system to work. But, as some of these additional libraries (in my case scripting engine plugins) are optional, it's probably wise to allow interception of this runtime error. This is probably what RuntimeError.E is for? I'll try it tonight. dd On Thu, 2008-03-06 at 18:23 +0100, Olaf Wagner wrote: > Quoting Tony Hosking : > > > This usually means you did not rebuild a clean set of binaries. i.e., > > some old binaries were lying around that made old assumptions about > > type ids. > > > On Mar 6, 2008, at 10:50 AM, Dragi?a Duri? wrote: > > > >> *** > >> *** runtime error: > >> *** A compile-time type is missing. > >> *** file "../src/runtime/common/RTLinker.i3" > >> *** > > Well, we'd have to admit it _is_ a little bit cryptic and may not > be very helpful and encouraging to the casual user :-) > > In theory, CM3 should be able to detect such version conflicts > even at link time (using fingerprints for compiled units etc.). > I seem to remember that older M3 versions (i.e. DEC SRC M3) were > more reliable in detecting such things; but my memory may be wrong > or the environment completely different (no shared objects for example). > > I'd like to better understand this kind of error: does it only show > up in connection with old libraries, only in connection with old > runtime modules (some special rules here), only when using shared > libraries and so on. Conceptually, it's a kind of version error and > should be notified as early as possible. > > Then we can think of ways to (a) detect it earlier (if possible) and > (b) improve the user notification ;-) > > Olaf -- Dragi?a Duri? From dragisha at m3w.org Thu Mar 6 20:02:32 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 06 Mar 2008 20:02:32 +0100 Subject: [M3devel] What is this??? How's it possible? In-Reply-To: <20080306182319.u2flyt1uskg0owkg@mail.elegosoft.com> References: <1204818659.8782.16.camel@hias.m3w.org> <4AB9C403-1CF8-4621-8E90-9B5BB1F8B4B4@cs.purdue.edu> <20080306182319.u2flyt1uskg0owkg@mail.elegosoft.com> Message-ID: <1204830152.8782.39.camel@hias.m3w.org> Tried: TRY RTLinker.AddUnit(LOOPHOLE(dl.sym(NARROW(r.get(i-1), TEXT) & "_M3").addr, RT0.Binder)); EXCEPT RuntimeError.E(expl) => <* DEBUG "Error adding " & r.get(i-1) *> END; It still fails with: *** *** runtime error: *** A compile-time type is missing. *** file "../src/runtime/common/RTLinker.i3" *** And RTType.Fail procedure, only one getting RuntimeError.MisingType as argument, looks like this: PROCEDURE Fail (rte: RTE; m: RT0.ModulePtr; x, y: ADDRESS) = <*FATAL ANY*> ... RTException.Raise (a); END Fail; But, I am getting above mentioned error. dd On Thu, 2008-03-06 at 18:23 +0100, Olaf Wagner wrote: > Quoting Tony Hosking : > > > This usually means you did not rebuild a clean set of binaries. i.e., > > some old binaries were lying around that made old assumptions about > > type ids. > > > On Mar 6, 2008, at 10:50 AM, Dragi?a Duri? wrote: > > > >> *** > >> *** runtime error: > >> *** A compile-time type is missing. > >> *** file "../src/runtime/common/RTLinker.i3" > >> *** > > Well, we'd have to admit it _is_ a little bit cryptic and may not > be very helpful and encouraging to the casual user :-) > > In theory, CM3 should be able to detect such version conflicts > even at link time (using fingerprints for compiled units etc.). > I seem to remember that older M3 versions (i.e. DEC SRC M3) were > more reliable in detecting such things; but my memory may be wrong > or the environment completely different (no shared objects for example). > > I'd like to better understand this kind of error: does it only show > up in connection with old libraries, only in connection with old > runtime modules (some special rules here), only when using shared > libraries and so on. Conceptually, it's a kind of version error and > should be notified as early as possible. > > Then we can think of ways to (a) detect it earlier (if possible) and > (b) improve the user notification ;-) > > Olaf -- Dragi?a Duri? From jayk123 at hotmail.com Fri Mar 7 07:00:46 2008 From: jayk123 at hotmail.com (Jay) Date: Fri, 7 Mar 2008 06:00:46 +0000 Subject: [M3devel] thread.fork getting nil on nt386gnu? In-Reply-To: <6346DE25-6495-49D0-A990-F48B989218AC@cs.purdue.edu> References: <20080224134642.48e7fowms8ooowo8@mail.elegosoft.com> <20080224143053.fdtxd48fkskggws0@mail.elegosoft.com> <20080224153541.59c105ro8e88480g@mail.elegosoft.com> <6346DE25-6495-49D0-A990-F48B989218AC@cs.purdue.edu> Message-ID: This asssertion is absent from the two other thread implementations, and is dependent on a loophole to underlying details that I didn't track down to determine if they are corrrect. So I merely removed this. Still can't connect to the X server for some reason.. - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Tue, 26 Feb 2008 13:29:28 -0500Looks like a corrupted/uninitialized type (method table contains a NIL pointer...). Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 25, 2008, at 1:15 PM, Jay wrote: any obvious ideas? I'll have to pause for a bit again..Jay at jay-win1 /cygdrive/c/dev2/cm3.2/m3-ui/juno-2/juno-app/NT386GNU/Juno.exe****** runtime error:*** Thread client error: NIL apply method passed to Thread.Fork!*** file "ThreadWin32.m3", line 578*** 14837 [sig] Juno 3908 c:\dev2\cm3.2\m3-ui\juno-2\juno-app\NT386GNU\Juno.exe: *** fatal error - called with threadlist_ix -1Hangup - Jay Need to know the score, the latest news, or you need your Hotmail?-get your "fix". Check it out. _________________________________________________________________ Shed those extra pounds with MSN and The Biggest Loser! http://biggestloser.msn.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Fri Mar 7 08:57:03 2008 From: jayk123 at hotmail.com (Jay) Date: Fri, 7 Mar 2008 07:57:03 +0000 Subject: [M3devel] multi arch installs? Message-ID: background: Windows has $PATH and $PATHEXT.$PATHEXT is used by the usual command line -- cmd -- to help find "executables". On one of my systems: C:\>set pathPath=C:\Python25\;C:\Tcl\bin;C:\Perl\site\bin;C:\Perl\bin;C:\Perl\bin;c:\ruby\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\QuickTime\QTSystem\PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.RB;.RBW;.tcl;.py;.pyw (oops, I'll go clean it up..) "Executables" are not just .exes. The commands ftype and assoc combine to enable running "anything", based on extension. For example: C:\>assoc .exe.exe=exefile C:\>ftype exefileexefile="%1" %* C:\>assoc .js.js=JSFile C:\>assoc .vbs.vbs=VBSFile C:\>assoc .py.py=Python.File C:\>assoc .tcl.tcl=ActiveTclScript C:\>ftype activetclscriptactivetclscript="C:\Tcl\bin\wish85.exe" "%1" %* C:\>ftype python.filepython.file="C:\Python25\python.exe" "%1" %* C:\>ftype jsfilejsfile=%SystemRoot%\System32\WScript.exe "%1" %* C:\>ftype vbsfilevbsfile=%SystemRoot%\System32\WScript.exe "%1" %* If you ignore the need for executables to have extensions, this provides much ofthe feature of Unix where any file is executable, possibly via a specified interpreter.It is better and worse.It is worse in that extensions are needed.It is better in that files don't have to give the possibly system-specific pathto their interpreter.It is worse in that files cannot give the path to the interpreter. It is worse in that it is a feature of the command line interpreter, instead of lower layers (kernel) as I believe on Unix. Now, I have a small problem of convenience.And you all MIGHT have a "problem" of "What is Posix?" My c:\cm3 contains a mix of NT386 and NT386GNU files. c:\cm3\pkg\cm3\NT386\cm3.exe c:\cm3\pkg\cm3\NT386GNU\cm3.exe ok but then what is c:\cm3\bin\cm3.exe ? Currently it is the most recently shipped file. For the case of .dlls, I have enabled "colocation" by changing NT386GNU .dlls to end in .so.So I have: c:\cm3\bin\m3.dll c:\cm3\bin\m3.so Probably this should be libm3.so and probably will be some day. I would a system whereby somehow the two targets can coexist better. Perhaps I should just suck it up and have c:\cm3\nt386 and c:\cm3\nt386gnu.Perhaps. Quick experimentation shows that you can have an "empty" extension. set PATHEXT=.;%PATHEXT% cd \cygwin\bin copy ls.exe foo .\foo => It works. You don't actually have to do anything with assoc and ftype. Oops. I didn't have to bring that up. The order of %PATHEXT% varies. So, the proposal is, in the interest of killing two birds with one stone, neither of whichwas necessarily even alive at the start, is that NT386GNU use no extension for .exes. \cm3\bin\cm3 is NT386GNU \cm3\bin\cm3.exe is NT386 (or NT386MINGNU) I started thinking about this partly because "Services for Unix" makes .exes with no extension.And because I suspect the handling here in Cygwin is hacky.There is various code such that if someone looks for "foo", they might find "foo.exe" instead. Upsides: "It works". Downsides: "foo" used to resolve via %PATH% + %PATHEXT% search..and it still does, but putting . at the start will find foo where foo.exe was more common. It is not general. This scheme does not enable any other "multi arch" install. Alternate proposal with a question: How should "multi arch" installs be achieved? Probably via making \cm3\bin\foo on Posix and \cm3\bin\foo.cmd on NT that somehow sniffs its environment, perhaps with uname, and then runs \cm3\pkg\foo\?\foo (or foo.exe) Upsides: It works, at least for executables. It is more general than my previous suggestion. Downsides: Extra processes created. What about .dlls/.sos? Maybe nobody cares? I find the idea of one unified file system/archive that works on "everything" fairly compelling. Yeah, it's big. Yeah, it's slower to install. But it means one download can be had independently of platform. Maybe simply \cm3\platform instead of \cm3? upsides: it works is general no new mechisms invented! downsides: none? too boring? :) \cm3\platform\pkg\foo\platform is redundant. Less able to do "cross" scenarios that I'm using a lot lately, where NT386 and NT386GNU can build each other? Not clear. Probably just a matter of $PATH and $CM3_INSTALL not coinciding. cm3 -ship could see if CM3_INSTALL ends in "\platform" and in that case not put in the second instance, but this possibly asking for trouble. Probably a bunch of code would have to learn about this. Just live with the duplication of "platform", in the unusual situation of a "multi arch" file system? Maybe there is nothing to do here. The matter of what the file extension should be on NT386GNU perhaps is a different matter, left up to the hypothetical users of NT386GNU (not me). "What is Posix?" Executables with no extension? Or don't care? Cygwin usually uses ".exe", but of course Cygwin bash allows running stuff with "shebang". Microsoft/Interix "Services for Unix" puts no extension on executables. I found that a bit impressive, maybe worth striving for, and then surprisingly easy. I am almost surprised Cygwin doesn't do this. I don't know if putting "." in $PATHEXT is any worse than adding \cygwin\bin to $PATH. "If it is good enough for Cygwin, it is good enough for CM3/NT386GNU users?" I am not proposing extensionless executables for NT386. That would just be considered wierd. However the "wrapper .cmd" files, maybe. - Jay _________________________________________________________________ Shed those extra pounds with MSN and The Biggest Loser! http://biggestloser.msn.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Fri Mar 7 14:41:23 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 7 Mar 2008 08:41:23 -0500 Subject: [M3devel] thread.fork getting nil on nt386gnu? In-Reply-To: References: <20080224134642.48e7fowms8ooowo8@mail.elegosoft.com> <20080224143053.fdtxd48fkskggws0@mail.elegosoft.com> <20080224153541.59c105ro8e88480g@mail.elegosoft.com> <6346DE25-6495-49D0-A990-F48B989218AC@cs.purdue.edu> Message-ID: <5F9985B2-99AE-4C63-8624-3A33F9BD0569@cs.purdue.edu> It is always dangerous to remove assertions that have worked in the past. Better to understand why they no longer work. Anyway, what this suggests is something in alignment because what the assertion checks is that the method table for the Thread.Closure object passed to Thread.Fork contains a non-NIL function pointer in the "apply" slot. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 7, 2008, at 1:00 AM, Jay wrote: > This asssertion is absent from the two other thread implementations, > and is dependent on a loophole to underlying details that I didn't > track down to determine if they are corrrect. So I merely removed > this. > > Still can't connect to the X server for some reason.. > > - Jay > CC: m3devel at elegosoft.com > From: hosking at cs.purdue.edu > To: jayk123 at hotmail.com > Subject: Re: [M3devel] thread.fork getting nil on nt386gnu? > Date: Tue, 26 Feb 2008 13:29:28 -0500 > > Looks like a corrupted/uninitialized type (method table contains a > NIL pointer...). > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > On Feb 25, 2008, at 1:15 PM, Jay wrote: > > any obvious ideas? I'll have to pause for a bit again.. > > Jay at jay-win1 /cygdrive/c/dev2/cm3.2/m3-ui/juno-2/juno-app/NT386GNU/ > Juno.exe > > *** > *** runtime error: > *** Thread client error: NIL apply method passed to Thread.Fork! > *** file "ThreadWin32.m3", line 578 > *** > 14837 [sig] Juno 3908 c:\dev2\cm3.2\m3-ui\juno-2\juno-app\NT386GNU > \Juno.exe: * > ** fatal error - called with threadlist_ix -1 > Hangup > > - Jay > > Need to know the score, the latest news, or you need your Hotmail?- > get your "fix". Check it out. > > > Shed those extra pounds with MSN and The Biggest Loser! Learn more. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Fri Mar 7 20:21:38 2008 From: jayk123 at hotmail.com (Jay) Date: Fri, 7 Mar 2008 19:21:38 +0000 Subject: [M3devel] thread.fork getting nil on nt386gnu? In-Reply-To: <5F9985B2-99AE-4C63-8624-3A33F9BD0569@cs.purdue.edu> References: <20080224134642.48e7fowms8ooowo8@mail.elegosoft.com> <20080224143053.fdtxd48fkskggws0@mail.elegosoft.com> <20080224153541.59c105ro8e88480g@mail.elegosoft.com> <6346DE25-6495-49D0-A990-F48B989218AC@cs.purdue.edu> <5F9985B2-99AE-4C63-8624-3A33F9BD0569@cs.purdue.edu> Message-ID: assertions are also notoriously wrong. Two out of three thread implementations don't have it.. - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Fri, 7 Mar 2008 08:41:23 -0500 It is always dangerous to remove assertions that have worked in the past. Better to understand why they no longer work. Anyway, what this suggests is something in alignment because what the assertion checks is that the method table for the Thread.Closure object passed to Thread.Fork contains a non-NIL function pointer in the "apply" slot. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 7, 2008, at 1:00 AM, Jay wrote: This asssertion is absent from the two other thread implementations, and is dependent on a loophole to underlying details that I didn't track down to determine if they are corrrect. So I merely removed this. Still can't connect to the X server for some reason.. - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Tue, 26 Feb 2008 13:29:28 -0500Looks like a corrupted/uninitialized type (method table contains a NIL pointer...). Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 25, 2008, at 1:15 PM, Jay wrote: any obvious ideas? I'll have to pause for a bit again..Jay at jay-win1 /cygdrive/c/dev2/cm3.2/m3-ui/juno-2/juno-app/NT386GNU/Juno.exe****** runtime error:*** Thread client error: NIL apply method passed to Thread.Fork!*** file "ThreadWin32.m3", line 578*** 14837 [sig] Juno 3908 c:\dev2\cm3.2\m3-ui\juno-2\juno-app\NT386GNU\Juno.exe: *** fatal error - called with threadlist_ix -1Hangup - Jay Need to know the score, the latest news, or you need your Hotmail?-get your "fix". Check it out. Shed those extra pounds with MSN and The Biggest Loser! Learn more. _________________________________________________________________ Connect and share in new ways with Windows Live. http://www.windowslive.com/share.html?ocid=TXT_TAGHM_Wave2_sharelife_012008 -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Fri Mar 7 20:48:29 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 7 Mar 2008 14:48:29 -0500 Subject: [M3devel] thread.fork getting nil on nt386gnu? In-Reply-To: References: <20080224134642.48e7fowms8ooowo8@mail.elegosoft.com> <20080224143053.fdtxd48fkskggws0@mail.elegosoft.com> <20080224153541.59c105ro8e88480g@mail.elegosoft.com> <6346DE25-6495-49D0-A990-F48B989218AC@cs.purdue.edu> <5F9985B2-99AE-4C63-8624-3A33F9BD0569@cs.purdue.edu> Message-ID: <4CF63A9C-93DD-486D-825C-A07630AD13D6@cs.purdue.edu> Umm. If the assertion is wrong then how has this code ever run before... On Mar 7, 2008, at 2:21 PM, Jay wrote: > assertions are also notoriously wrong. > Two out of three thread implementations don't have it.. > > - Jay > > CC: m3devel at elegosoft.com > From: hosking at cs.purdue.edu > To: jayk123 at hotmail.com > Subject: Re: [M3devel] thread.fork getting nil on nt386gnu? > Date: Fri, 7 Mar 2008 08:41:23 -0500 > > It is always dangerous to remove assertions that have worked in the > past. Better to understand why they no longer work. Anyway, what > this suggests is something in alignment because what the assertion > checks is that the method table for the Thread.Closure object passed > to Thread.Fork contains a non-NIL function pointer in the "apply" > slot. > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > On Mar 7, 2008, at 1:00 AM, Jay wrote: > > This asssertion is absent from the two other thread implementations, > and is dependent on a loophole to underlying details that I didn't > track down to determine if they are corrrect. So I merely removed > this. > > Still can't connect to the X server for some reason.. > > - Jay > CC: m3devel at elegosoft.com > From: hosking at cs.purdue.edu > To: jayk123 at hotmail.com > Subject: Re: [M3devel] thread.fork getting nil on nt386gnu? > Date: Tue, 26 Feb 2008 13:29:28 -0500 > > Looks like a corrupted/uninitialized type (method table contains a > NIL pointer...). > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > On Feb 25, 2008, at 1:15 PM, Jay wrote: > > any obvious ideas? I'll have to pause for a bit again.. > > Jay at jay-win1 /cygdrive/c/dev2/cm3.2/m3-ui/juno-2/juno-app/NT386GNU/ > Juno.exe > > *** > *** runtime error: > *** Thread client error: NIL apply method passed to Thread.Fork! > *** file "ThreadWin32.m3", line 578 > *** > 14837 [sig] Juno 3908 c:\dev2\cm3.2\m3-ui\juno-2\juno-app\NT386GNU > \Juno.exe: * > ** fatal error - called with threadlist_ix -1 > Hangup > > - Jay > > Need to know the score, the latest news, or you need your Hotmail?- > get your "fix". Check it out. > > > Shed those extra pounds with MSN and The Biggest Loser! Learn more. > > > Connect and share in new ways with Windows Live. Get it now! -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Fri Mar 7 22:38:58 2008 From: jayk123 at hotmail.com (Jay) Date: Fri, 7 Mar 2008 21:38:58 +0000 Subject: [M3devel] thread.fork getting nil on nt386gnu? In-Reply-To: <4CF63A9C-93DD-486D-825C-A07630AD13D6@cs.purdue.edu> References: <20080224134642.48e7fowms8ooowo8@mail.elegosoft.com> <20080224143053.fdtxd48fkskggws0@mail.elegosoft.com> <20080224153541.59c105ro8e88480g@mail.elegosoft.com> <6346DE25-6495-49D0-A990-F48B989218AC@cs.purdue.edu> <5F9985B2-99AE-4C63-8624-3A33F9BD0569@cs.purdue.edu> <4CF63A9C-93DD-486D-825C-A07630AD13D6@cs.purdue.edu> Message-ID: Assertions, like a lot of code, often succeed by luck. There's little point in asserting a pointer isn't null right before dereferencing it. You are going to crash on one line or the next. - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Fri, 7 Mar 2008 14:48:29 -0500Umm. If the assertion is wrong then how has this code ever run before... On Mar 7, 2008, at 2:21 PM, Jay wrote: assertions are also notoriously wrong.Two out of three thread implementations don't have it.. - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Fri, 7 Mar 2008 08:41:23 -0500 It is always dangerous to remove assertions that have worked in the past. Better to understand why they no longer work. Anyway, what this suggests is something in alignment because what the assertion checks is that the method table for the Thread.Closure object passed to Thread.Fork contains a non-NIL function pointer in the "apply" slot. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 7, 2008, at 1:00 AM, Jay wrote: This asssertion is absent from the two other thread implementations, and is dependent on a loophole to underlying details that I didn't track down to determine if they are corrrect. So I merely removed this. Still can't connect to the X server for some reason.. - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Tue, 26 Feb 2008 13:29:28 -0500Looks like a corrupted/uninitialized type (method table contains a NIL pointer...). Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 25, 2008, at 1:15 PM, Jay wrote: any obvious ideas? I'll have to pause for a bit again..Jay at jay-win1 /cygdrive/c/dev2/cm3.2/m3-ui/juno-2/juno-app/NT386GNU/Juno.exe****** runtime error:*** Thread client error: NIL apply method passed to Thread.Fork!*** file "ThreadWin32.m3", line 578*** 14837 [sig] Juno 3908 c:\dev2\cm3.2\m3-ui\juno-2\juno-app\NT386GNU\Juno.exe: *** fatal error - called with threadlist_ix -1Hangup - Jay Need to know the score, the latest news, or you need your Hotmail?-get your "fix". Check it out. Shed those extra pounds with MSN and The Biggest Loser! Learn more. Connect and share in new ways with Windows Live. Get it now! _________________________________________________________________ Shed those extra pounds with MSN and The Biggest Loser! http://biggestloser.msn.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Sat Mar 8 00:17:55 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 7 Mar 2008 18:17:55 -0500 Subject: [M3devel] thread.fork getting nil on nt386gnu? In-Reply-To: References: <20080224134642.48e7fowms8ooowo8@mail.elegosoft.com> <20080224143053.fdtxd48fkskggws0@mail.elegosoft.com> <20080224153541.59c105ro8e88480g@mail.elegosoft.com> <6346DE25-6495-49D0-A990-F48B989218AC@cs.purdue.edu> <5F9985B2-99AE-4C63-8624-3A33F9BD0569@cs.purdue.edu> <4CF63A9C-93DD-486D-825C-A07630AD13D6@cs.purdue.edu> Message-ID: This wasn't an ASSERT assertion. It was an explicit check that the apply method was not null, along with a helpful message to the user. On Mar 7, 2008, at 4:38 PM, Jay wrote: > Assertions, like a lot of code, often succeed by luck. > There's little point in asserting a pointer isn't null right before > dereferencing it. You are going to crash on one line or the next. > > - Jay > > CC: m3devel at elegosoft.com > From: hosking at cs.purdue.edu > To: jayk123 at hotmail.com > Subject: Re: [M3devel] thread.fork getting nil on nt386gnu? > Date: Fri, 7 Mar 2008 14:48:29 -0500 > > Umm. If the assertion is wrong then how has this code ever run > before... > > On Mar 7, 2008, at 2:21 PM, Jay wrote: > > assertions are also notoriously wrong. > Two out of three thread implementations don't have it.. > > - Jay > > CC: m3devel at elegosoft.com > From: hosking at cs.purdue.edu > To: jayk123 at hotmail.com > Subject: Re: [M3devel] thread.fork getting nil on nt386gnu? > Date: Fri, 7 Mar 2008 08:41:23 -0500 > > It is always dangerous to remove assertions that have worked in the > past. Better to understand why they no longer work. Anyway, what > this suggests is something in alignment because what the assertion > checks is that the method table for the Thread.Closure object passed > to Thread.Fork contains a non-NIL function pointer in the "apply" > slot. > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > On Mar 7, 2008, at 1:00 AM, Jay wrote: > > This asssertion is absent from the two other thread implementations, > and is dependent on a loophole to underlying details that I didn't > track down to determine if they are corrrect. So I merely removed > this. > > Still can't connect to the X server for some reason.. > > - Jay > CC: m3devel at elegosoft.com > From: hosking at cs.purdue.edu > To: jayk123 at hotmail.com > Subject: Re: [M3devel] thread.fork getting nil on nt386gnu? > Date: Tue, 26 Feb 2008 13:29:28 -0500 > > Looks like a corrupted/uninitialized type (method table contains a > NIL pointer...). > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > On Feb 25, 2008, at 1:15 PM, Jay wrote: > > any obvious ideas? I'll have to pause for a bit again.. > > Jay at jay-win1 /cygdrive/c/dev2/cm3.2/m3-ui/juno-2/juno-app/NT386GNU/ > Juno.exe > > *** > *** runtime error: > *** Thread client error: NIL apply method passed to Thread.Fork! > *** file "ThreadWin32.m3", line 578 > *** > 14837 [sig] Juno 3908 c:\dev2\cm3.2\m3-ui\juno-2\juno-app\NT386GNU > \Juno.exe: * > ** fatal error - called with threadlist_ix -1 > Hangup > > - Jay > > Need to know the score, the latest news, or you need your Hotmail?- > get your "fix". Check it out. > > > Shed those extra pounds with MSN and The Biggest Loser! Learn more. > > > Connect and share in new ways with Windows Live. Get it now! > > > Shed those extra pounds with MSN and The Biggest Loser! Learn more. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Sat Mar 8 01:57:37 2008 From: jayk123 at hotmail.com (Jay) Date: Sat, 8 Mar 2008 00:57:37 +0000 Subject: [M3devel] thread.fork getting nil on nt386gnu? In-Reply-To: References: <20080224134642.48e7fowms8ooowo8@mail.elegosoft.com> <20080224143053.fdtxd48fkskggws0@mail.elegosoft.com> <20080224153541.59c105ro8e88480g@mail.elegosoft.com> <6346DE25-6495-49D0-A990-F48B989218AC@cs.purdue.edu> <5F9985B2-99AE-4C63-8624-3A33F9BD0569@cs.purdue.edu> <4CF63A9C-93DD-486D-825C-A07630AD13D6@cs.purdue.edu> Message-ID: Same thing.. I should look into what the underlying representation is and why it changed from "working" to non working. Note that this code runs about three times in the relevant scenario and only "fails" the third time, but yet, I'm pretty sure but will double check, the error message is plain wrong. I think I saw it run the thread function, it wasn't actually NULL. I'll have to double check that though. Imho, it could /easily/ be inspecting other nearby data that just happens to usually not be zero on NT386 but happens to sometimes be zero on NT386GNU. Without commiting the change, running the same code on pthreads/alarmthreads on other platforms might be "interesting". I wouldn't bet either way. I find there are bugs /everywhere/. Code that is trying to find bugs is no more immune from bugs than the code it is attempting to check. More code leads to more bugs. More asserts lead to more incorrect asserts. More runtime checks lead to more incorrect runtime checks. etc. Of course, checks are still good, and like everything else they must be written carefully and like everything else, when things go apparently wrong, pretty much all code should be assumed wrong until proven otherwise (read it, run it, read what it depends on, etc.). Code is guilty until proven innocent, many times. One rotten apple spoils the bunch.... - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Fri, 7 Mar 2008 18:17:55 -0500 This wasn't an ASSERT assertion. It was an explicit check that the apply method was not null, along with a helpful message to the user. On Mar 7, 2008, at 4:38 PM, Jay wrote: Assertions, like a lot of code, often succeed by luck.There's little point in asserting a pointer isn't null right before dereferencing it. You are going to crash on one line or the next. - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Fri, 7 Mar 2008 14:48:29 -0500Umm. If the assertion is wrong then how has this code ever run before... On Mar 7, 2008, at 2:21 PM, Jay wrote: assertions are also notoriously wrong.Two out of three thread implementations don't have it.. - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Fri, 7 Mar 2008 08:41:23 -0500 It is always dangerous to remove assertions that have worked in the past. Better to understand why they no longer work. Anyway, what this suggests is something in alignment because what the assertion checks is that the method table for the Thread.Closure object passed to Thread.Fork contains a non-NIL function pointer in the "apply" slot. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 7, 2008, at 1:00 AM, Jay wrote: This asssertion is absent from the two other thread implementations, and is dependent on a loophole to underlying details that I didn't track down to determine if they are corrrect. So I merely removed this. Still can't connect to the X server for some reason.. - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Tue, 26 Feb 2008 13:29:28 -0500Looks like a corrupted/uninitialized type (method table contains a NIL pointer...). Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 25, 2008, at 1:15 PM, Jay wrote: any obvious ideas? I'll have to pause for a bit again..Jay at jay-win1 /cygdrive/c/dev2/cm3.2/m3-ui/juno-2/juno-app/NT386GNU/Juno.exe****** runtime error:*** Thread client error: NIL apply method passed to Thread.Fork!*** file "ThreadWin32.m3", line 578*** 14837 [sig] Juno 3908 c:\dev2\cm3.2\m3-ui\juno-2\juno-app\NT386GNU\Juno.exe: *** fatal error - called with threadlist_ix -1Hangup - Jay Need to know the score, the latest news, or you need your Hotmail?-get your "fix". Check it out. Shed those extra pounds with MSN and The Biggest Loser! Learn more. Connect and share in new ways with Windows Live. Get it now! Shed those extra pounds with MSN and The Biggest Loser! Learn more. _________________________________________________________________ Helping your favorite cause is as easy as instant messaging.?You IM, we give. http://im.live.com/Messenger/IM/Home/?source=text_hotmail_join -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Sat Mar 8 02:23:18 2008 From: jayk123 at hotmail.com (Jay) Date: Sat, 8 Mar 2008 01:23:18 +0000 Subject: [M3devel] FW: thread.fork getting nil on nt386gnu? In-Reply-To: References: <20080224134642.48e7fowms8ooowo8@mail.elegosoft.com> <20080224143053.fdtxd48fkskggws0@mail.elegosoft.com> <20080224153541.59c105ro8e88480g@mail.elegosoft.com> <6346DE25-6495-49D0-A990-F48B989218AC@cs.purdue.edu> <5F9985B2-99AE-4C63-8624-3A33F9BD0569@cs.purdue.edu> <4CF63A9C-93DD-486D-825C-A07630AD13D6@cs.purdue.edu> Message-ID: My mail keeps getting truncated. I'll at least pay for hotmail to remove the annoying ads, maybe that'll help.. From: jayk123 at hotmail.com Same thing..I should look into what the underlying representation is and why it changed from "working" to non working.Note that this code runs about three times in the relevant scenario and only "fails" the third time, but yet, I'm pretty sure but will double check, the error message is plain wrong. I think I saw it run the thread function, it wasn't actually NULL. I'll have to double check that though. Imho, it could /easily/ be inspecting other nearby data that just happens to usually not be zero on NT386 but happens to sometimes be zero on NT386GNU. Without commiting the change, running the same code on pthreads/alarmthreads on other platforms might be "interesting". I wouldn't bet either way. I find there are bugs /everywhere/. Code that is trying to find bugs is no more immune from bugs than the code it is attempting to check. More code leads to more bugs. More asserts lead to more incorrect asserts. More runtime checks lead to more incorrect runtime checks. etc. Of course, checks are still good, and like everything else they must be written carefully and like everything else, when things go apparently wrong, pretty much all code should be assumed wrong until proven otherwise (read it, run it, read what it depends on, etc.). Code is guilty until proven innocent, many times. One rotten apple spoils the bunch.... - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Fri, 7 Mar 2008 18:17:55 -0500 This wasn't an ASSERT assertion. It was an explicit check that the apply method was not null, along with a helpful message to the user. On Mar 7, 2008, at 4:38 PM, Jay wrote: Assertions, like a lot of code, often succeed by luck.There's little point in asserting a pointer isn't null right before dereferencing it. You are going to crash on one line or the next. - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Fri, 7 Mar 2008 14:48:29 -0500Umm. If the assertion is wrong then how has this code ever run before... On Mar 7, 2008, at 2:21 PM, Jay wrote: assertions are also notoriously wrong.Two out of three thread implementations don't have it.. - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Fri, 7 Mar 2008 08:41:23 -0500 It is always dangerous to remove assertions that have worked in the past. Better to understand why they no longer work. Anyway, what this suggests is something in alignment because what the assertion checks is that the method table for the Thread.Closure object passed to Thread.Fork contains a non-NIL function pointer in the "apply" slot. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 7, 2008, at 1:00 AM, Jay wrote: This asssertion is absent from the two other thread implementations, and is dependent on a loophole to underlying details that I didn't track down to determine if they are corrrect. So I merely removed this. Still can't connect to the X server for some reason.. - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Tue, 26 Feb 2008 13:29:28 -0500Looks like a corrupted/uninitialized type (method table contains a NIL pointer...). Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 25, 2008, at 1:15 PM, Jay wrote: any obvious ideas? I'll have to pause for a bit again..Jay at jay-win1 /cygdrive/c/dev2/cm3.2/m3-ui/juno-2/juno-app/NT386GNU/Juno.exe****** runtime error:*** Thread client error: NIL apply method passed to Thread.Fork!*** file "ThreadWin32.m3", line 578*** 14837 [sig] Juno 3908 c:\dev2\cm3.2\m3-ui\juno-2\juno-app\NT386GNU\Juno.exe: *** fatal error - called with threadlist_ix -1Hangup - Jay _________________________________________________________________ Helping your favorite cause is as easy as instant messaging.?You IM, we give. http://im.live.com/Messenger/IM/Home/?source=text_hotmail_join -------------- next part -------------- An HTML attachment was scrubbed... URL: From rodney.bates at wichita.edu Sun Mar 9 01:19:32 2008 From: rodney.bates at wichita.edu (Rodney M. Bates) Date: Sat, 08 Mar 2008 18:19:32 -0600 Subject: [M3devel] Assertions (Was FW: thread.fork getting nil on nt386gnu?) In-Reply-To: References: <20080224134642.48e7fowms8ooowo8@mail.elegosoft.com> <20080224143053.fdtxd48fkskggws0@mail.elegosoft.com> <20080224153541.59c105ro8e88480g@mail.elegosoft.com> <6346DE25-6495-49D0-A990-F48B989218AC@cs.purdue.edu> <5F9985B2-99AE-4C63-8624-3A33F9BD0569@cs.purdue.edu> <4CF63A9C-93DD-486D-825C-A07630AD13D6@cs.purdue.edu> Message-ID: <47D32D14.8020501@wichita.edu> Jay wrote: > I find there are bugs /everywhere/. Code that is trying to find bugs is > no more immune from bugs than the code it is attempting to check. More > code leads to more bugs. More asserts lead to more incorrect asserts. > More runtime checks lead to more incorrect runtime checks. etc. Of > course, checks are still good, and like everything else they must be > written carefully and like everything else, when things go apparently > wrong, pretty much all code should be assumed wrong until proven > otherwise (read it, run it, read what it depends on, etc.). Code is > guilty until proven innocent, many times. One rotten apple spoils the > bunch.... > > Well, here is my documented experience. For more than 20 years, I have kept detailed logs of every bug in most programming projects of significant size. This is from a project in Modula-3 that I have worked on and off on for years, with around 105K SLOC, From about 15% of the work for which I kept compile-time error data, 75% of all programming errors were found at compile time. This is a count of required changes to the code, not of compile time messages, which, of course, usually greatly outnumber the number of actual errors. Where an equivalent change needed to be made systematically in many places, I count this as one change. No doubt records from earlier in the project would show a higher percentage of compile-time errors. I have kept logs for all of the errors discovered at runtime. From about 40% of the more recent of these logs that are already classified and tallied, the breakdown of how the errors manifested is; Type of error % of all % of errors found errors at runtime Compile time 75 % N/A Failures of correct assertions 10 % 40 % Runtime errors 6.5 % 26 % Observed bad output/behavior 6.5 % 26 % Runaway loops/recursion 1.03 % 4.10 % Bad internal values, observed in a .42 % 1.68 % debugger or debug output Overzealous assertions .37 % 1.47 % Uncaught exceptions .34 % 1.36 % Synchronization/deadlock, etc. .11 % .42 % By "overzealous assertions", I mean assertions that assert something that they should not, and thus crash the program unnecessarily. Instances of assertions finding real bugs outnumber overzealous assertions more than 25 to one. A very good tradeoff, I think. The valid assertion failures were more numerous earlier in the project. Finding bugs by assertion failure also has the big advantage that it does not require verifying correct output/behavior, or even deciding manually what is correct. Of course, such results depend on the programmer's style of deciding what to assert. The count of assertions coded in this program is 652. YMMV. -- ------------------------------------------------------------- Rodney M. Bates, retired assistant professor Dept. of Computer Science, Wichita State University Wichita, KS 67260-0083 316-978-3922 rodney.bates at wichita.edu From wagner at elegosoft.com Sun Mar 9 20:11:33 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Sun, 09 Mar 2008 20:11:33 +0100 Subject: [M3devel] RuntimeException catching broken? Re: assertion failure in RTCollector In-Reply-To: <81E268D8-B32A-45F0-A14D-61D7B833F4B2@cs.purdue.edu> References: <20080220170733.e5h5zpdlkw80s4ks@mail.elegosoft.com> <20080220185655.ll7dysf07swo4g04@mail.elegosoft.com> <3C31A48B-E365-41D3-956D-9069BDC028D9@cs.purdue.edu> <20080309120433.0vi02calq8sw0kc0@mail.elegosoft.com> <2501041B-CB64-40B3-8FCE-6485EB52C10F@cs.purdue.edu> <20080309190636.kl4fvqcqm84cgw8c@mail.elegosoft.com> <81E268D8-B32A-45F0-A14D-61D7B833F4B2@cs.purdue.edu> Message-ID: <20080309201133.ls6r99ads0c4sk0w@mail.elegosoft.com> Now for somethings completely different ;-) I've written this simple test program to verify Dragisa's report about not-working runtime exceptions: MODULE Main; IMPORT RuntimeError, IO, Process; CONST Tag = RuntimeError.Tag; PROCEDURE CatchAssert() = BEGIN TRY <*ASSERT FALSE*> EXCEPT RuntimeError.E( t ) => IO.Put( "OK: caught RuntimeError.Assert: " & Tag( t ) & "\n" ); END; END CatchAssert; PROCEDURE TestAll() = BEGIN CatchAssert(); END TestAll; BEGIN TRY TestAll(); EXCEPT ELSE IO.Put( "ERROR: caught unexpected exception\n" ); Process.Exit( 1 ); END; END Main. As he has reported, the exception is not caught. I've tried the debugger and set breakpoints at RTHooks.Raise() and RTHooks.AssertFailed(), but it did never stop there. Any ideas what's wrong there? You probably know better than me how exceptions are implemented. For today, I think I'll add that program as p208 to m3tests and then turn my attention to dinner :-) Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From wagner at elegosoft.com Mon Mar 10 00:33:05 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Mon, 10 Mar 2008 00:33:05 +0100 Subject: [M3devel] What is this??? How's it possible? In-Reply-To: <1204830152.8782.39.camel@hias.m3w.org> References: <1204818659.8782.16.camel@hias.m3w.org> <4AB9C403-1CF8-4621-8E90-9B5BB1F8B4B4@cs.purdue.edu> <20080306182319.u2flyt1uskg0owkg@mail.elegosoft.com> <1204830152.8782.39.camel@hias.m3w.org> Message-ID: <20080310003305.40ubkn1ke8s80scw@mail.elegosoft.com> Quoting Dragi?a Duri? : > Tried: > > TRY > RTLinker.AddUnit(LOOPHOLE(dl.sym(NARROW(r.get(i-1), TEXT) > & "_M3").addr, RT0.Binder)); > EXCEPT RuntimeError.E(expl) => > <* DEBUG "Error adding " & r.get(i-1) *> > END; > > It still fails with: > > *** > *** runtime error: > *** A compile-time type is missing. > *** file "../src/runtime/common/RTLinker.i3" > *** > > And RTType.Fail procedure, only one getting RuntimeError.MisingType > as argument, looks like this: > > PROCEDURE Fail (rte: RTE; m: RT0.ModulePtr; x, y: ADDRESS) = > <*FATAL ANY*> > ... > RTException.Raise (a); > END Fail; > > But, I am getting above mentioned error. Please retry with the latest m3core. I accidentally broke exception handling some weeks ago :-/ Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From wagner at elegosoft.com Mon Mar 10 00:38:29 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Mon, 10 Mar 2008 00:38:29 +0100 Subject: [M3devel] RuntimeException catching broken? Re: assertion failure in RTCollector In-Reply-To: <20080309201133.ls6r99ads0c4sk0w@mail.elegosoft.com> References: <20080220170733.e5h5zpdlkw80s4ks@mail.elegosoft.com> <20080220185655.ll7dysf07swo4g04@mail.elegosoft.com> <3C31A48B-E365-41D3-956D-9069BDC028D9@cs.purdue.edu> <20080309120433.0vi02calq8sw0kc0@mail.elegosoft.com> <2501041B-CB64-40B3-8FCE-6485EB52C10F@cs.purdue.edu> <20080309190636.kl4fvqcqm84cgw8c@mail.elegosoft.com> <81E268D8-B32A-45F0-A14D-61D7B833F4B2@cs.purdue.edu> <20080309201133.ls6r99ads0c4sk0w@mail.elegosoft.com> Message-ID: <20080310003829.ra46lb6k0soskwwk@mail.elegosoft.com> Please forget my last message. It seems I accidentally broke this myself some weeks ago while working on FreeBSD PTHREADS and committing too much (changes that were only temporary work-arounds). The test program should work again now. Quoting Olaf Wagner : > Now for somethings completely different ;-) > > I've written this simple test program to verify Dragisa's report > about not-working runtime exceptions: > > MODULE Main; > > IMPORT RuntimeError, IO, Process; > > CONST Tag = RuntimeError.Tag; > > PROCEDURE CatchAssert() = > BEGIN > TRY > <*ASSERT FALSE*> > EXCEPT > RuntimeError.E( t ) => > IO.Put( "OK: caught RuntimeError.Assert: " & Tag( t ) & "\n" ); > END; > END CatchAssert; > > PROCEDURE TestAll() = > BEGIN > CatchAssert(); > END TestAll; > > BEGIN > TRY > TestAll(); > EXCEPT > ELSE > IO.Put( "ERROR: caught unexpected exception\n" ); > Process.Exit( 1 ); > END; > END Main. > > As he has reported, the exception is not caught. I've tried the debugger > and set breakpoints at RTHooks.Raise() and RTHooks.AssertFailed(), but > it did never stop there. > > Any ideas what's wrong there? You probably know better than me how > exceptions are implemented. > > For today, I think I'll add that program as p208 to m3tests and then > turn my attention to dinner :-) > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From rcoleburn at scires.com Sat Mar 1 05:10:40 2008 From: rcoleburn at scires.com (Randy Coleburn) Date: Fri, 29 Feb 2008 23:10:40 -0500 Subject: [M3devel] Is the climate right for a new promotion? In-Reply-To: <47C45BB4.40009@wichita.edu> References: <47C45BB4.40009@wichita.edu> Message-ID: <47C890ED.1E75.00D7.1@scires.com> I agree! --Randy >>> "Rodney M. Bates" 2/26/2008 1:34 PM >>> I have been thinking lately that the climate might be right for a new attempt to promote Modula 3. I see two major trends and a small one that suggest this. Fist, probably nearly once week on the average, I see new articles that say that software developers are not prepared to utilize the now- proliferating multi-core processors. This actually greatly surprises me, because I had lots of exposure, decades ago, to all the concurrent programming stuff--threads, synchronization, scheduling, etc. But apparently, most programmers have been doing strictly sequential programming, and I have to admit, I myself only use concurrent programming techniques a minority of the time. In Modula-3, we have a language that already has better support than most, and certainly better than all the popular languages. Moreover, the implementation is out if front too, with Tony's recent thread and GC work for multi-processors. There are also a few articles out there that say, in effect, it is hopeless to implement threads and synchronization correctly as library code on top of existing C/C++. Second, the expansion of the internet and its vulnerability to automated attacks from anywhere in the world has greatly upped the ante on correctness of software. Before, it was enough if a program handled all the realistic cases that a reasonable user would present it with. Now, the whole theoretical space of inputs has to be handled with at least some level of correctness to avoid vulnerabilities. 1000-byte passwords are the standard example. All this means the limits of testing as a means of finding the bugs are a far greater obstacle than they once were. This makes the time right for another push toward static languages, which were somewhat of a popular flop the first time. Third, and perhaps rather more narrow, the developers of safety- critical software are reexamining the value of testing in the light of various close-calls, for example in airplanes. They are starting to look again at static methods. We have a language which, though little-known, is already designed and implemented and has a lot of convincing history. It has a lot of things that can be promoted. Aside from good thread support and the best safety properties around, it is much simpler than almost anything, yet more powerful than almost anything. A serious push would probably need the support of some large company or organization. But I think the climate is better than it has been in decades, and the technical arguments are very strong. -- ------------------------------------------------------------- Rodney M. Bates, retired assistant professor Dept. of Computer Science, Wichita State University Wichita, KS 67260-0083 316-978-3922 rodney.bates at wichita.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Sat Mar 8 04:32:49 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 7 Mar 2008 22:32:49 -0500 Subject: [M3devel] A start on Apple XCode 3.0 support for Modula-3 Message-ID: These go in ~/Library/Application Support/Developer/Shared/Xcode/ Specifications to get syntax highlighting and completion. No support for a lexer class to do proper indenting, etc., but at least this is a starting point. Building also still needs to be done with a Custom Build Tool Target set up to invoke cm3 in the package root directory. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: m3makefile.xclangspec Type: application/octet-stream Size: 3779 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: modula3.pbfilespec Type: application/octet-stream Size: 694 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: modula3.xclangspec Type: application/octet-stream Size: 4529 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 11 00:31:14 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 10 Mar 2008 19:31:14 -0400 Subject: [M3devel] FW: hand.c In-Reply-To: References: <995081.11151.qm@web27115.mail.ukl.yahoo.com> Message-ID: <25CBA31B-515C-480A-B51A-C4BDFED646EC@cs.purdue.edu> OK, weird. Now I'm getting e-mail from 6 months ago. Talk about deja vu. Any ideas what's going on? Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Sep 15, 2007, at 10:39 PM, Jay wrote: > I have since made some more of the changes. > > > > From: jayk123 at hotmail.com > To: m3devel at elegosoft.com > Subject: hand.c > Date: Mon, 3 Sep 2007 23:15:04 +0000 > > I'm not seeing commit mail. > Maybe my fault with the hotmail? > > I churned it relatively a lot. It's a small file. > It compiles with older Microsoft C compilers -- optionally using > __int64 instead of long long. (Visual C+ 8.0 worked as it was.) > I would have factored out the LL suffixes, but, really, the vast > majoriy, if not all, of the L and LL suffixes were unnecessary, right? > I removed most of them. > It is valid C++ now -- optional prototypes. > There was a race condition in initialization, fixed. > gcc -Wall complained about a safe signed/unsigned compare, fixed. > > Not changed: > all these longs should be unsigned long, right? > Or at least the bits, if not the input bit indices? > Negative bit indices should be either rejected or converted to 0, > and then treated as unsigned from there, right? > > _lowbits and _highbits should be > extern const ? > only on x86, right? I'm still checking if they are referenced > for all targets or only x86 > > HighBits and LowBits should be, like..ok switched to > #if LONG_MAX == 2147483647 > extern const .. hard coded 32 bit values ... > #elif LONG_MAX == 9223372036854775807 > extern const .. hard coded 64 bit values ... > #else > either compute them like currently or #error > #endif > > right? > > And if it is 32 bits, share the values of HighBits with _highbits, > LowBits with _lowbits? > They aren't /quite/ the same, but almost. _lowbits and _highbits > have one extra value, the zeros at the ends I think it was. > > - Jay > > Get news, entertainment and everything you care about at Live.com. > Check it out! > > Kick back and relax with hot games and cool activities at the > Messenger Caf?. Play now! -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Tue Mar 11 02:44:28 2008 From: jayk123 at hotmail.com (Jay) Date: Tue, 11 Mar 2008 01:44:28 +0000 Subject: [M3devel] FW: hand.c In-Reply-To: <25CBA31B-515C-480A-B51A-C4BDFED646EC@cs.purdue.edu> References: <995081.11151.qm@web27115.mail.ukl.yahoo.com> <25CBA31B-515C-480A-B51A-C4BDFED646EC@cs.purdue.edu> Message-ID: Invitation for an infinite stream of: "me too" also messages about "awaiting moderator approval". - Jay From: hosking at cs.purdue.eduTo: m3devel at elegosoft.comDate: Mon, 10 Mar 2008 19:31:14 -0400Subject: Re: [M3devel] FW: hand.c OK, weird. Now I'm getting e-mail from 6 months ago. Talk about deja vu. Any ideas what's going on? Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Sep 15, 2007, at 10:39 PM, Jay wrote: I have since made some more of the changes. From: jayk123 at hotmail.comTo: m3devel at elegosoft.comSubject: hand.cDate: Mon, 3 Sep 2007 23:15:04 +0000I'm not seeing commit mail.Maybe my fault with the hotmail? I churned it relatively a lot. It's a small file.It compiles with older Microsoft C compilers -- optionally using __int64 instead of long long. (Visual C+ 8.0 worked as it was.) I would have factored out the LL suffixes, but, really, the vast majoriy, if not all, of the L and LL suffixes were unnecessary, right? I removed most of them.It is valid C++ now -- optional prototypes.There was a race condition in initialization, fixed.gcc -Wall complained about a safe signed/unsigned compare, fixed. Not changed: all these longs should be unsigned long, right? Or at least the bits, if not the input bit indices? Negative bit indices should be either rejected or converted to 0, and then treated as unsigned from there, right? _lowbits and _highbits should be extern const ? only on x86, right? I'm still checking if they are referenced for all targets or only x86 HighBits and LowBits should be, like..ok switched to #if LONG_MAX == 2147483647 extern const .. hard coded 32 bit values ... #elif LONG_MAX == 9223372036854775807 extern const .. hard coded 64 bit values ... #else either compute them like currently or #error #endif right? And if it is 32 bits, share the values of HighBits with _highbits, LowBits with _lowbits?They aren't /quite/ the same, but almost. _lowbits and _highbits have one extra value, the zeros at the ends I think it was. - Jay Get news, entertainment and everything you care about at Live.com. Check it out! Kick back and relax with hot games and cool activities at the Messenger Caf?. Play now! _________________________________________________________________ Climb to the top of the charts!?Play the word scramble challenge with star power. http://club.live.com/star_shuffle.aspx?icid=starshuffle_wlmailtextlink_jan -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Tue Mar 11 09:03:09 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Tue, 11 Mar 2008 09:03:09 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead Message-ID: <1205222589.32152.27.camel@hias.m3w.org> I have single linked list that I use to send messages to many threads. It's end is locked for addition, and current end is given to each new client connecting. This way, all client are going towards end of the list so it's head becomes unreferenced and goes away by GC. Except it does not. I've added WeakRef cleanup and sequential id's so I can record maximum freed id for checking. No single cleanup happens. I'll cross check my code and count my references from application side, but maybe someone else has similar problems? dd -- Dragi?a Duri? From wagner at elegosoft.com Tue Mar 11 11:31:03 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 11 Mar 2008 11:31:03 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <1205222589.32152.27.camel@hias.m3w.org> References: <1205222589.32152.27.camel@hias.m3w.org> Message-ID: <20080311113103.as92wmzolcws0csw@mail.elegosoft.com> Quoting Dragi?a Duri? : > I have single linked list that I use to send messages to many threads. > It's end is locked for addition, and current end is given to each new > client connecting. This way, all client are going towards end of the > list so it's head becomes unreferenced and goes away by GC. > > Except it does not. > > I've added WeakRef cleanup and sequential id's so I can record maximum > freed id for checking. No single cleanup happens. > > I'll cross check my code and count my references from application side, > but maybe someone else has similar problems? I'm not aware of any recent changes in this area. I could only assume that the behaviour of the garbage collector may have changed, as Antony Hosking has done considerable work on that, but I haven't noticed any general problems. (I think there's still a problem with generational collection and FreeBSD PTHREADs, but you're not using that, are you?) Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From hosking at cs.purdue.edu Tue Mar 11 14:50:27 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 11 Mar 2008 09:50:27 -0400 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <1205222589.32152.27.camel@hias.m3w.org> References: <1205222589.32152.27.camel@hias.m3w.org> Message-ID: Sounds like we need some sort of testcase. Would you be able to devise one? It will be hard to make it deterministic, but at least we should see a non-zero cleanup count. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: > I have single linked list that I use to send messages to many threads. > It's end is locked for addition, and current end is given to each new > client connecting. This way, all client are going towards end of the > list so it's head becomes unreferenced and goes away by GC. > > Except it does not. > > I've added WeakRef cleanup and sequential id's so I can record maximum > freed id for checking. No single cleanup happens. > > I'll cross check my code and count my references from application > side, > but maybe someone else has similar problems? > > dd > -- > Dragi?a Duri? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronny.forberger at elegosoft.com Tue Mar 11 16:49:03 2008 From: ronny.forberger at elegosoft.com (Ronny Forberger) Date: Tue, 11 Mar 2008 16:49:03 +0100 Subject: [M3devel] FW: hand.c In-Reply-To: <25CBA31B-515C-480A-B51A-C4BDFED646EC@cs.purdue.edu> References: <995081.11151.qm@web27115.mail.ukl.yahoo.com> <25CBA31B-515C-480A-B51A-C4BDFED646EC@cs.purdue.edu> Message-ID: <20080311164903.25piw31hus80s4sk@mail.elegosoft.com> The mailling list software had unfortunately held back these mails, as they could not be processed for some reason like not being able to contact the spam analyzer when sometimes that one gets restarted. It did not make any retry, so I had to send them out manually. :( Ronny -- Message from: Tony Hosking Date: Di 11 M?r 2008 00:31:14 CET Subject: Re: [M3devel] FW: hand.c > OK, weird. Now I'm getting e-mail from 6 months ago. Talk about deja > vu. Any ideas what's going on? > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > On Sep 15, 2007, at 10:39 PM, Jay wrote: > >> I have since made some more of the changes. >> >> >> >> From: jayk123 at hotmail.com >> To: m3devel at elegosoft.com >> Subject: hand.c >> Date: Mon, 3 Sep 2007 23:15:04 +0000 >> >> I'm not seeing commit mail. >> Maybe my fault with the hotmail? >> >> I churned it relatively a lot. It's a small file. >> It compiles with older Microsoft C compilers -- optionally using >> __int64 instead of long long. (Visual C+ 8.0 worked as it was.) >> I would have factored out the LL suffixes, but, really, the vast >> majoriy, if not all, of the L and LL suffixes were unnecessary, >> right? >> I removed most of them. >> It is valid C++ now -- optional prototypes. >> There was a race condition in initialization, fixed. >> gcc -Wall complained about a safe signed/unsigned compare, fixed. >> >> Not changed: >> all these longs should be unsigned long, right? >> Or at least the bits, if not the input bit indices? >> Negative bit indices should be either rejected or converted to 0, >> and then treated as unsigned from there, right? >> >> _lowbits and _highbits should be >> extern const ? >> only on x86, right? I'm still checking if they are referenced >> for all targets or only x86 >> >> HighBits and LowBits should be, like..ok switched to >> #if LONG_MAX == 2147483647 >> extern const .. hard coded 32 bit values ... >> #elif LONG_MAX == 9223372036854775807 >> extern const .. hard coded 64 bit values ... >> #else >> either compute them like currently or #error >> #endif >> >> right? >> >> And if it is 32 bits, share the values of HighBits with _highbits, >> LowBits with _lowbits? >> They aren't /quite/ the same, but almost. _lowbits and _highbits >> have one extra value, the zeros at the ends I think it was. >> >> - Jay >> >> Get news, entertainment and everything you care about at Live.com. >> Check it out! >> >> Kick back and relax with hot games and cool activities at the >> Messenger Caf?. Play now! -- Ronny Forberger Systemadministration & IT-Support elego Software Solutions GmbH Gustav-Meyer-Allee 25 Geb?ude 12, Raum 227 D-13355 Berlin Tel. +49 30 23 45 86 96 ronny.forberger at elegosoft.com Fax +49 30 23 45 86 95 http://www.elegosoft.com Gesch?ftsf?hrer: Olaf Wagner, Sitz Berlin Amtsgericht Berlin-Charlottenburg, HRB 77719, USt-IdNr: DE163214194 From dragisha at m3w.org Wed Mar 12 09:21:38 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Wed, 12 Mar 2008 09:21:38 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: References: <1205222589.32152.27.camel@hias.m3w.org> Message-ID: <1205310098.32152.43.camel@hias.m3w.org> Problem in my case can be thoroughness of GC... Because it's linked list of WeakRef-ed objects, it's always first element in list to become unreachable first - all others are reachable at least through previous (in list) WeakRef-ed object. So, behaviour I have observed can be because of some thousands of freeable objects, that one is missed too often. I have other object being cleaned and counted... There also cleanup invocation looks like it's lagging too much. I'll come with numbers after I have more tests. dd On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking wrote: > Sounds like we need some sort of testcase. Would you be able to > devise one? It will be hard to make it deterministic, but at least we > should see a non-zero cleanup count. > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: > > > I have single linked list that I use to send messages to many > > threads. > > It's end is locked for addition, and current end is given to each > > new > > client connecting. This way, all client are going towards end of the > > list so it's head becomes unreferenced and goes away by GC. > > > > Except it does not. > > > > I've added WeakRef cleanup and sequential id's so I can record > > maximum > > freed id for checking. No single cleanup happens. > > > > I'll cross check my code and count my references from application > > side, > > but maybe someone else has similar problems? > > > > dd > > -- > > Dragi?a Duri? > > > -- Dragi?a Duri? From hosking at cs.purdue.edu Wed Mar 12 16:12:57 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 12 Mar 2008 11:12:57 -0400 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <1205310098.32152.43.camel@hias.m3w.org> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> Message-ID: <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> This is probably a result of conservatism for references from thread stacks which results in spurious retention. It may be necessary to be more careful about what references are held on the stack in the threads implementation (and elsewhere in your code). Until we diagnose the place where objects are being retained it will be hard to pinpoint. You should realize that weak refs are a problem in many language implementations, not just Modula-3. I look forward to hearing more definitively from you. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: > Problem in my case can be thoroughness of GC... Because it's linked > list of WeakRef-ed objects, it's always first element in list to > become > unreachable first - all others are reachable at least through previous > (in list) WeakRef-ed object. So, behaviour I have observed can be > because of some thousands of freeable objects, that one is missed too > often. > > I have other object being cleaned and counted... There also cleanup > invocation looks like it's lagging too much. I'll come with numbers > after I have more tests. > > dd > > On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking wrote: >> Sounds like we need some sort of testcase. Would you be able to >> devise one? It will be hard to make it deterministic, but at least >> we >> should see a non-zero cleanup count. >> >> Antony Hosking | Associate Professor | Computer Science | Purdue >> University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> >> >> On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: >> >>> I have single linked list that I use to send messages to many >>> threads. >>> It's end is locked for addition, and current end is given to each >>> new >>> client connecting. This way, all client are going towards end of the >>> list so it's head becomes unreferenced and goes away by GC. >>> >>> Except it does not. >>> >>> I've added WeakRef cleanup and sequential id's so I can record >>> maximum >>> freed id for checking. No single cleanup happens. >>> >>> I'll cross check my code and count my references from application >>> side, >>> but maybe someone else has similar problems? >>> >>> dd >>> -- >>> Dragi?a Duri? >>> >> > -- > Dragi?a Duri? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Thu Mar 13 11:16:36 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 13 Mar 2008 11:16:36 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> Message-ID: <1205403396.32152.96.camel@hias.m3w.org> What happens to thread stacks once threads are Join-ed? dd On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking wrote: > This is probably a result of conservatism for references from thread > stacks which results in spurious retention. It may be necessary to be > more careful about what references are held on the stack in the > threads implementation (and elsewhere in your code). Until we > diagnose the place where objects are being retained it will be hard to > pinpoint. You should realize that weak refs are a problem in many > language implementations, not just Modula-3. I look forward to > hearing more definitively from you. > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: > > > Problem in my case can be thoroughness of GC... Because it's linked > > list of WeakRef-ed objects, it's always first element in list to > > become > > unreachable first - all others are reachable at least through > > previous > > (in list) WeakRef-ed object. So, behaviour I have observed can be > > because of some thousands of freeable objects, that one is missed > > too > > often. > > > > I have other object being cleaned and counted... There also cleanup > > invocation looks like it's lagging too much. I'll come with numbers > > after I have more tests. > > > > dd > > > > On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking wrote: > > > Sounds like we need some sort of testcase. Would you be able to > > > devise one? It will be hard to make it deterministic, but at > > > least we > > > should see a non-zero cleanup count. > > > > > > Antony Hosking | Associate Professor | Computer Science | Purdue > > > University > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: > > > > > > > I have single linked list that I use to send messages to many > > > > threads. > > > > It's end is locked for addition, and current end is given to > > > > each > > > > new > > > > client connecting. This way, all client are going towards end of > > > > the > > > > list so it's head becomes unreferenced and goes away by GC. > > > > > > > > Except it does not. > > > > > > > > I've added WeakRef cleanup and sequential id's so I can record > > > > maximum > > > > freed id for checking. No single cleanup happens. > > > > > > > > I'll cross check my code and count my references from > > > > application > > > > side, > > > > but maybe someone else has similar problems? > > > > > > > > dd > > > > -- > > > > Dragi?a Duri? > > > > > > > > > -- > > Dragi?a Duri? > > > > -- Dragi?a Duri? From dragisha at m3w.org Thu Mar 13 13:41:40 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 13 Mar 2008 13:41:40 +0100 Subject: [M3devel] argh Message-ID: <1205412100.32152.100.camel@hias.m3w.org> I've added Convert.FromLongInt to both Convert.i3 and Convert.m3, shipped m3core to my current install, build cm3-min and started making system, and it always fails here. Hints? dd === package m3-sys/cm3 === +++ cm3 -build -override -DROOT='/usr/src/redhat/BUILD/cm3-base-5.6.0/cm3' -DCM3_VERSION_TEXT='d5.6.0' -DCM3_VERSION_NUMBER='050600' -DCM3_LAST_CHANGED='2008-01-31' +++ --- building in LINUXLIBC6 --- new source -> compiling Version.m3 new "Version.mo" -> linking cm3 Fatal Error: incomplete program Convert.FromLongInt: is exported, but not implemented: Convert.i3 *** execution of cm3 -build -override -DROOT='/usr/src/redhat/BUILD/cm3-base-5.6.0/cm3' -DCM3_VERSION_TEXT='d5.6.0' -DCM3_VERSION_NUMBER='050600' -DCM3_LAST_CHANGED='2008-01-31' failed *** -- Dragi?a Duri? From dragisha at m3w.org Thu Mar 13 14:21:59 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 13 Mar 2008 14:21:59 +0100 Subject: [M3devel] argh In-Reply-To: <1205412100.32152.100.camel@hias.m3w.org> References: <1205412100.32152.100.camel@hias.m3w.org> Message-ID: <1205414519.32152.102.camel@hias.m3w.org> Of course... disregard this :) Anyone interested in RPM building? I can submit a patch for it, nothing spectacular but works nicely. dd On Thu, 2008-03-13 at 13:41 +0100, Dragi?a Duri? wrote: > I've added Convert.FromLongInt to both Convert.i3 and Convert.m3, > shipped m3core to my current install, build cm3-min and started making > system, and it always fails here. > > Hints? > > dd > > === package m3-sys/cm3 === > +++ cm3 -build -override > -DROOT='/usr/src/redhat/BUILD/cm3-base-5.6.0/cm3' > -DCM3_VERSION_TEXT='d5.6.0' -DCM3_VERSION_NUMBER='050600' > -DCM3_LAST_CHANGED='2008-01-31' +++ > --- building in LINUXLIBC6 --- > > new source -> compiling Version.m3 > new "Version.mo" -> linking cm3 > > Fatal Error: incomplete program > > Convert.FromLongInt: is exported, but not implemented: Convert.i3 > *** execution of cm3 -build -override > -DROOT='/usr/src/redhat/BUILD/cm3-base-5.6.0/cm3' > -DCM3_VERSION_TEXT='d5.6.0' -DCM3_VERSION_NUMBER='050600' > -DCM3_LAST_CHANGED='2008-01-31' failed *** > -- Dragi?a Duri? From hosking at cs.purdue.edu Thu Mar 13 14:46:27 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 13 Mar 2008 09:46:27 -0400 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <1205403396.32152.96.camel@hias.m3w.org> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> Message-ID: <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> They should get unlinked from the global ring. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: > What happens to thread stacks once threads are Join-ed? > > dd > > On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking wrote: >> This is probably a result of conservatism for references from thread >> stacks which results in spurious retention. It may be necessary to >> be >> more careful about what references are held on the stack in the >> threads implementation (and elsewhere in your code). Until we >> diagnose the place where objects are being retained it will be hard >> to >> pinpoint. You should realize that weak refs are a problem in many >> language implementations, not just Modula-3. I look forward to >> hearing more definitively from you. >> >> Antony Hosking | Associate Professor | Computer Science | Purdue >> University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> >> >> On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: >> >>> Problem in my case can be thoroughness of GC... Because it's linked >>> list of WeakRef-ed objects, it's always first element in list to >>> become >>> unreachable first - all others are reachable at least through >>> previous >>> (in list) WeakRef-ed object. So, behaviour I have observed can be >>> because of some thousands of freeable objects, that one is missed >>> too >>> often. >>> >>> I have other object being cleaned and counted... There also cleanup >>> invocation looks like it's lagging too much. I'll come with numbers >>> after I have more tests. >>> >>> dd >>> >>> On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking wrote: >>>> Sounds like we need some sort of testcase. Would you be able to >>>> devise one? It will be hard to make it deterministic, but at >>>> least we >>>> should see a non-zero cleanup count. >>>> >>>> Antony Hosking | Associate Professor | Computer Science | Purdue >>>> University >>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>> >>>> >>>> >>>> >>>> >>>> On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: >>>> >>>>> I have single linked list that I use to send messages to many >>>>> threads. >>>>> It's end is locked for addition, and current end is given to >>>>> each >>>>> new >>>>> client connecting. This way, all client are going towards end of >>>>> the >>>>> list so it's head becomes unreferenced and goes away by GC. >>>>> >>>>> Except it does not. >>>>> >>>>> I've added WeakRef cleanup and sequential id's so I can record >>>>> maximum >>>>> freed id for checking. No single cleanup happens. >>>>> >>>>> I'll cross check my code and count my references from >>>>> application >>>>> side, >>>>> but maybe someone else has similar problems? >>>>> >>>>> dd >>>>> -- >>>>> Dragi?a Duri? >>>>> >>>> >>> -- >>> Dragi?a Duri? >>> >> >> > -- > Dragi?a Duri? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Thu Mar 13 14:56:28 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 13 Mar 2008 14:56:28 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> Message-ID: <1205416588.32152.127.camel@hias.m3w.org> And not only that, their stackbase is also set to NIL, meaning their stack is not regarded in any way during future collections - meaning all local variables are forgotten once apply method returns. That way, "spuriousness" is not an issue, once thread returns? dd On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: > They should get unlinked from the global ring. > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: > > > What happens to thread stacks once threads are Join-ed? > > > > dd > > > > On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking wrote: > > > This is probably a result of conservatism for references from > > > thread > > > stacks which results in spurious retention. It may be necessary > > > to be > > > more careful about what references are held on the stack in the > > > threads implementation (and elsewhere in your code). Until we > > > diagnose the place where objects are being retained it will be > > > hard to > > > pinpoint. You should realize that weak refs are a problem in many > > > language implementations, not just Modula-3. I look forward to > > > hearing more definitively from you. > > > > > > Antony Hosking | Associate Professor | Computer Science | Purdue > > > University > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: > > > > > > > Problem in my case can be thoroughness of GC... Because it's > > > > linked > > > > list of WeakRef-ed objects, it's always first element in list to > > > > become > > > > unreachable first - all others are reachable at least through > > > > previous > > > > (in list) WeakRef-ed object. So, behaviour I have observed can > > > > be > > > > because of some thousands of freeable objects, that one is > > > > missed > > > > too > > > > often. > > > > > > > > I have other object being cleaned and counted... There also > > > > cleanup > > > > invocation looks like it's lagging too much. I'll come with > > > > numbers > > > > after I have more tests. > > > > > > > > dd > > > > > > > > On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking wrote: > > > > > Sounds like we need some sort of testcase. Would you be able > > > > > to > > > > > devise one? It will be hard to make it deterministic, but at > > > > > least we > > > > > should see a non-zero cleanup count. > > > > > > > > > > Antony Hosking | Associate Professor | Computer Science | > > > > > Purdue > > > > > University > > > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: > > > > > > > > > > > I have single linked list that I use to send messages to > > > > > > many > > > > > > threads. > > > > > > It's end is locked for addition, and current end is given to > > > > > > each > > > > > > new > > > > > > client connecting. This way, all client are going towards > > > > > > end of > > > > > > the > > > > > > list so it's head becomes unreferenced and goes away by GC. > > > > > > > > > > > > Except it does not. > > > > > > > > > > > > I've added WeakRef cleanup and sequential id's so I can > > > > > > record > > > > > > maximum > > > > > > freed id for checking. No single cleanup happens. > > > > > > > > > > > > I'll cross check my code and count my references from > > > > > > application > > > > > > side, > > > > > > but maybe someone else has similar problems? > > > > > > > > > > > > dd > > > > > > -- > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > -- > > > > Dragi?a Duri? > > > > > > > > > > > > -- > > Dragi?a Duri? > > > > -- Dragi?a Duri? From hosking at cs.purdue.edu Thu Mar 13 15:08:53 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 13 Mar 2008 10:08:53 -0400 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <1205416588.32152.127.camel@hias.m3w.org> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> Message-ID: Indeed! Can you devise a testcase? Note that the generational collector will retain old objects for some time until a major collection occurs, Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: > And not only that, their stackbase is also set to NIL, meaning their > stack is not regarded in any way during future collections - meaning > all > local variables are forgotten once apply method returns. > > That way, "spuriousness" is not an issue, once thread returns? > > dd > > On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: >> They should get unlinked from the global ring. >> >> Antony Hosking | Associate Professor | Computer Science | Purdue >> University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> >> >> On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: >> >>> What happens to thread stacks once threads are Join-ed? >>> >>> dd >>> >>> On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking wrote: >>>> This is probably a result of conservatism for references from >>>> thread >>>> stacks which results in spurious retention. It may be necessary >>>> to be >>>> more careful about what references are held on the stack in the >>>> threads implementation (and elsewhere in your code). Until we >>>> diagnose the place where objects are being retained it will be >>>> hard to >>>> pinpoint. You should realize that weak refs are a problem in many >>>> language implementations, not just Modula-3. I look forward to >>>> hearing more definitively from you. >>>> >>>> Antony Hosking | Associate Professor | Computer Science | Purdue >>>> University >>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>> >>>> >>>> >>>> >>>> >>>> On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: >>>> >>>>> Problem in my case can be thoroughness of GC... Because it's >>>>> linked >>>>> list of WeakRef-ed objects, it's always first element in list to >>>>> become >>>>> unreachable first - all others are reachable at least through >>>>> previous >>>>> (in list) WeakRef-ed object. So, behaviour I have observed can >>>>> be >>>>> because of some thousands of freeable objects, that one is >>>>> missed >>>>> too >>>>> often. >>>>> >>>>> I have other object being cleaned and counted... There also >>>>> cleanup >>>>> invocation looks like it's lagging too much. I'll come with >>>>> numbers >>>>> after I have more tests. >>>>> >>>>> dd >>>>> >>>>> On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking wrote: >>>>>> Sounds like we need some sort of testcase. Would you be able >>>>>> to >>>>>> devise one? It will be hard to make it deterministic, but at >>>>>> least we >>>>>> should see a non-zero cleanup count. >>>>>> >>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>> Purdue >>>>>> University >>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: >>>>>> >>>>>>> I have single linked list that I use to send messages to >>>>>>> many >>>>>>> threads. >>>>>>> It's end is locked for addition, and current end is given to >>>>>>> each >>>>>>> new >>>>>>> client connecting. This way, all client are going towards >>>>>>> end of >>>>>>> the >>>>>>> list so it's head becomes unreferenced and goes away by GC. >>>>>>> >>>>>>> Except it does not. >>>>>>> >>>>>>> I've added WeakRef cleanup and sequential id's so I can >>>>>>> record >>>>>>> maximum >>>>>>> freed id for checking. No single cleanup happens. >>>>>>> >>>>>>> I'll cross check my code and count my references from >>>>>>> application >>>>>>> side, >>>>>>> but maybe someone else has similar problems? >>>>>>> >>>>>>> dd >>>>>>> -- >>>>>>> Dragi?a Duri? >>>>>>> >>>>>> >>>>> -- >>>>> Dragi?a Duri? >>>>> >>>> >>>> >>> -- >>> Dragi?a Duri? >>> >> >> > -- > Dragi?a Duri? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Thu Mar 13 15:20:04 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 13 Mar 2008 15:20:04 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> Message-ID: <1205418004.32152.137.camel@hias.m3w.org> Garbage collector is one piece of runtime I am groking only now. First, I'll try to analyze whole weakref structure and flow and see where these numbers mismatch so much: [15:13:58] 96 users, 1 ops and 0 unresolved; minId = 38348; thrs = 102 [15:13:58] started = 1991; cleaned = 1579; nilled = 1889; joined = 1889 [15:13:58] Dangling thrs-a-b-n = 5 [15:13:58] Live queue end is 38396 (head = 38340), maxClean = 37821, cleaned = 35309 joined is number of finished threads, cleaned is number of timer their Client.T's weakref was cleaned. Same is head = 38340 = number of head elements from my message queue (all are unlinked) and cleaned = 35309 = number of elements RTHeapRep's FinalCleanup happened. This process' uptime is around 5 hours. Allocations are as evenly distributed as it's important to get picture. dd On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: > Indeed! Can you devise a testcase? Note that the generational > collector will retain old objects for some time until a major > collection occurs, > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: > > > And not only that, their stackbase is also set to NIL, meaning their > > stack is not regarded in any way during future collections - meaning > > all > > local variables are forgotten once apply method returns. > > > > That way, "spuriousness" is not an issue, once thread returns? > > > > dd > > > > On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: > > > They should get unlinked from the global ring. > > > > > > Antony Hosking | Associate Professor | Computer Science | Purdue > > > University > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: > > > > > > > What happens to thread stacks once threads are Join-ed? > > > > > > > > dd > > > > > > > > On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking wrote: > > > > > This is probably a result of conservatism for references from > > > > > thread > > > > > stacks which results in spurious retention. It may be > > > > > necessary > > > > > to be > > > > > more careful about what references are held on the stack in > > > > > the > > > > > threads implementation (and elsewhere in your code). Until we > > > > > diagnose the place where objects are being retained it will be > > > > > hard to > > > > > pinpoint. You should realize that weak refs are a problem in > > > > > many > > > > > language implementations, not just Modula-3. I look forward > > > > > to > > > > > hearing more definitively from you. > > > > > > > > > > Antony Hosking | Associate Professor | Computer Science | > > > > > Purdue > > > > > University > > > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: > > > > > > > > > > > Problem in my case can be thoroughness of GC... Because it's > > > > > > linked > > > > > > list of WeakRef-ed objects, it's always first element in > > > > > > list to > > > > > > become > > > > > > unreachable first - all others are reachable at least > > > > > > through > > > > > > previous > > > > > > (in list) WeakRef-ed object. So, behaviour I have observed > > > > > > can > > > > > > be > > > > > > because of some thousands of freeable objects, that one is > > > > > > missed > > > > > > too > > > > > > often. > > > > > > > > > > > > I have other object being cleaned and counted... There also > > > > > > cleanup > > > > > > invocation looks like it's lagging too much. I'll come with > > > > > > numbers > > > > > > after I have more tests. > > > > > > > > > > > > dd > > > > > > > > > > > > On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking wrote: > > > > > > > Sounds like we need some sort of testcase. Would you be > > > > > > > able > > > > > > > to > > > > > > > devise one? It will be hard to make it deterministic, but > > > > > > > at > > > > > > > least we > > > > > > > should see a non-zero cleanup count. > > > > > > > > > > > > > > Antony Hosking | Associate Professor | Computer Science | > > > > > > > Purdue > > > > > > > University > > > > > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: > > > > > > > > > > > > > > > I have single linked list that I use to send messages to > > > > > > > > many > > > > > > > > threads. > > > > > > > > It's end is locked for addition, and current end is > > > > > > > > given to > > > > > > > > each > > > > > > > > new > > > > > > > > client connecting. This way, all client are going > > > > > > > > towards > > > > > > > > end of > > > > > > > > the > > > > > > > > list so it's head becomes unreferenced and goes away by > > > > > > > > GC. > > > > > > > > > > > > > > > > Except it does not. > > > > > > > > > > > > > > > > I've added WeakRef cleanup and sequential id's so I can > > > > > > > > record > > > > > > > > maximum > > > > > > > > freed id for checking. No single cleanup happens. > > > > > > > > > > > > > > > > I'll cross check my code and count my references from > > > > > > > > application > > > > > > > > side, > > > > > > > > but maybe someone else has similar problems? > > > > > > > > > > > > > > > > dd > > > > > > > > -- > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > -- > > > > Dragi?a Duri? > > > > > > > > > > > > -- > > Dragi?a Duri? > > > -- Dragi?a Duri? From dragisha at m3w.org Thu Mar 13 15:22:17 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 13 Mar 2008 15:22:17 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> Message-ID: <1205418137.32152.139.camel@hias.m3w.org> Can I force "major collection" ? dd On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: > Indeed! Can you devise a testcase? Note that the generational > collector will retain old objects for some time until a major > collection occurs, > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: > > > And not only that, their stackbase is also set to NIL, meaning their > > stack is not regarded in any way during future collections - meaning > > all > > local variables are forgotten once apply method returns. > > > > That way, "spuriousness" is not an issue, once thread returns? > > > > dd > > > > On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: > > > They should get unlinked from the global ring. > > > > > > Antony Hosking | Associate Professor | Computer Science | Purdue > > > University > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: > > > > > > > What happens to thread stacks once threads are Join-ed? > > > > > > > > dd > > > > > > > > On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking wrote: > > > > > This is probably a result of conservatism for references from > > > > > thread > > > > > stacks which results in spurious retention. It may be > > > > > necessary > > > > > to be > > > > > more careful about what references are held on the stack in > > > > > the > > > > > threads implementation (and elsewhere in your code). Until we > > > > > diagnose the place where objects are being retained it will be > > > > > hard to > > > > > pinpoint. You should realize that weak refs are a problem in > > > > > many > > > > > language implementations, not just Modula-3. I look forward > > > > > to > > > > > hearing more definitively from you. > > > > > > > > > > Antony Hosking | Associate Professor | Computer Science | > > > > > Purdue > > > > > University > > > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: > > > > > > > > > > > Problem in my case can be thoroughness of GC... Because it's > > > > > > linked > > > > > > list of WeakRef-ed objects, it's always first element in > > > > > > list to > > > > > > become > > > > > > unreachable first - all others are reachable at least > > > > > > through > > > > > > previous > > > > > > (in list) WeakRef-ed object. So, behaviour I have observed > > > > > > can > > > > > > be > > > > > > because of some thousands of freeable objects, that one is > > > > > > missed > > > > > > too > > > > > > often. > > > > > > > > > > > > I have other object being cleaned and counted... There also > > > > > > cleanup > > > > > > invocation looks like it's lagging too much. I'll come with > > > > > > numbers > > > > > > after I have more tests. > > > > > > > > > > > > dd > > > > > > > > > > > > On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking wrote: > > > > > > > Sounds like we need some sort of testcase. Would you be > > > > > > > able > > > > > > > to > > > > > > > devise one? It will be hard to make it deterministic, but > > > > > > > at > > > > > > > least we > > > > > > > should see a non-zero cleanup count. > > > > > > > > > > > > > > Antony Hosking | Associate Professor | Computer Science | > > > > > > > Purdue > > > > > > > University > > > > > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: > > > > > > > > > > > > > > > I have single linked list that I use to send messages to > > > > > > > > many > > > > > > > > threads. > > > > > > > > It's end is locked for addition, and current end is > > > > > > > > given to > > > > > > > > each > > > > > > > > new > > > > > > > > client connecting. This way, all client are going > > > > > > > > towards > > > > > > > > end of > > > > > > > > the > > > > > > > > list so it's head becomes unreferenced and goes away by > > > > > > > > GC. > > > > > > > > > > > > > > > > Except it does not. > > > > > > > > > > > > > > > > I've added WeakRef cleanup and sequential id's so I can > > > > > > > > record > > > > > > > > maximum > > > > > > > > freed id for checking. No single cleanup happens. > > > > > > > > > > > > > > > > I'll cross check my code and count my references from > > > > > > > > application > > > > > > > > side, > > > > > > > > but maybe someone else has similar problems? > > > > > > > > > > > > > > > > dd > > > > > > > > -- > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > -- > > > > Dragi?a Duri? > > > > > > > > > > > > -- > > Dragi?a Duri? > > > -- Dragi?a Duri? From hosking at cs.purdue.edu Thu Mar 13 15:54:23 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 13 Mar 2008 10:54:23 -0400 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <1205418137.32152.139.camel@hias.m3w.org> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> Message-ID: <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> Yes. RTCollectorSRC.FinishCollection; RTCollectorSRC.StartCollection. This will finish the current (possibly minor) collection, and start a major collection. The major collection must finish before you can guarantee garbage has been freed, so you may want to RTCollectorSRC.FinishCollection. This sequence is packaged as RTCollector.Collect. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? wrote: > Can I force "major collection" ? > > dd > > On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: >> Indeed! Can you devise a testcase? Note that the generational >> collector will retain old objects for some time until a major >> collection occurs, >> >> Antony Hosking | Associate Professor | Computer Science | Purdue >> University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> >> >> On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: >> >>> And not only that, their stackbase is also set to NIL, meaning their >>> stack is not regarded in any way during future collections - meaning >>> all >>> local variables are forgotten once apply method returns. >>> >>> That way, "spuriousness" is not an issue, once thread returns? >>> >>> dd >>> >>> On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: >>>> They should get unlinked from the global ring. >>>> >>>> Antony Hosking | Associate Professor | Computer Science | Purdue >>>> University >>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>> >>>> >>>> >>>> >>>> >>>> On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: >>>> >>>>> What happens to thread stacks once threads are Join-ed? >>>>> >>>>> dd >>>>> >>>>> On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking wrote: >>>>>> This is probably a result of conservatism for references from >>>>>> thread >>>>>> stacks which results in spurious retention. It may be >>>>>> necessary >>>>>> to be >>>>>> more careful about what references are held on the stack in >>>>>> the >>>>>> threads implementation (and elsewhere in your code). Until we >>>>>> diagnose the place where objects are being retained it will be >>>>>> hard to >>>>>> pinpoint. You should realize that weak refs are a problem in >>>>>> many >>>>>> language implementations, not just Modula-3. I look forward >>>>>> to >>>>>> hearing more definitively from you. >>>>>> >>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>> Purdue >>>>>> University >>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: >>>>>> >>>>>>> Problem in my case can be thoroughness of GC... Because it's >>>>>>> linked >>>>>>> list of WeakRef-ed objects, it's always first element in >>>>>>> list to >>>>>>> become >>>>>>> unreachable first - all others are reachable at least >>>>>>> through >>>>>>> previous >>>>>>> (in list) WeakRef-ed object. So, behaviour I have observed >>>>>>> can >>>>>>> be >>>>>>> because of some thousands of freeable objects, that one is >>>>>>> missed >>>>>>> too >>>>>>> often. >>>>>>> >>>>>>> I have other object being cleaned and counted... There also >>>>>>> cleanup >>>>>>> invocation looks like it's lagging too much. I'll come with >>>>>>> numbers >>>>>>> after I have more tests. >>>>>>> >>>>>>> dd >>>>>>> >>>>>>> On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking wrote: >>>>>>>> Sounds like we need some sort of testcase. Would you be >>>>>>>> able >>>>>>>> to >>>>>>>> devise one? It will be hard to make it deterministic, but >>>>>>>> at >>>>>>>> least we >>>>>>>> should see a non-zero cleanup count. >>>>>>>> >>>>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>>>> Purdue >>>>>>>> University >>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: >>>>>>>> >>>>>>>>> I have single linked list that I use to send messages to >>>>>>>>> many >>>>>>>>> threads. >>>>>>>>> It's end is locked for addition, and current end is >>>>>>>>> given to >>>>>>>>> each >>>>>>>>> new >>>>>>>>> client connecting. This way, all client are going >>>>>>>>> towards >>>>>>>>> end of >>>>>>>>> the >>>>>>>>> list so it's head becomes unreferenced and goes away by >>>>>>>>> GC. >>>>>>>>> >>>>>>>>> Except it does not. >>>>>>>>> >>>>>>>>> I've added WeakRef cleanup and sequential id's so I can >>>>>>>>> record >>>>>>>>> maximum >>>>>>>>> freed id for checking. No single cleanup happens. >>>>>>>>> >>>>>>>>> I'll cross check my code and count my references from >>>>>>>>> application >>>>>>>>> side, >>>>>>>>> but maybe someone else has similar problems? >>>>>>>>> >>>>>>>>> dd >>>>>>>>> -- >>>>>>>>> Dragi?a Duri? >>>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> Dragi?a Duri? >>>>>>> >>>>>> >>>>>> >>>>> -- >>>>> Dragi?a Duri? >>>>> >>>> >>>> >>> -- >>> Dragi?a Duri? >>> >> > -- > Dragi?a Duri? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Thu Mar 13 17:28:54 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 13 Mar 2008 17:28:54 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> Message-ID: <1205425734.32152.144.camel@hias.m3w.org> [17:25:19] 117 users, 1 ops and 1 unresolved; minId = 18617; thrs = 120 [17:25:19] started = 663; cleaned = 452; nilled = 543; joined = 543 [17:25:19] Dangling thrs-a-b-n = 1 [17:25:19] Live queue end is 18621 (head = 18616), maxClean = 18611, cleaned = 16583 About one hour uptime, after I added RTCollector.Collect() call every 20 seconds. dd On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking wrote: > Yes. RTCollectorSRC.FinishCollection; > RTCollectorSRC.StartCollection. This will finish the current > (possibly minor) collection, and start a major collection. The major > collection must finish before you can guarantee garbage has been > freed, so you may want to RTCollectorSRC.FinishCollection. This > sequence is packaged as RTCollector.Collect. > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? wrote: > > > Can I force "major collection" ? > > > > dd > > > > On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: > > > Indeed! Can you devise a testcase? Note that the generational > > > collector will retain old objects for some time until a major > > > collection occurs, > > > > > > Antony Hosking | Associate Professor | Computer Science | Purdue > > > University > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: > > > > > > > And not only that, their stackbase is also set to NIL, meaning > > > > their > > > > stack is not regarded in any way during future collections - > > > > meaning > > > > all > > > > local variables are forgotten once apply method returns. > > > > > > > > That way, "spuriousness" is not an issue, once thread returns? > > > > > > > > dd > > > > > > > > On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: > > > > > They should get unlinked from the global ring. > > > > > > > > > > Antony Hosking | Associate Professor | Computer Science | > > > > > Purdue > > > > > University > > > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: > > > > > > > > > > > What happens to thread stacks once threads are Join-ed? > > > > > > > > > > > > dd > > > > > > > > > > > > On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking wrote: > > > > > > > This is probably a result of conservatism for references > > > > > > > from > > > > > > > thread > > > > > > > stacks which results in spurious retention. It may be > > > > > > > necessary > > > > > > > to be > > > > > > > more careful about what references are held on the stack > > > > > > > in > > > > > > > the > > > > > > > threads implementation (and elsewhere in your code). > > > > > > > Until we > > > > > > > diagnose the place where objects are being retained it > > > > > > > will be > > > > > > > hard to > > > > > > > pinpoint. You should realize that weak refs are a problem > > > > > > > in > > > > > > > many > > > > > > > language implementations, not just Modula-3. I look > > > > > > > forward > > > > > > > to > > > > > > > hearing more definitively from you. > > > > > > > > > > > > > > Antony Hosking | Associate Professor | Computer Science | > > > > > > > Purdue > > > > > > > University > > > > > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: > > > > > > > > > > > > > > > Problem in my case can be thoroughness of GC... Because > > > > > > > > it's > > > > > > > > linked > > > > > > > > list of WeakRef-ed objects, it's always first element in > > > > > > > > list to > > > > > > > > become > > > > > > > > unreachable first - all others are reachable at least > > > > > > > > through > > > > > > > > previous > > > > > > > > (in list) WeakRef-ed object. So, behaviour I have > > > > > > > > observed > > > > > > > > can > > > > > > > > be > > > > > > > > because of some thousands of freeable objects, that one > > > > > > > > is > > > > > > > > missed > > > > > > > > too > > > > > > > > often. > > > > > > > > > > > > > > > > I have other object being cleaned and counted... There > > > > > > > > also > > > > > > > > cleanup > > > > > > > > invocation looks like it's lagging too much. I'll come > > > > > > > > with > > > > > > > > numbers > > > > > > > > after I have more tests. > > > > > > > > > > > > > > > > dd > > > > > > > > > > > > > > > > On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking wrote: > > > > > > > > > Sounds like we need some sort of testcase. Would you > > > > > > > > > be > > > > > > > > > able > > > > > > > > > to > > > > > > > > > devise one? It will be hard to make it deterministic, > > > > > > > > > but > > > > > > > > > at > > > > > > > > > least we > > > > > > > > > should see a non-zero cleanup count. > > > > > > > > > > > > > > > > > > Antony Hosking | Associate Professor | Computer > > > > > > > > > Science | > > > > > > > > > Purdue > > > > > > > > > University > > > > > > > > > 305 N. University Street | West Lafayette | IN 47907 | > > > > > > > > > USA > > > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: > > > > > > > > > > > > > > > > > > > I have single linked list that I use to send > > > > > > > > > > messages to > > > > > > > > > > many > > > > > > > > > > threads. > > > > > > > > > > It's end is locked for addition, and current end is > > > > > > > > > > given to > > > > > > > > > > each > > > > > > > > > > new > > > > > > > > > > client connecting. This way, all client are going > > > > > > > > > > towards > > > > > > > > > > end of > > > > > > > > > > the > > > > > > > > > > list so it's head becomes unreferenced and goes away > > > > > > > > > > by > > > > > > > > > > GC. > > > > > > > > > > > > > > > > > > > > Except it does not. > > > > > > > > > > > > > > > > > > > > I've added WeakRef cleanup and sequential id's so I > > > > > > > > > > can > > > > > > > > > > record > > > > > > > > > > maximum > > > > > > > > > > freed id for checking. No single cleanup happens. > > > > > > > > > > > > > > > > > > > > I'll cross check my code and count my references > > > > > > > > > > from > > > > > > > > > > application > > > > > > > > > > side, > > > > > > > > > > but maybe someone else has similar problems? > > > > > > > > > > > > > > > > > > > > dd > > > > > > > > > > -- > > > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > -- > > > > Dragi?a Duri? > > > > > > > > > -- > > Dragi?a Duri? > > > -- Dragi?a Duri? From hosking at cs.purdue.edu Thu Mar 13 17:45:26 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 13 Mar 2008 12:45:26 -0400 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <1205425734.32152.144.camel@hias.m3w.org> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> <1205425734.32152.144.camel@hias.m3w.org> Message-ID: <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> Realizing that conservative roots collectors like ours can retain data unnecessarily. If a thread stack refers to a page that holds a reference that (transitively) refers to some otherwise dead object then that object will be retained. We say that the page is pinned. Smartening up the collector to ignore dead objects in pinned pages is possible, but not currently implemented. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 13, 2008, at 12:28 PM, Dragi?a Duri? wrote: > [17:25:19] 117 users, 1 ops and 1 unresolved; minId = > 18617; thrs = 120 > [17:25:19] started = 663; cleaned = 452; nilled = 543; > joined = 543 > [17:25:19] Dangling thrs-a-b-n = 1 > [17:25:19] Live queue end is 18621 (head = 18616), > maxClean = 18611, cleaned = 16583 > > About one hour uptime, after I added RTCollector.Collect() call > every 20 > seconds. > > dd > > On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking wrote: >> Yes. RTCollectorSRC.FinishCollection; >> RTCollectorSRC.StartCollection. This will finish the current >> (possibly minor) collection, and start a major collection. The major >> collection must finish before you can guarantee garbage has been >> freed, so you may want to RTCollectorSRC.FinishCollection. This >> sequence is packaged as RTCollector.Collect. >> >> Antony Hosking | Associate Professor | Computer Science | Purdue >> University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> >> >> On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? wrote: >> >>> Can I force "major collection" ? >>> >>> dd >>> >>> On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: >>>> Indeed! Can you devise a testcase? Note that the generational >>>> collector will retain old objects for some time until a major >>>> collection occurs, >>>> >>>> Antony Hosking | Associate Professor | Computer Science | Purdue >>>> University >>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>> >>>> >>>> >>>> >>>> >>>> On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: >>>> >>>>> And not only that, their stackbase is also set to NIL, meaning >>>>> their >>>>> stack is not regarded in any way during future collections - >>>>> meaning >>>>> all >>>>> local variables are forgotten once apply method returns. >>>>> >>>>> That way, "spuriousness" is not an issue, once thread returns? >>>>> >>>>> dd >>>>> >>>>> On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: >>>>>> They should get unlinked from the global ring. >>>>>> >>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>> Purdue >>>>>> University >>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: >>>>>> >>>>>>> What happens to thread stacks once threads are Join-ed? >>>>>>> >>>>>>> dd >>>>>>> >>>>>>> On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking wrote: >>>>>>>> This is probably a result of conservatism for references >>>>>>>> from >>>>>>>> thread >>>>>>>> stacks which results in spurious retention. It may be >>>>>>>> necessary >>>>>>>> to be >>>>>>>> more careful about what references are held on the stack >>>>>>>> in >>>>>>>> the >>>>>>>> threads implementation (and elsewhere in your code). >>>>>>>> Until we >>>>>>>> diagnose the place where objects are being retained it >>>>>>>> will be >>>>>>>> hard to >>>>>>>> pinpoint. You should realize that weak refs are a problem >>>>>>>> in >>>>>>>> many >>>>>>>> language implementations, not just Modula-3. I look >>>>>>>> forward >>>>>>>> to >>>>>>>> hearing more definitively from you. >>>>>>>> >>>>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>>>> Purdue >>>>>>>> University >>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: >>>>>>>> >>>>>>>>> Problem in my case can be thoroughness of GC... Because >>>>>>>>> it's >>>>>>>>> linked >>>>>>>>> list of WeakRef-ed objects, it's always first element in >>>>>>>>> list to >>>>>>>>> become >>>>>>>>> unreachable first - all others are reachable at least >>>>>>>>> through >>>>>>>>> previous >>>>>>>>> (in list) WeakRef-ed object. So, behaviour I have >>>>>>>>> observed >>>>>>>>> can >>>>>>>>> be >>>>>>>>> because of some thousands of freeable objects, that one >>>>>>>>> is >>>>>>>>> missed >>>>>>>>> too >>>>>>>>> often. >>>>>>>>> >>>>>>>>> I have other object being cleaned and counted... There >>>>>>>>> also >>>>>>>>> cleanup >>>>>>>>> invocation looks like it's lagging too much. I'll come >>>>>>>>> with >>>>>>>>> numbers >>>>>>>>> after I have more tests. >>>>>>>>> >>>>>>>>> dd >>>>>>>>> >>>>>>>>> On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking wrote: >>>>>>>>>> Sounds like we need some sort of testcase. Would you >>>>>>>>>> be >>>>>>>>>> able >>>>>>>>>> to >>>>>>>>>> devise one? It will be hard to make it deterministic, >>>>>>>>>> but >>>>>>>>>> at >>>>>>>>>> least we >>>>>>>>>> should see a non-zero cleanup count. >>>>>>>>>> >>>>>>>>>> Antony Hosking | Associate Professor | Computer >>>>>>>>>> Science | >>>>>>>>>> Purdue >>>>>>>>>> University >>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | >>>>>>>>>> USA >>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: >>>>>>>>>> >>>>>>>>>>> I have single linked list that I use to send >>>>>>>>>>> messages to >>>>>>>>>>> many >>>>>>>>>>> threads. >>>>>>>>>>> It's end is locked for addition, and current end is >>>>>>>>>>> given to >>>>>>>>>>> each >>>>>>>>>>> new >>>>>>>>>>> client connecting. This way, all client are going >>>>>>>>>>> towards >>>>>>>>>>> end of >>>>>>>>>>> the >>>>>>>>>>> list so it's head becomes unreferenced and goes away >>>>>>>>>>> by >>>>>>>>>>> GC. >>>>>>>>>>> >>>>>>>>>>> Except it does not. >>>>>>>>>>> >>>>>>>>>>> I've added WeakRef cleanup and sequential id's so I >>>>>>>>>>> can >>>>>>>>>>> record >>>>>>>>>>> maximum >>>>>>>>>>> freed id for checking. No single cleanup happens. >>>>>>>>>>> >>>>>>>>>>> I'll cross check my code and count my references >>>>>>>>>>> from >>>>>>>>>>> application >>>>>>>>>>> side, >>>>>>>>>>> but maybe someone else has similar problems? >>>>>>>>>>> >>>>>>>>>>> dd >>>>>>>>>>> -- >>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Dragi?a Duri? >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> Dragi?a Duri? >>>>>>> >>>>>> >>>>>> >>>>> -- >>>>> Dragi?a Duri? >>>>> >>>> >>> -- >>> Dragi?a Duri? >>> >> > -- > Dragi?a Duri? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Thu Mar 13 18:43:19 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 13 Mar 2008 18:43:19 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> <1205425734.32152.144.camel@hias.m3w.org> <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> Message-ID: <1205430199.32152.152.camel@hias.m3w.org> I understand transitiveness :), but I've made big effort to escape that. Lifetime of thread equals lifetime of Client.T instance, and that instance is kept in thread's closure, and also in local variable. Both are set to NIL before thread termination - all terminations and joins happen and are counted and shown. Pointers to Message objects are kept in two places. Hub.T keeps pointer to list tail, and each Client.T contains pointer to it's current element. That pointer is forwarded as Message's are added to end. No temporary variable, not even WITH substitution is used. And these threads are ones dying with their Client.T's. "head" is new mechanism used to unlink all elements with id less than minimal id used in active Client.T instances. Unlinked totally and all as one free for garbage collector to finish them. I see no transition. dd On Thu, 2008-03-13 at 12:45 -0400, Tony Hosking wrote: > Realizing that conservative roots collectors like ours can retain data > unnecessarily. If a thread stack refers to a page that holds a > reference that (transitively) refers to some otherwise dead object > then that object will be retained. We say that the page is pinned. > Smartening up the collector to ignore dead objects in pinned pages is > possible, but not currently implemented. > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > On Mar 13, 2008, at 12:28 PM, Dragi?a Duri? wrote: > > > [17:25:19] 117 users, 1 ops and 1 unresolved; minId = > > 18617; thrs = 120 > > [17:25:19] started = 663; cleaned = 452; nilled = 543; > > joined = 543 > > [17:25:19] Dangling thrs-a-b-n = 1 > > [17:25:19] Live queue end is 18621 (head = 18616), > > maxClean = 18611, cleaned = 16583 > > > > About one hour uptime, after I added RTCollector.Collect() call > > every 20 > > seconds. > > > > dd > > > > On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking wrote: > > > Yes. RTCollectorSRC.FinishCollection; > > > RTCollectorSRC.StartCollection. This will finish the current > > > (possibly minor) collection, and start a major collection. The > > > major > > > collection must finish before you can guarantee garbage has been > > > freed, so you may want to RTCollectorSRC.FinishCollection. This > > > sequence is packaged as RTCollector.Collect. > > > > > > Antony Hosking | Associate Professor | Computer Science | Purdue > > > University > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? wrote: > > > > > > > Can I force "major collection" ? > > > > > > > > dd > > > > > > > > On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: > > > > > Indeed! Can you devise a testcase? Note that the generational > > > > > collector will retain old objects for some time until a major > > > > > collection occurs, > > > > > > > > > > Antony Hosking | Associate Professor | Computer Science | > > > > > Purdue > > > > > University > > > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: > > > > > > > > > > > And not only that, their stackbase is also set to NIL, > > > > > > meaning > > > > > > their > > > > > > stack is not regarded in any way during future collections - > > > > > > meaning > > > > > > all > > > > > > local variables are forgotten once apply method returns. > > > > > > > > > > > > That way, "spuriousness" is not an issue, once thread > > > > > > returns? > > > > > > > > > > > > dd > > > > > > > > > > > > On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: > > > > > > > They should get unlinked from the global ring. > > > > > > > > > > > > > > Antony Hosking | Associate Professor | Computer Science | > > > > > > > Purdue > > > > > > > University > > > > > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: > > > > > > > > > > > > > > > What happens to thread stacks once threads are Join-ed? > > > > > > > > > > > > > > > > dd > > > > > > > > > > > > > > > > On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking wrote: > > > > > > > > > This is probably a result of conservatism for > > > > > > > > > references > > > > > > > > > from > > > > > > > > > thread > > > > > > > > > stacks which results in spurious retention. It may be > > > > > > > > > necessary > > > > > > > > > to be > > > > > > > > > more careful about what references are held on the > > > > > > > > > stack > > > > > > > > > in > > > > > > > > > the > > > > > > > > > threads implementation (and elsewhere in your code). > > > > > > > > > Until we > > > > > > > > > diagnose the place where objects are being retained it > > > > > > > > > will be > > > > > > > > > hard to > > > > > > > > > pinpoint. You should realize that weak refs are a > > > > > > > > > problem > > > > > > > > > in > > > > > > > > > many > > > > > > > > > language implementations, not just Modula-3. I look > > > > > > > > > forward > > > > > > > > > to > > > > > > > > > hearing more definitively from you. > > > > > > > > > > > > > > > > > > Antony Hosking | Associate Professor | Computer > > > > > > > > > Science | > > > > > > > > > Purdue > > > > > > > > > University > > > > > > > > > 305 N. University Street | West Lafayette | IN 47907 | > > > > > > > > > USA > > > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: > > > > > > > > > > > > > > > > > > > Problem in my case can be thoroughness of GC... > > > > > > > > > > Because > > > > > > > > > > it's > > > > > > > > > > linked > > > > > > > > > > list of WeakRef-ed objects, it's always first > > > > > > > > > > element in > > > > > > > > > > list to > > > > > > > > > > become > > > > > > > > > > unreachable first - all others are reachable at > > > > > > > > > > least > > > > > > > > > > through > > > > > > > > > > previous > > > > > > > > > > (in list) WeakRef-ed object. So, behaviour I have > > > > > > > > > > observed > > > > > > > > > > can > > > > > > > > > > be > > > > > > > > > > because of some thousands of freeable objects, that > > > > > > > > > > one > > > > > > > > > > is > > > > > > > > > > missed > > > > > > > > > > too > > > > > > > > > > often. > > > > > > > > > > > > > > > > > > > > I have other object being cleaned and counted... > > > > > > > > > > There > > > > > > > > > > also > > > > > > > > > > cleanup > > > > > > > > > > invocation looks like it's lagging too much. I'll > > > > > > > > > > come > > > > > > > > > > with > > > > > > > > > > numbers > > > > > > > > > > after I have more tests. > > > > > > > > > > > > > > > > > > > > dd > > > > > > > > > > > > > > > > > > > > On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking > > > > > > > > > > wrote: > > > > > > > > > > > Sounds like we need some sort of testcase. Would > > > > > > > > > > > you > > > > > > > > > > > be > > > > > > > > > > > able > > > > > > > > > > > to > > > > > > > > > > > devise one? It will be hard to make it > > > > > > > > > > > deterministic, > > > > > > > > > > > but > > > > > > > > > > > at > > > > > > > > > > > least we > > > > > > > > > > > should see a non-zero cleanup count. > > > > > > > > > > > > > > > > > > > > > > Antony Hosking | Associate Professor | Computer > > > > > > > > > > > Science | > > > > > > > > > > > Purdue > > > > > > > > > > > University > > > > > > > > > > > 305 N. University Street | West Lafayette | IN > > > > > > > > > > > 47907 | > > > > > > > > > > > USA > > > > > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: > > > > > > > > > > > > > > > > > > > > > > > I have single linked list that I use to send > > > > > > > > > > > > messages to > > > > > > > > > > > > many > > > > > > > > > > > > threads. > > > > > > > > > > > > It's end is locked for addition, and current end > > > > > > > > > > > > is > > > > > > > > > > > > given to > > > > > > > > > > > > each > > > > > > > > > > > > new > > > > > > > > > > > > client connecting. This way, all client are > > > > > > > > > > > > going > > > > > > > > > > > > towards > > > > > > > > > > > > end of > > > > > > > > > > > > the > > > > > > > > > > > > list so it's head becomes unreferenced and goes > > > > > > > > > > > > away > > > > > > > > > > > > by > > > > > > > > > > > > GC. > > > > > > > > > > > > > > > > > > > > > > > > Except it does not. > > > > > > > > > > > > > > > > > > > > > > > > I've added WeakRef cleanup and sequential id's > > > > > > > > > > > > so I > > > > > > > > > > > > can > > > > > > > > > > > > record > > > > > > > > > > > > maximum > > > > > > > > > > > > freed id for checking. No single cleanup > > > > > > > > > > > > happens. > > > > > > > > > > > > > > > > > > > > > > > > I'll cross check my code and count my references > > > > > > > > > > > > from > > > > > > > > > > > > application > > > > > > > > > > > > side, > > > > > > > > > > > > but maybe someone else has similar problems? > > > > > > > > > > > > > > > > > > > > > > > > dd > > > > > > > > > > > > -- > > > > > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > -- > > > > Dragi?a Duri? > > > > > > > > > -- > > Dragi?a Duri? > > > -- Dragi?a Duri? From hosking at cs.purdue.edu Thu Mar 13 22:41:56 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 13 Mar 2008 17:41:56 -0400 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <1205430199.32152.152.camel@hias.m3w.org> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> <1205425734.32152.144.camel@hias.m3w.org> <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> <1205430199.32152.152.camel@hias.m3w.org> Message-ID: The thing about conservatism is that even if you NIL *all* references to an object that just happens to lie on a pinned page, that object (and objects reachable from it) will be retained until the page is no longer pinned. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 13, 2008, at 1:43 PM, Dragi?a Duri? wrote: > I understand transitiveness :), but I've made big effort to escape > that. > > Lifetime of thread equals lifetime of Client.T instance, and that > instance is kept in thread's closure, and also in local variable. Both > are set to NIL before thread termination - all terminations and joins > happen and are counted and shown. > > Pointers to Message objects are kept in two places. Hub.T keeps > pointer > to list tail, and each Client.T contains pointer to it's current > element. That pointer is forwarded as Message's are added to end. No > temporary variable, not even WITH substitution is used. And these > threads are ones dying with their Client.T's. > > "head" is new mechanism used to unlink all elements with id less than > minimal id used in active Client.T instances. Unlinked totally and all > as one free for garbage collector to finish them. > > I see no transition. > > dd > > On Thu, 2008-03-13 at 12:45 -0400, Tony Hosking wrote: >> Realizing that conservative roots collectors like ours can retain >> data >> unnecessarily. If a thread stack refers to a page that holds a >> reference that (transitively) refers to some otherwise dead object >> then that object will be retained. We say that the page is pinned. >> Smartening up the collector to ignore dead objects in pinned pages is >> possible, but not currently implemented. >> >> Antony Hosking | Associate Professor | Computer Science | Purdue >> University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> >> >> On Mar 13, 2008, at 12:28 PM, Dragi?a Duri? wrote: >> >>> [17:25:19] 117 users, 1 ops and 1 unresolved; minId = >>> 18617; thrs = 120 >>> [17:25:19] started = 663; cleaned = 452; nilled = 543; >>> joined = 543 >>> [17:25:19] Dangling thrs-a-b-n = 1 >>> [17:25:19] Live queue end is 18621 (head = 18616), >>> maxClean = 18611, cleaned = 16583 >>> >>> About one hour uptime, after I added RTCollector.Collect() call >>> every 20 >>> seconds. >>> >>> dd >>> >>> On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking wrote: >>>> Yes. RTCollectorSRC.FinishCollection; >>>> RTCollectorSRC.StartCollection. This will finish the current >>>> (possibly minor) collection, and start a major collection. The >>>> major >>>> collection must finish before you can guarantee garbage has been >>>> freed, so you may want to RTCollectorSRC.FinishCollection. This >>>> sequence is packaged as RTCollector.Collect. >>>> >>>> Antony Hosking | Associate Professor | Computer Science | Purdue >>>> University >>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>> >>>> >>>> >>>> >>>> >>>> On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? wrote: >>>> >>>>> Can I force "major collection" ? >>>>> >>>>> dd >>>>> >>>>> On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: >>>>>> Indeed! Can you devise a testcase? Note that the generational >>>>>> collector will retain old objects for some time until a major >>>>>> collection occurs, >>>>>> >>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>> Purdue >>>>>> University >>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: >>>>>> >>>>>>> And not only that, their stackbase is also set to NIL, >>>>>>> meaning >>>>>>> their >>>>>>> stack is not regarded in any way during future collections - >>>>>>> meaning >>>>>>> all >>>>>>> local variables are forgotten once apply method returns. >>>>>>> >>>>>>> That way, "spuriousness" is not an issue, once thread >>>>>>> returns? >>>>>>> >>>>>>> dd >>>>>>> >>>>>>> On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: >>>>>>>> They should get unlinked from the global ring. >>>>>>>> >>>>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>>>> Purdue >>>>>>>> University >>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: >>>>>>>> >>>>>>>>> What happens to thread stacks once threads are Join-ed? >>>>>>>>> >>>>>>>>> dd >>>>>>>>> >>>>>>>>> On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking wrote: >>>>>>>>>> This is probably a result of conservatism for >>>>>>>>>> references >>>>>>>>>> from >>>>>>>>>> thread >>>>>>>>>> stacks which results in spurious retention. It may be >>>>>>>>>> necessary >>>>>>>>>> to be >>>>>>>>>> more careful about what references are held on the >>>>>>>>>> stack >>>>>>>>>> in >>>>>>>>>> the >>>>>>>>>> threads implementation (and elsewhere in your code). >>>>>>>>>> Until we >>>>>>>>>> diagnose the place where objects are being retained it >>>>>>>>>> will be >>>>>>>>>> hard to >>>>>>>>>> pinpoint. You should realize that weak refs are a >>>>>>>>>> problem >>>>>>>>>> in >>>>>>>>>> many >>>>>>>>>> language implementations, not just Modula-3. I look >>>>>>>>>> forward >>>>>>>>>> to >>>>>>>>>> hearing more definitively from you. >>>>>>>>>> >>>>>>>>>> Antony Hosking | Associate Professor | Computer >>>>>>>>>> Science | >>>>>>>>>> Purdue >>>>>>>>>> University >>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | >>>>>>>>>> USA >>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: >>>>>>>>>> >>>>>>>>>>> Problem in my case can be thoroughness of GC... >>>>>>>>>>> Because >>>>>>>>>>> it's >>>>>>>>>>> linked >>>>>>>>>>> list of WeakRef-ed objects, it's always first >>>>>>>>>>> element in >>>>>>>>>>> list to >>>>>>>>>>> become >>>>>>>>>>> unreachable first - all others are reachable at >>>>>>>>>>> least >>>>>>>>>>> through >>>>>>>>>>> previous >>>>>>>>>>> (in list) WeakRef-ed object. So, behaviour I have >>>>>>>>>>> observed >>>>>>>>>>> can >>>>>>>>>>> be >>>>>>>>>>> because of some thousands of freeable objects, that >>>>>>>>>>> one >>>>>>>>>>> is >>>>>>>>>>> missed >>>>>>>>>>> too >>>>>>>>>>> often. >>>>>>>>>>> >>>>>>>>>>> I have other object being cleaned and counted... >>>>>>>>>>> There >>>>>>>>>>> also >>>>>>>>>>> cleanup >>>>>>>>>>> invocation looks like it's lagging too much. I'll >>>>>>>>>>> come >>>>>>>>>>> with >>>>>>>>>>> numbers >>>>>>>>>>> after I have more tests. >>>>>>>>>>> >>>>>>>>>>> dd >>>>>>>>>>> >>>>>>>>>>> On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking >>>>>>>>>>> wrote: >>>>>>>>>>>> Sounds like we need some sort of testcase. Would >>>>>>>>>>>> you >>>>>>>>>>>> be >>>>>>>>>>>> able >>>>>>>>>>>> to >>>>>>>>>>>> devise one? It will be hard to make it >>>>>>>>>>>> deterministic, >>>>>>>>>>>> but >>>>>>>>>>>> at >>>>>>>>>>>> least we >>>>>>>>>>>> should see a non-zero cleanup count. >>>>>>>>>>>> >>>>>>>>>>>> Antony Hosking | Associate Professor | Computer >>>>>>>>>>>> Science | >>>>>>>>>>>> Purdue >>>>>>>>>>>> University >>>>>>>>>>>> 305 N. University Street | West Lafayette | IN >>>>>>>>>>>> 47907 | >>>>>>>>>>>> USA >>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I have single linked list that I use to send >>>>>>>>>>>>> messages to >>>>>>>>>>>>> many >>>>>>>>>>>>> threads. >>>>>>>>>>>>> It's end is locked for addition, and current end >>>>>>>>>>>>> is >>>>>>>>>>>>> given to >>>>>>>>>>>>> each >>>>>>>>>>>>> new >>>>>>>>>>>>> client connecting. This way, all client are >>>>>>>>>>>>> going >>>>>>>>>>>>> towards >>>>>>>>>>>>> end of >>>>>>>>>>>>> the >>>>>>>>>>>>> list so it's head becomes unreferenced and goes >>>>>>>>>>>>> away >>>>>>>>>>>>> by >>>>>>>>>>>>> GC. >>>>>>>>>>>>> >>>>>>>>>>>>> Except it does not. >>>>>>>>>>>>> >>>>>>>>>>>>> I've added WeakRef cleanup and sequential id's >>>>>>>>>>>>> so I >>>>>>>>>>>>> can >>>>>>>>>>>>> record >>>>>>>>>>>>> maximum >>>>>>>>>>>>> freed id for checking. No single cleanup >>>>>>>>>>>>> happens. >>>>>>>>>>>>> >>>>>>>>>>>>> I'll cross check my code and count my references >>>>>>>>>>>>> from >>>>>>>>>>>>> application >>>>>>>>>>>>> side, >>>>>>>>>>>>> but maybe someone else has similar problems? >>>>>>>>>>>>> >>>>>>>>>>>>> dd >>>>>>>>>>>>> -- >>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Dragi?a Duri? >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> Dragi?a Duri? >>>>>>> >>>>>> >>>>> -- >>>>> Dragi?a Duri? >>>>> >>>> >>> -- >>> Dragi?a Duri? >>> >> > -- > Dragi?a Duri? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Thu Mar 13 22:46:14 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 13 Mar 2008 22:46:14 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> <1205425734.32152.144.camel@hias.m3w.org> <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> <1205430199.32152.152.camel@hias.m3w.org> Message-ID: <1205444774.32152.158.camel@hias.m3w.org> Are objects on pages prepacked/moved? On Thu, 2008-03-13 at 17:41 -0400, Tony Hosking wrote: > The thing about conservatism is that even if you NIL *all* references > to an object that just happens to lie on a pinned page, that object > (and objects reachable from it) will be retained until the page is no > longer pinned. > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > On Mar 13, 2008, at 1:43 PM, Dragi?a Duri? wrote: > > > I understand transitiveness :), but I've made big effort to escape > > that. > > > > Lifetime of thread equals lifetime of Client.T instance, and that > > instance is kept in thread's closure, and also in local variable. > > Both > > are set to NIL before thread termination - all terminations and > > joins > > happen and are counted and shown. > > > > Pointers to Message objects are kept in two places. Hub.T keeps > > pointer > > to list tail, and each Client.T contains pointer to it's current > > element. That pointer is forwarded as Message's are added to end. No > > temporary variable, not even WITH substitution is used. And these > > threads are ones dying with their Client.T's. > > > > "head" is new mechanism used to unlink all elements with id less > > than > > minimal id used in active Client.T instances. Unlinked totally and > > all > > as one free for garbage collector to finish them. > > > > I see no transition. > > > > dd > > > > On Thu, 2008-03-13 at 12:45 -0400, Tony Hosking wrote: > > > Realizing that conservative roots collectors like ours can retain > > > data > > > unnecessarily. If a thread stack refers to a page that holds a > > > reference that (transitively) refers to some otherwise dead object > > > then that object will be retained. We say that the page is > > > pinned. > > > Smartening up the collector to ignore dead objects in pinned pages > > > is > > > possible, but not currently implemented. > > > > > > Antony Hosking | Associate Professor | Computer Science | Purdue > > > University > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 12:28 PM, Dragi?a Duri? wrote: > > > > > > > [17:25:19] 117 users, 1 ops and 1 unresolved; minId = > > > > 18617; thrs = 120 > > > > [17:25:19] started = 663; cleaned = 452; nilled = 543; > > > > joined = 543 > > > > [17:25:19] Dangling thrs-a-b-n = 1 > > > > [17:25:19] Live queue end is 18621 (head = 18616), > > > > maxClean = 18611, cleaned = 16583 > > > > > > > > About one hour uptime, after I added RTCollector.Collect() call > > > > every 20 > > > > seconds. > > > > > > > > dd > > > > > > > > On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking wrote: > > > > > Yes. RTCollectorSRC.FinishCollection; > > > > > RTCollectorSRC.StartCollection. This will finish the current > > > > > (possibly minor) collection, and start a major collection. > > > > > The > > > > > major > > > > > collection must finish before you can guarantee garbage has > > > > > been > > > > > freed, so you may want to RTCollectorSRC.FinishCollection. > > > > > This > > > > > sequence is packaged as RTCollector.Collect. > > > > > > > > > > Antony Hosking | Associate Professor | Computer Science | > > > > > Purdue > > > > > University > > > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? wrote: > > > > > > > > > > > Can I force "major collection" ? > > > > > > > > > > > > dd > > > > > > > > > > > > On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: > > > > > > > Indeed! Can you devise a testcase? Note that the > > > > > > > generational > > > > > > > collector will retain old objects for some time until a > > > > > > > major > > > > > > > collection occurs, > > > > > > > > > > > > > > Antony Hosking | Associate Professor | Computer Science | > > > > > > > Purdue > > > > > > > University > > > > > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: > > > > > > > > > > > > > > > And not only that, their stackbase is also set to NIL, > > > > > > > > meaning > > > > > > > > their > > > > > > > > stack is not regarded in any way during future > > > > > > > > collections - > > > > > > > > meaning > > > > > > > > all > > > > > > > > local variables are forgotten once apply method returns. > > > > > > > > > > > > > > > > That way, "spuriousness" is not an issue, once thread > > > > > > > > returns? > > > > > > > > > > > > > > > > dd > > > > > > > > > > > > > > > > On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: > > > > > > > > > They should get unlinked from the global ring. > > > > > > > > > > > > > > > > > > Antony Hosking | Associate Professor | Computer > > > > > > > > > Science | > > > > > > > > > Purdue > > > > > > > > > University > > > > > > > > > 305 N. University Street | West Lafayette | IN 47907 | > > > > > > > > > USA > > > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: > > > > > > > > > > > > > > > > > > > What happens to thread stacks once threads are > > > > > > > > > > Join-ed? > > > > > > > > > > > > > > > > > > > > dd > > > > > > > > > > > > > > > > > > > > On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking > > > > > > > > > > wrote: > > > > > > > > > > > This is probably a result of conservatism for > > > > > > > > > > > references > > > > > > > > > > > from > > > > > > > > > > > thread > > > > > > > > > > > stacks which results in spurious retention. It > > > > > > > > > > > may be > > > > > > > > > > > necessary > > > > > > > > > > > to be > > > > > > > > > > > more careful about what references are held on the > > > > > > > > > > > stack > > > > > > > > > > > in > > > > > > > > > > > the > > > > > > > > > > > threads implementation (and elsewhere in your > > > > > > > > > > > code). > > > > > > > > > > > Until we > > > > > > > > > > > diagnose the place where objects are being > > > > > > > > > > > retained it > > > > > > > > > > > will be > > > > > > > > > > > hard to > > > > > > > > > > > pinpoint. You should realize that weak refs are a > > > > > > > > > > > problem > > > > > > > > > > > in > > > > > > > > > > > many > > > > > > > > > > > language implementations, not just Modula-3. I > > > > > > > > > > > look > > > > > > > > > > > forward > > > > > > > > > > > to > > > > > > > > > > > hearing more definitively from you. > > > > > > > > > > > > > > > > > > > > > > Antony Hosking | Associate Professor | Computer > > > > > > > > > > > Science | > > > > > > > > > > > Purdue > > > > > > > > > > > University > > > > > > > > > > > 305 N. University Street | West Lafayette | IN > > > > > > > > > > > 47907 | > > > > > > > > > > > USA > > > > > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: > > > > > > > > > > > > > > > > > > > > > > > Problem in my case can be thoroughness of GC... > > > > > > > > > > > > Because > > > > > > > > > > > > it's > > > > > > > > > > > > linked > > > > > > > > > > > > list of WeakRef-ed objects, it's always first > > > > > > > > > > > > element in > > > > > > > > > > > > list to > > > > > > > > > > > > become > > > > > > > > > > > > unreachable first - all others are reachable at > > > > > > > > > > > > least > > > > > > > > > > > > through > > > > > > > > > > > > previous > > > > > > > > > > > > (in list) WeakRef-ed object. So, behaviour I > > > > > > > > > > > > have > > > > > > > > > > > > observed > > > > > > > > > > > > can > > > > > > > > > > > > be > > > > > > > > > > > > because of some thousands of freeable objects, > > > > > > > > > > > > that > > > > > > > > > > > > one > > > > > > > > > > > > is > > > > > > > > > > > > missed > > > > > > > > > > > > too > > > > > > > > > > > > often. > > > > > > > > > > > > > > > > > > > > > > > > I have other object being cleaned and counted... > > > > > > > > > > > > There > > > > > > > > > > > > also > > > > > > > > > > > > cleanup > > > > > > > > > > > > invocation looks like it's lagging too much. > > > > > > > > > > > > I'll > > > > > > > > > > > > come > > > > > > > > > > > > with > > > > > > > > > > > > numbers > > > > > > > > > > > > after I have more tests. > > > > > > > > > > > > > > > > > > > > > > > > dd > > > > > > > > > > > > > > > > > > > > > > > > On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Sounds like we need some sort of testcase. > > > > > > > > > > > > > Would > > > > > > > > > > > > > you > > > > > > > > > > > > > be > > > > > > > > > > > > > able > > > > > > > > > > > > > to > > > > > > > > > > > > > devise one? It will be hard to make it > > > > > > > > > > > > > deterministic, > > > > > > > > > > > > > but > > > > > > > > > > > > > at > > > > > > > > > > > > > least we > > > > > > > > > > > > > should see a non-zero cleanup count. > > > > > > > > > > > > > > > > > > > > > > > > > > Antony Hosking | Associate Professor | > > > > > > > > > > > > > Computer > > > > > > > > > > > > > Science | > > > > > > > > > > > > > Purdue > > > > > > > > > > > > > University > > > > > > > > > > > > > 305 N. University Street | West Lafayette | IN > > > > > > > > > > > > > 47907 | > > > > > > > > > > > > > USA > > > > > > > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 > > > > > > > > > > > > > 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > I have single linked list that I use to send > > > > > > > > > > > > > > messages to > > > > > > > > > > > > > > many > > > > > > > > > > > > > > threads. > > > > > > > > > > > > > > It's end is locked for addition, and current > > > > > > > > > > > > > > end > > > > > > > > > > > > > > is > > > > > > > > > > > > > > given to > > > > > > > > > > > > > > each > > > > > > > > > > > > > > new > > > > > > > > > > > > > > client connecting. This way, all client are > > > > > > > > > > > > > > going > > > > > > > > > > > > > > towards > > > > > > > > > > > > > > end of > > > > > > > > > > > > > > the > > > > > > > > > > > > > > list so it's head becomes unreferenced and > > > > > > > > > > > > > > goes > > > > > > > > > > > > > > away > > > > > > > > > > > > > > by > > > > > > > > > > > > > > GC. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Except it does not. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I've added WeakRef cleanup and sequential > > > > > > > > > > > > > > id's > > > > > > > > > > > > > > so I > > > > > > > > > > > > > > can > > > > > > > > > > > > > > record > > > > > > > > > > > > > > maximum > > > > > > > > > > > > > > freed id for checking. No single cleanup > > > > > > > > > > > > > > happens. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'll cross check my code and count my > > > > > > > > > > > > > > references > > > > > > > > > > > > > > from > > > > > > > > > > > > > > application > > > > > > > > > > > > > > side, > > > > > > > > > > > > > > but maybe someone else has similar problems? > > > > > > > > > > > > > > > > > > > > > > > > > > > > dd > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > -- > > > > Dragi?a Duri? > > > > > > > > > -- > > Dragi?a Duri? > > > -- Dragi?a Duri? From hosking at cs.purdue.edu Thu Mar 13 23:03:36 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 13 Mar 2008 18:03:36 -0400 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <1205444774.32152.158.camel@hias.m3w.org> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> <1205425734.32152.144.camel@hias.m3w.org> <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> <1205430199.32152.152.camel@hias.m3w.org> <1205444774.32152.158.camel@hias.m3w.org> Message-ID: <5649BA54-5B2F-423C-AEF7-65FF09CF3763@cs.purdue.edu> I'm not sure I understand what you mean? On Mar 13, 2008, at 5:46 PM, Dragi?a Duri? wrote: > Are objects on pages prepacked/moved? > > On Thu, 2008-03-13 at 17:41 -0400, Tony Hosking wrote: >> The thing about conservatism is that even if you NIL *all* references >> to an object that just happens to lie on a pinned page, that object >> (and objects reachable from it) will be retained until the page is no >> longer pinned. >> >> Antony Hosking | Associate Professor | Computer Science | Purdue >> University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> >> >> On Mar 13, 2008, at 1:43 PM, Dragi?a Duri? wrote: >> >>> I understand transitiveness :), but I've made big effort to escape >>> that. >>> >>> Lifetime of thread equals lifetime of Client.T instance, and that >>> instance is kept in thread's closure, and also in local variable. >>> Both >>> are set to NIL before thread termination - all terminations and >>> joins >>> happen and are counted and shown. >>> >>> Pointers to Message objects are kept in two places. Hub.T keeps >>> pointer >>> to list tail, and each Client.T contains pointer to it's current >>> element. That pointer is forwarded as Message's are added to end. No >>> temporary variable, not even WITH substitution is used. And these >>> threads are ones dying with their Client.T's. >>> >>> "head" is new mechanism used to unlink all elements with id less >>> than >>> minimal id used in active Client.T instances. Unlinked totally and >>> all >>> as one free for garbage collector to finish them. >>> >>> I see no transition. >>> >>> dd >>> >>> On Thu, 2008-03-13 at 12:45 -0400, Tony Hosking wrote: >>>> Realizing that conservative roots collectors like ours can retain >>>> data >>>> unnecessarily. If a thread stack refers to a page that holds a >>>> reference that (transitively) refers to some otherwise dead object >>>> then that object will be retained. We say that the page is >>>> pinned. >>>> Smartening up the collector to ignore dead objects in pinned pages >>>> is >>>> possible, but not currently implemented. >>>> >>>> Antony Hosking | Associate Professor | Computer Science | Purdue >>>> University >>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>> >>>> >>>> >>>> >>>> >>>> On Mar 13, 2008, at 12:28 PM, Dragi?a Duri? wrote: >>>> >>>>> [17:25:19] 117 users, 1 ops and 1 unresolved; minId = >>>>> 18617; thrs = 120 >>>>> [17:25:19] started = 663; cleaned = 452; nilled = 543; >>>>> joined = 543 >>>>> [17:25:19] Dangling thrs-a-b-n = 1 >>>>> [17:25:19] Live queue end is 18621 (head = 18616), >>>>> maxClean = 18611, cleaned = 16583 >>>>> >>>>> About one hour uptime, after I added RTCollector.Collect() call >>>>> every 20 >>>>> seconds. >>>>> >>>>> dd >>>>> >>>>> On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking wrote: >>>>>> Yes. RTCollectorSRC.FinishCollection; >>>>>> RTCollectorSRC.StartCollection. This will finish the current >>>>>> (possibly minor) collection, and start a major collection. >>>>>> The >>>>>> major >>>>>> collection must finish before you can guarantee garbage has >>>>>> been >>>>>> freed, so you may want to RTCollectorSRC.FinishCollection. >>>>>> This >>>>>> sequence is packaged as RTCollector.Collect. >>>>>> >>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>> Purdue >>>>>> University >>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? wrote: >>>>>> >>>>>>> Can I force "major collection" ? >>>>>>> >>>>>>> dd >>>>>>> >>>>>>> On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: >>>>>>>> Indeed! Can you devise a testcase? Note that the >>>>>>>> generational >>>>>>>> collector will retain old objects for some time until a >>>>>>>> major >>>>>>>> collection occurs, >>>>>>>> >>>>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>>>> Purdue >>>>>>>> University >>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: >>>>>>>> >>>>>>>>> And not only that, their stackbase is also set to NIL, >>>>>>>>> meaning >>>>>>>>> their >>>>>>>>> stack is not regarded in any way during future >>>>>>>>> collections - >>>>>>>>> meaning >>>>>>>>> all >>>>>>>>> local variables are forgotten once apply method returns. >>>>>>>>> >>>>>>>>> That way, "spuriousness" is not an issue, once thread >>>>>>>>> returns? >>>>>>>>> >>>>>>>>> dd >>>>>>>>> >>>>>>>>> On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: >>>>>>>>>> They should get unlinked from the global ring. >>>>>>>>>> >>>>>>>>>> Antony Hosking | Associate Professor | Computer >>>>>>>>>> Science | >>>>>>>>>> Purdue >>>>>>>>>> University >>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | >>>>>>>>>> USA >>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: >>>>>>>>>> >>>>>>>>>>> What happens to thread stacks once threads are >>>>>>>>>>> Join-ed? >>>>>>>>>>> >>>>>>>>>>> dd >>>>>>>>>>> >>>>>>>>>>> On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking >>>>>>>>>>> wrote: >>>>>>>>>>>> This is probably a result of conservatism for >>>>>>>>>>>> references >>>>>>>>>>>> from >>>>>>>>>>>> thread >>>>>>>>>>>> stacks which results in spurious retention. It >>>>>>>>>>>> may be >>>>>>>>>>>> necessary >>>>>>>>>>>> to be >>>>>>>>>>>> more careful about what references are held on the >>>>>>>>>>>> stack >>>>>>>>>>>> in >>>>>>>>>>>> the >>>>>>>>>>>> threads implementation (and elsewhere in your >>>>>>>>>>>> code). >>>>>>>>>>>> Until we >>>>>>>>>>>> diagnose the place where objects are being >>>>>>>>>>>> retained it >>>>>>>>>>>> will be >>>>>>>>>>>> hard to >>>>>>>>>>>> pinpoint. You should realize that weak refs are a >>>>>>>>>>>> problem >>>>>>>>>>>> in >>>>>>>>>>>> many >>>>>>>>>>>> language implementations, not just Modula-3. I >>>>>>>>>>>> look >>>>>>>>>>>> forward >>>>>>>>>>>> to >>>>>>>>>>>> hearing more definitively from you. >>>>>>>>>>>> >>>>>>>>>>>> Antony Hosking | Associate Professor | Computer >>>>>>>>>>>> Science | >>>>>>>>>>>> Purdue >>>>>>>>>>>> University >>>>>>>>>>>> 305 N. University Street | West Lafayette | IN >>>>>>>>>>>> 47907 | >>>>>>>>>>>> USA >>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Problem in my case can be thoroughness of GC... >>>>>>>>>>>>> Because >>>>>>>>>>>>> it's >>>>>>>>>>>>> linked >>>>>>>>>>>>> list of WeakRef-ed objects, it's always first >>>>>>>>>>>>> element in >>>>>>>>>>>>> list to >>>>>>>>>>>>> become >>>>>>>>>>>>> unreachable first - all others are reachable at >>>>>>>>>>>>> least >>>>>>>>>>>>> through >>>>>>>>>>>>> previous >>>>>>>>>>>>> (in list) WeakRef-ed object. So, behaviour I >>>>>>>>>>>>> have >>>>>>>>>>>>> observed >>>>>>>>>>>>> can >>>>>>>>>>>>> be >>>>>>>>>>>>> because of some thousands of freeable objects, >>>>>>>>>>>>> that >>>>>>>>>>>>> one >>>>>>>>>>>>> is >>>>>>>>>>>>> missed >>>>>>>>>>>>> too >>>>>>>>>>>>> often. >>>>>>>>>>>>> >>>>>>>>>>>>> I have other object being cleaned and counted... >>>>>>>>>>>>> There >>>>>>>>>>>>> also >>>>>>>>>>>>> cleanup >>>>>>>>>>>>> invocation looks like it's lagging too much. >>>>>>>>>>>>> I'll >>>>>>>>>>>>> come >>>>>>>>>>>>> with >>>>>>>>>>>>> numbers >>>>>>>>>>>>> after I have more tests. >>>>>>>>>>>>> >>>>>>>>>>>>> dd >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> Sounds like we need some sort of testcase. >>>>>>>>>>>>>> Would >>>>>>>>>>>>>> you >>>>>>>>>>>>>> be >>>>>>>>>>>>>> able >>>>>>>>>>>>>> to >>>>>>>>>>>>>> devise one? It will be hard to make it >>>>>>>>>>>>>> deterministic, >>>>>>>>>>>>>> but >>>>>>>>>>>>>> at >>>>>>>>>>>>>> least we >>>>>>>>>>>>>> should see a non-zero cleanup count. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Antony Hosking | Associate Professor | >>>>>>>>>>>>>> Computer >>>>>>>>>>>>>> Science | >>>>>>>>>>>>>> Purdue >>>>>>>>>>>>>> University >>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN >>>>>>>>>>>>>> 47907 | >>>>>>>>>>>>>> USA >>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 >>>>>>>>>>>>>> 5484 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I have single linked list that I use to send >>>>>>>>>>>>>>> messages to >>>>>>>>>>>>>>> many >>>>>>>>>>>>>>> threads. >>>>>>>>>>>>>>> It's end is locked for addition, and current >>>>>>>>>>>>>>> end >>>>>>>>>>>>>>> is >>>>>>>>>>>>>>> given to >>>>>>>>>>>>>>> each >>>>>>>>>>>>>>> new >>>>>>>>>>>>>>> client connecting. This way, all client are >>>>>>>>>>>>>>> going >>>>>>>>>>>>>>> towards >>>>>>>>>>>>>>> end of >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> list so it's head becomes unreferenced and >>>>>>>>>>>>>>> goes >>>>>>>>>>>>>>> away >>>>>>>>>>>>>>> by >>>>>>>>>>>>>>> GC. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Except it does not. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I've added WeakRef cleanup and sequential >>>>>>>>>>>>>>> id's >>>>>>>>>>>>>>> so I >>>>>>>>>>>>>>> can >>>>>>>>>>>>>>> record >>>>>>>>>>>>>>> maximum >>>>>>>>>>>>>>> freed id for checking. No single cleanup >>>>>>>>>>>>>>> happens. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'll cross check my code and count my >>>>>>>>>>>>>>> references >>>>>>>>>>>>>>> from >>>>>>>>>>>>>>> application >>>>>>>>>>>>>>> side, >>>>>>>>>>>>>>> but maybe someone else has similar problems? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> dd >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Dragi?a Duri? >>>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> Dragi?a Duri? >>>>>>> >>>>>> >>>>> -- >>>>> Dragi?a Duri? >>>>> >>>> >>> -- >>> Dragi?a Duri? >>> >> > -- > Dragi?a Duri? From dragisha at m3w.org Thu Mar 13 23:09:38 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 13 Mar 2008 23:09:38 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <5649BA54-5B2F-423C-AEF7-65FF09CF3763@cs.purdue.edu> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> <1205425734.32152.144.camel@hias.m3w.org> <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> <1205430199.32152.152.camel@hias.m3w.org> <1205444774.32152.158.camel@hias.m3w.org> <5649BA54-5B2F-423C-AEF7-65FF09CF3763@cs.purdue.edu> Message-ID: <1205446178.32152.162.camel@hias.m3w.org> When object becomes unreachable, and it sits on page with other reachable objects, it's pinned? What I asked is... As garbage collector moves objects around, would it pack reachable objects over unreachable ones in these pages? Thus kicking them out of pages where they're pinned? dd On Thu, 2008-03-13 at 18:03 -0400, Tony Hosking wrote: > I'm not sure I understand what you mean? > > On Mar 13, 2008, at 5:46 PM, Dragi?a Duri? wrote: > > > Are objects on pages prepacked/moved? > > > > On Thu, 2008-03-13 at 17:41 -0400, Tony Hosking wrote: > >> The thing about conservatism is that even if you NIL *all* references > >> to an object that just happens to lie on a pinned page, that object > >> (and objects reachable from it) will be retained until the page is no > >> longer pinned. > >> > >> Antony Hosking | Associate Professor | Computer Science | Purdue > >> University > >> 305 N. University Street | West Lafayette | IN 47907 | USA > >> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >> > >> > >> > >> > >> > >> On Mar 13, 2008, at 1:43 PM, Dragi?a Duri? wrote: > >> > >>> I understand transitiveness :), but I've made big effort to escape > >>> that. > >>> > >>> Lifetime of thread equals lifetime of Client.T instance, and that > >>> instance is kept in thread's closure, and also in local variable. > >>> Both > >>> are set to NIL before thread termination - all terminations and > >>> joins > >>> happen and are counted and shown. > >>> > >>> Pointers to Message objects are kept in two places. Hub.T keeps > >>> pointer > >>> to list tail, and each Client.T contains pointer to it's current > >>> element. That pointer is forwarded as Message's are added to end. No > >>> temporary variable, not even WITH substitution is used. And these > >>> threads are ones dying with their Client.T's. > >>> > >>> "head" is new mechanism used to unlink all elements with id less > >>> than > >>> minimal id used in active Client.T instances. Unlinked totally and > >>> all > >>> as one free for garbage collector to finish them. > >>> > >>> I see no transition. > >>> > >>> dd > >>> > >>> On Thu, 2008-03-13 at 12:45 -0400, Tony Hosking wrote: > >>>> Realizing that conservative roots collectors like ours can retain > >>>> data > >>>> unnecessarily. If a thread stack refers to a page that holds a > >>>> reference that (transitively) refers to some otherwise dead object > >>>> then that object will be retained. We say that the page is > >>>> pinned. > >>>> Smartening up the collector to ignore dead objects in pinned pages > >>>> is > >>>> possible, but not currently implemented. > >>>> > >>>> Antony Hosking | Associate Professor | Computer Science | Purdue > >>>> University > >>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On Mar 13, 2008, at 12:28 PM, Dragi?a Duri? wrote: > >>>> > >>>>> [17:25:19] 117 users, 1 ops and 1 unresolved; minId = > >>>>> 18617; thrs = 120 > >>>>> [17:25:19] started = 663; cleaned = 452; nilled = 543; > >>>>> joined = 543 > >>>>> [17:25:19] Dangling thrs-a-b-n = 1 > >>>>> [17:25:19] Live queue end is 18621 (head = 18616), > >>>>> maxClean = 18611, cleaned = 16583 > >>>>> > >>>>> About one hour uptime, after I added RTCollector.Collect() call > >>>>> every 20 > >>>>> seconds. > >>>>> > >>>>> dd > >>>>> > >>>>> On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking wrote: > >>>>>> Yes. RTCollectorSRC.FinishCollection; > >>>>>> RTCollectorSRC.StartCollection. This will finish the current > >>>>>> (possibly minor) collection, and start a major collection. > >>>>>> The > >>>>>> major > >>>>>> collection must finish before you can guarantee garbage has > >>>>>> been > >>>>>> freed, so you may want to RTCollectorSRC.FinishCollection. > >>>>>> This > >>>>>> sequence is packaged as RTCollector.Collect. > >>>>>> > >>>>>> Antony Hosking | Associate Professor | Computer Science | > >>>>>> Purdue > >>>>>> University > >>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? wrote: > >>>>>> > >>>>>>> Can I force "major collection" ? > >>>>>>> > >>>>>>> dd > >>>>>>> > >>>>>>> On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: > >>>>>>>> Indeed! Can you devise a testcase? Note that the > >>>>>>>> generational > >>>>>>>> collector will retain old objects for some time until a > >>>>>>>> major > >>>>>>>> collection occurs, > >>>>>>>> > >>>>>>>> Antony Hosking | Associate Professor | Computer Science | > >>>>>>>> Purdue > >>>>>>>> University > >>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: > >>>>>>>> > >>>>>>>>> And not only that, their stackbase is also set to NIL, > >>>>>>>>> meaning > >>>>>>>>> their > >>>>>>>>> stack is not regarded in any way during future > >>>>>>>>> collections - > >>>>>>>>> meaning > >>>>>>>>> all > >>>>>>>>> local variables are forgotten once apply method returns. > >>>>>>>>> > >>>>>>>>> That way, "spuriousness" is not an issue, once thread > >>>>>>>>> returns? > >>>>>>>>> > >>>>>>>>> dd > >>>>>>>>> > >>>>>>>>> On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: > >>>>>>>>>> They should get unlinked from the global ring. > >>>>>>>>>> > >>>>>>>>>> Antony Hosking | Associate Professor | Computer > >>>>>>>>>> Science | > >>>>>>>>>> Purdue > >>>>>>>>>> University > >>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | > >>>>>>>>>> USA > >>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: > >>>>>>>>>> > >>>>>>>>>>> What happens to thread stacks once threads are > >>>>>>>>>>> Join-ed? > >>>>>>>>>>> > >>>>>>>>>>> dd > >>>>>>>>>>> > >>>>>>>>>>> On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking > >>>>>>>>>>> wrote: > >>>>>>>>>>>> This is probably a result of conservatism for > >>>>>>>>>>>> references > >>>>>>>>>>>> from > >>>>>>>>>>>> thread > >>>>>>>>>>>> stacks which results in spurious retention. It > >>>>>>>>>>>> may be > >>>>>>>>>>>> necessary > >>>>>>>>>>>> to be > >>>>>>>>>>>> more careful about what references are held on the > >>>>>>>>>>>> stack > >>>>>>>>>>>> in > >>>>>>>>>>>> the > >>>>>>>>>>>> threads implementation (and elsewhere in your > >>>>>>>>>>>> code). > >>>>>>>>>>>> Until we > >>>>>>>>>>>> diagnose the place where objects are being > >>>>>>>>>>>> retained it > >>>>>>>>>>>> will be > >>>>>>>>>>>> hard to > >>>>>>>>>>>> pinpoint. You should realize that weak refs are a > >>>>>>>>>>>> problem > >>>>>>>>>>>> in > >>>>>>>>>>>> many > >>>>>>>>>>>> language implementations, not just Modula-3. I > >>>>>>>>>>>> look > >>>>>>>>>>>> forward > >>>>>>>>>>>> to > >>>>>>>>>>>> hearing more definitively from you. > >>>>>>>>>>>> > >>>>>>>>>>>> Antony Hosking | Associate Professor | Computer > >>>>>>>>>>>> Science | > >>>>>>>>>>>> Purdue > >>>>>>>>>>>> University > >>>>>>>>>>>> 305 N. University Street | West Lafayette | IN > >>>>>>>>>>>> 47907 | > >>>>>>>>>>>> USA > >>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> Problem in my case can be thoroughness of GC... > >>>>>>>>>>>>> Because > >>>>>>>>>>>>> it's > >>>>>>>>>>>>> linked > >>>>>>>>>>>>> list of WeakRef-ed objects, it's always first > >>>>>>>>>>>>> element in > >>>>>>>>>>>>> list to > >>>>>>>>>>>>> become > >>>>>>>>>>>>> unreachable first - all others are reachable at > >>>>>>>>>>>>> least > >>>>>>>>>>>>> through > >>>>>>>>>>>>> previous > >>>>>>>>>>>>> (in list) WeakRef-ed object. So, behaviour I > >>>>>>>>>>>>> have > >>>>>>>>>>>>> observed > >>>>>>>>>>>>> can > >>>>>>>>>>>>> be > >>>>>>>>>>>>> because of some thousands of freeable objects, > >>>>>>>>>>>>> that > >>>>>>>>>>>>> one > >>>>>>>>>>>>> is > >>>>>>>>>>>>> missed > >>>>>>>>>>>>> too > >>>>>>>>>>>>> often. > >>>>>>>>>>>>> > >>>>>>>>>>>>> I have other object being cleaned and counted... > >>>>>>>>>>>>> There > >>>>>>>>>>>>> also > >>>>>>>>>>>>> cleanup > >>>>>>>>>>>>> invocation looks like it's lagging too much. > >>>>>>>>>>>>> I'll > >>>>>>>>>>>>> come > >>>>>>>>>>>>> with > >>>>>>>>>>>>> numbers > >>>>>>>>>>>>> after I have more tests. > >>>>>>>>>>>>> > >>>>>>>>>>>>> dd > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> Sounds like we need some sort of testcase. > >>>>>>>>>>>>>> Would > >>>>>>>>>>>>>> you > >>>>>>>>>>>>>> be > >>>>>>>>>>>>>> able > >>>>>>>>>>>>>> to > >>>>>>>>>>>>>> devise one? It will be hard to make it > >>>>>>>>>>>>>> deterministic, > >>>>>>>>>>>>>> but > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> least we > >>>>>>>>>>>>>> should see a non-zero cleanup count. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Antony Hosking | Associate Professor | > >>>>>>>>>>>>>> Computer > >>>>>>>>>>>>>> Science | > >>>>>>>>>>>>>> Purdue > >>>>>>>>>>>>>> University > >>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN > >>>>>>>>>>>>>> 47907 | > >>>>>>>>>>>>>> USA > >>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 > >>>>>>>>>>>>>> 5484 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? > >>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I have single linked list that I use to send > >>>>>>>>>>>>>>> messages to > >>>>>>>>>>>>>>> many > >>>>>>>>>>>>>>> threads. > >>>>>>>>>>>>>>> It's end is locked for addition, and current > >>>>>>>>>>>>>>> end > >>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>> given to > >>>>>>>>>>>>>>> each > >>>>>>>>>>>>>>> new > >>>>>>>>>>>>>>> client connecting. This way, all client are > >>>>>>>>>>>>>>> going > >>>>>>>>>>>>>>> towards > >>>>>>>>>>>>>>> end of > >>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>> list so it's head becomes unreferenced and > >>>>>>>>>>>>>>> goes > >>>>>>>>>>>>>>> away > >>>>>>>>>>>>>>> by > >>>>>>>>>>>>>>> GC. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Except it does not. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I've added WeakRef cleanup and sequential > >>>>>>>>>>>>>>> id's > >>>>>>>>>>>>>>> so I > >>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>> record > >>>>>>>>>>>>>>> maximum > >>>>>>>>>>>>>>> freed id for checking. No single cleanup > >>>>>>>>>>>>>>> happens. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I'll cross check my code and count my > >>>>>>>>>>>>>>> references > >>>>>>>>>>>>>>> from > >>>>>>>>>>>>>>> application > >>>>>>>>>>>>>>> side, > >>>>>>>>>>>>>>> but maybe someone else has similar problems? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> dd > >>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>> Dragi?a Duri? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> -- > >>>>>>>>>>>>> Dragi?a Duri? > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> Dragi?a Duri? > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> Dragi?a Duri? > >>>>>>>>> > >>>>>>>> > >>>>>>> -- > >>>>>>> Dragi?a Duri? > >>>>>>> > >>>>>> > >>>>> -- > >>>>> Dragi?a Duri? > >>>>> > >>>> > >>> -- > >>> Dragi?a Duri? > >>> > >> > > -- > > Dragi?a Duri? > -- Dragi?a Duri? From hosking at cs.purdue.edu Thu Mar 13 23:15:59 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 13 Mar 2008 18:15:59 -0400 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <1205446178.32152.162.camel@hias.m3w.org> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> <1205425734.32152.144.camel@hias.m3w.org> <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> <1205430199.32152.152.camel@hias.m3w.org> <1205444774.32152.158.camel@hias.m3w.org> <5649BA54-5B2F-423C-AEF7-65FF09CF3763@cs.purdue.edu> <1205446178.32152.162.camel@hias.m3w.org> Message-ID: <4DD89DC7-FA3C-4B61-A2A1-5C8488CF3A3B@cs.purdue.edu> No, when a page is pinned because one of its objects is directly referenced from a thread stack then all objects on that page, both live and dead, are retained. The collector does not move objects to or from pinned pages. The objects on the page can only be freed when the page is no longer pinned by a direct reference from any thread stack. Moral of the story is to NIL references aggressively in local variables. On Mar 13, 2008, at 6:09 PM, Dragi?a Duri? wrote: > When object becomes unreachable, and it sits on page with other > reachable objects, it's pinned? > > What I asked is... As garbage collector moves objects around, would > it > pack reachable objects over unreachable ones in these pages? Thus > kicking them out of pages where they're pinned? > > dd > > On Thu, 2008-03-13 at 18:03 -0400, Tony Hosking wrote: >> I'm not sure I understand what you mean? >> >> On Mar 13, 2008, at 5:46 PM, Dragi?a Duri? wrote: >> >>> Are objects on pages prepacked/moved? >>> >>> On Thu, 2008-03-13 at 17:41 -0400, Tony Hosking wrote: >>>> The thing about conservatism is that even if you NIL *all* >>>> references >>>> to an object that just happens to lie on a pinned page, that object >>>> (and objects reachable from it) will be retained until the page >>>> is no >>>> longer pinned. >>>> >>>> Antony Hosking | Associate Professor | Computer Science | Purdue >>>> University >>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>> >>>> >>>> >>>> >>>> >>>> On Mar 13, 2008, at 1:43 PM, Dragi?a Duri? wrote: >>>> >>>>> I understand transitiveness :), but I've made big effort to escape >>>>> that. >>>>> >>>>> Lifetime of thread equals lifetime of Client.T instance, and that >>>>> instance is kept in thread's closure, and also in local variable. >>>>> Both >>>>> are set to NIL before thread termination - all terminations and >>>>> joins >>>>> happen and are counted and shown. >>>>> >>>>> Pointers to Message objects are kept in two places. Hub.T keeps >>>>> pointer >>>>> to list tail, and each Client.T contains pointer to it's current >>>>> element. That pointer is forwarded as Message's are added to >>>>> end. No >>>>> temporary variable, not even WITH substitution is used. And these >>>>> threads are ones dying with their Client.T's. >>>>> >>>>> "head" is new mechanism used to unlink all elements with id less >>>>> than >>>>> minimal id used in active Client.T instances. Unlinked totally and >>>>> all >>>>> as one free for garbage collector to finish them. >>>>> >>>>> I see no transition. >>>>> >>>>> dd >>>>> >>>>> On Thu, 2008-03-13 at 12:45 -0400, Tony Hosking wrote: >>>>>> Realizing that conservative roots collectors like ours can retain >>>>>> data >>>>>> unnecessarily. If a thread stack refers to a page that holds a >>>>>> reference that (transitively) refers to some otherwise dead >>>>>> object >>>>>> then that object will be retained. We say that the page is >>>>>> pinned. >>>>>> Smartening up the collector to ignore dead objects in pinned >>>>>> pages >>>>>> is >>>>>> possible, but not currently implemented. >>>>>> >>>>>> Antony Hosking | Associate Professor | Computer Science | Purdue >>>>>> University >>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mar 13, 2008, at 12:28 PM, Dragi?a Duri? wrote: >>>>>> >>>>>>> [17:25:19] 117 users, 1 ops and 1 unresolved; minId = >>>>>>> 18617; thrs = 120 >>>>>>> [17:25:19] started = 663; cleaned = 452; nilled = 543; >>>>>>> joined = 543 >>>>>>> [17:25:19] Dangling thrs-a-b-n = 1 >>>>>>> [17:25:19] Live queue end is 18621 (head = 18616), >>>>>>> maxClean = 18611, cleaned = 16583 >>>>>>> >>>>>>> About one hour uptime, after I added RTCollector.Collect() call >>>>>>> every 20 >>>>>>> seconds. >>>>>>> >>>>>>> dd >>>>>>> >>>>>>> On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking wrote: >>>>>>>> Yes. RTCollectorSRC.FinishCollection; >>>>>>>> RTCollectorSRC.StartCollection. This will finish the current >>>>>>>> (possibly minor) collection, and start a major collection. >>>>>>>> The >>>>>>>> major >>>>>>>> collection must finish before you can guarantee garbage has >>>>>>>> been >>>>>>>> freed, so you may want to RTCollectorSRC.FinishCollection. >>>>>>>> This >>>>>>>> sequence is packaged as RTCollector.Collect. >>>>>>>> >>>>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>>>> Purdue >>>>>>>> University >>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? wrote: >>>>>>>> >>>>>>>>> Can I force "major collection" ? >>>>>>>>> >>>>>>>>> dd >>>>>>>>> >>>>>>>>> On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: >>>>>>>>>> Indeed! Can you devise a testcase? Note that the >>>>>>>>>> generational >>>>>>>>>> collector will retain old objects for some time until a >>>>>>>>>> major >>>>>>>>>> collection occurs, >>>>>>>>>> >>>>>>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>>>>>> Purdue >>>>>>>>>> University >>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: >>>>>>>>>> >>>>>>>>>>> And not only that, their stackbase is also set to NIL, >>>>>>>>>>> meaning >>>>>>>>>>> their >>>>>>>>>>> stack is not regarded in any way during future >>>>>>>>>>> collections - >>>>>>>>>>> meaning >>>>>>>>>>> all >>>>>>>>>>> local variables are forgotten once apply method returns. >>>>>>>>>>> >>>>>>>>>>> That way, "spuriousness" is not an issue, once thread >>>>>>>>>>> returns? >>>>>>>>>>> >>>>>>>>>>> dd >>>>>>>>>>> >>>>>>>>>>> On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: >>>>>>>>>>>> They should get unlinked from the global ring. >>>>>>>>>>>> >>>>>>>>>>>> Antony Hosking | Associate Professor | Computer >>>>>>>>>>>> Science | >>>>>>>>>>>> Purdue >>>>>>>>>>>> University >>>>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | >>>>>>>>>>>> USA >>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: >>>>>>>>>>>> >>>>>>>>>>>>> What happens to thread stacks once threads are >>>>>>>>>>>>> Join-ed? >>>>>>>>>>>>> >>>>>>>>>>>>> dd >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> This is probably a result of conservatism for >>>>>>>>>>>>>> references >>>>>>>>>>>>>> from >>>>>>>>>>>>>> thread >>>>>>>>>>>>>> stacks which results in spurious retention. It >>>>>>>>>>>>>> may be >>>>>>>>>>>>>> necessary >>>>>>>>>>>>>> to be >>>>>>>>>>>>>> more careful about what references are held on the >>>>>>>>>>>>>> stack >>>>>>>>>>>>>> in >>>>>>>>>>>>>> the >>>>>>>>>>>>>> threads implementation (and elsewhere in your >>>>>>>>>>>>>> code). >>>>>>>>>>>>>> Until we >>>>>>>>>>>>>> diagnose the place where objects are being >>>>>>>>>>>>>> retained it >>>>>>>>>>>>>> will be >>>>>>>>>>>>>> hard to >>>>>>>>>>>>>> pinpoint. You should realize that weak refs are a >>>>>>>>>>>>>> problem >>>>>>>>>>>>>> in >>>>>>>>>>>>>> many >>>>>>>>>>>>>> language implementations, not just Modula-3. I >>>>>>>>>>>>>> look >>>>>>>>>>>>>> forward >>>>>>>>>>>>>> to >>>>>>>>>>>>>> hearing more definitively from you. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Antony Hosking | Associate Professor | Computer >>>>>>>>>>>>>> Science | >>>>>>>>>>>>>> Purdue >>>>>>>>>>>>>> University >>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN >>>>>>>>>>>>>> 47907 | >>>>>>>>>>>>>> USA >>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Problem in my case can be thoroughness of GC... >>>>>>>>>>>>>>> Because >>>>>>>>>>>>>>> it's >>>>>>>>>>>>>>> linked >>>>>>>>>>>>>>> list of WeakRef-ed objects, it's always first >>>>>>>>>>>>>>> element in >>>>>>>>>>>>>>> list to >>>>>>>>>>>>>>> become >>>>>>>>>>>>>>> unreachable first - all others are reachable at >>>>>>>>>>>>>>> least >>>>>>>>>>>>>>> through >>>>>>>>>>>>>>> previous >>>>>>>>>>>>>>> (in list) WeakRef-ed object. So, behaviour I >>>>>>>>>>>>>>> have >>>>>>>>>>>>>>> observed >>>>>>>>>>>>>>> can >>>>>>>>>>>>>>> be >>>>>>>>>>>>>>> because of some thousands of freeable objects, >>>>>>>>>>>>>>> that >>>>>>>>>>>>>>> one >>>>>>>>>>>>>>> is >>>>>>>>>>>>>>> missed >>>>>>>>>>>>>>> too >>>>>>>>>>>>>>> often. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I have other object being cleaned and counted... >>>>>>>>>>>>>>> There >>>>>>>>>>>>>>> also >>>>>>>>>>>>>>> cleanup >>>>>>>>>>>>>>> invocation looks like it's lagging too much. >>>>>>>>>>>>>>> I'll >>>>>>>>>>>>>>> come >>>>>>>>>>>>>>> with >>>>>>>>>>>>>>> numbers >>>>>>>>>>>>>>> after I have more tests. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> dd >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> Sounds like we need some sort of testcase. >>>>>>>>>>>>>>>> Would >>>>>>>>>>>>>>>> you >>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>> able >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>> devise one? It will be hard to make it >>>>>>>>>>>>>>>> deterministic, >>>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> least we >>>>>>>>>>>>>>>> should see a non-zero cleanup count. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Antony Hosking | Associate Professor | >>>>>>>>>>>>>>>> Computer >>>>>>>>>>>>>>>> Science | >>>>>>>>>>>>>>>> Purdue >>>>>>>>>>>>>>>> University >>>>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN >>>>>>>>>>>>>>>> 47907 | >>>>>>>>>>>>>>>> USA >>>>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 >>>>>>>>>>>>>>>> 5484 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I have single linked list that I use to send >>>>>>>>>>>>>>>>> messages to >>>>>>>>>>>>>>>>> many >>>>>>>>>>>>>>>>> threads. >>>>>>>>>>>>>>>>> It's end is locked for addition, and current >>>>>>>>>>>>>>>>> end >>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>> given to >>>>>>>>>>>>>>>>> each >>>>>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>> client connecting. This way, all client are >>>>>>>>>>>>>>>>> going >>>>>>>>>>>>>>>>> towards >>>>>>>>>>>>>>>>> end of >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> list so it's head becomes unreferenced and >>>>>>>>>>>>>>>>> goes >>>>>>>>>>>>>>>>> away >>>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>> GC. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Except it does not. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I've added WeakRef cleanup and sequential >>>>>>>>>>>>>>>>> id's >>>>>>>>>>>>>>>>> so I >>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>> record >>>>>>>>>>>>>>>>> maximum >>>>>>>>>>>>>>>>> freed id for checking. No single cleanup >>>>>>>>>>>>>>>>> happens. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I'll cross check my code and count my >>>>>>>>>>>>>>>>> references >>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>> application >>>>>>>>>>>>>>>>> side, >>>>>>>>>>>>>>>>> but maybe someone else has similar problems? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> dd >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Dragi?a Duri? >>>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> Dragi?a Duri? >>>>>>> >>>>>> >>>>> -- >>>>> Dragi?a Duri? >>>>> >>>> >>> -- >>> Dragi?a Duri? >> > -- > Dragi?a Duri? From dragisha at m3w.org Thu Mar 13 23:21:42 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 13 Mar 2008 23:21:42 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <4DD89DC7-FA3C-4B61-A2A1-5C8488CF3A3B@cs.purdue.edu> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> <1205425734.32152.144.camel@hias.m3w.org> <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> <1205430199.32152.152.camel@hias.m3w.org> <1205444774.32152.158.camel@hias.m3w.org> <5649BA54-5B2F-423C-AEF7-65FF09CF3763@cs.purdue.edu> <1205446178.32152.162.camel@hias.m3w.org> <4DD89DC7-FA3C-4B61-A2A1-5C8488CF3A3B@cs.purdue.edu> Message-ID: <1205446902.32152.168.camel@hias.m3w.org> I see... What bugs me now is a fact about my flow... I am already doing this, and there is always few thousand objects uncleaned from of type Message... I'll follow this through, to crosscheck everything again. What is nice, after I made this agressive RTCollector.Collect "policy", my memory footprint remains same for many hours... It was swelling before. Thanks, dd On Thu, 2008-03-13 at 18:15 -0400, Tony Hosking wrote: > No, when a page is pinned because one of its objects is directly > referenced from a thread stack then all objects on that page, both > live and dead, are retained. The collector does not move objects to > or from pinned pages. The objects on the page can only be freed when > the page is no longer pinned by a direct reference from any thread > stack. Moral of the story is to NIL references aggressively in local > variables. > > On Mar 13, 2008, at 6:09 PM, Dragi?a Duri? wrote: > > > When object becomes unreachable, and it sits on page with other > > reachable objects, it's pinned? > > > > What I asked is... As garbage collector moves objects around, would > > it > > pack reachable objects over unreachable ones in these pages? Thus > > kicking them out of pages where they're pinned? > > > > dd > > > > On Thu, 2008-03-13 at 18:03 -0400, Tony Hosking wrote: > >> I'm not sure I understand what you mean? > >> > >> On Mar 13, 2008, at 5:46 PM, Dragi?a Duri? wrote: > >> > >>> Are objects on pages prepacked/moved? > >>> > >>> On Thu, 2008-03-13 at 17:41 -0400, Tony Hosking wrote: > >>>> The thing about conservatism is that even if you NIL *all* > >>>> references > >>>> to an object that just happens to lie on a pinned page, that object > >>>> (and objects reachable from it) will be retained until the page > >>>> is no > >>>> longer pinned. > >>>> > >>>> Antony Hosking | Associate Professor | Computer Science | Purdue > >>>> University > >>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On Mar 13, 2008, at 1:43 PM, Dragi?a Duri? wrote: > >>>> > >>>>> I understand transitiveness :), but I've made big effort to escape > >>>>> that. > >>>>> > >>>>> Lifetime of thread equals lifetime of Client.T instance, and that > >>>>> instance is kept in thread's closure, and also in local variable. > >>>>> Both > >>>>> are set to NIL before thread termination - all terminations and > >>>>> joins > >>>>> happen and are counted and shown. > >>>>> > >>>>> Pointers to Message objects are kept in two places. Hub.T keeps > >>>>> pointer > >>>>> to list tail, and each Client.T contains pointer to it's current > >>>>> element. That pointer is forwarded as Message's are added to > >>>>> end. No > >>>>> temporary variable, not even WITH substitution is used. And these > >>>>> threads are ones dying with their Client.T's. > >>>>> > >>>>> "head" is new mechanism used to unlink all elements with id less > >>>>> than > >>>>> minimal id used in active Client.T instances. Unlinked totally and > >>>>> all > >>>>> as one free for garbage collector to finish them. > >>>>> > >>>>> I see no transition. > >>>>> > >>>>> dd > >>>>> > >>>>> On Thu, 2008-03-13 at 12:45 -0400, Tony Hosking wrote: > >>>>>> Realizing that conservative roots collectors like ours can retain > >>>>>> data > >>>>>> unnecessarily. If a thread stack refers to a page that holds a > >>>>>> reference that (transitively) refers to some otherwise dead > >>>>>> object > >>>>>> then that object will be retained. We say that the page is > >>>>>> pinned. > >>>>>> Smartening up the collector to ignore dead objects in pinned > >>>>>> pages > >>>>>> is > >>>>>> possible, but not currently implemented. > >>>>>> > >>>>>> Antony Hosking | Associate Professor | Computer Science | Purdue > >>>>>> University > >>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> On Mar 13, 2008, at 12:28 PM, Dragi?a Duri? wrote: > >>>>>> > >>>>>>> [17:25:19] 117 users, 1 ops and 1 unresolved; minId = > >>>>>>> 18617; thrs = 120 > >>>>>>> [17:25:19] started = 663; cleaned = 452; nilled = 543; > >>>>>>> joined = 543 > >>>>>>> [17:25:19] Dangling thrs-a-b-n = 1 > >>>>>>> [17:25:19] Live queue end is 18621 (head = 18616), > >>>>>>> maxClean = 18611, cleaned = 16583 > >>>>>>> > >>>>>>> About one hour uptime, after I added RTCollector.Collect() call > >>>>>>> every 20 > >>>>>>> seconds. > >>>>>>> > >>>>>>> dd > >>>>>>> > >>>>>>> On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking wrote: > >>>>>>>> Yes. RTCollectorSRC.FinishCollection; > >>>>>>>> RTCollectorSRC.StartCollection. This will finish the current > >>>>>>>> (possibly minor) collection, and start a major collection. > >>>>>>>> The > >>>>>>>> major > >>>>>>>> collection must finish before you can guarantee garbage has > >>>>>>>> been > >>>>>>>> freed, so you may want to RTCollectorSRC.FinishCollection. > >>>>>>>> This > >>>>>>>> sequence is packaged as RTCollector.Collect. > >>>>>>>> > >>>>>>>> Antony Hosking | Associate Professor | Computer Science | > >>>>>>>> Purdue > >>>>>>>> University > >>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? wrote: > >>>>>>>> > >>>>>>>>> Can I force "major collection" ? > >>>>>>>>> > >>>>>>>>> dd > >>>>>>>>> > >>>>>>>>> On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: > >>>>>>>>>> Indeed! Can you devise a testcase? Note that the > >>>>>>>>>> generational > >>>>>>>>>> collector will retain old objects for some time until a > >>>>>>>>>> major > >>>>>>>>>> collection occurs, > >>>>>>>>>> > >>>>>>>>>> Antony Hosking | Associate Professor | Computer Science | > >>>>>>>>>> Purdue > >>>>>>>>>> University > >>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: > >>>>>>>>>> > >>>>>>>>>>> And not only that, their stackbase is also set to NIL, > >>>>>>>>>>> meaning > >>>>>>>>>>> their > >>>>>>>>>>> stack is not regarded in any way during future > >>>>>>>>>>> collections - > >>>>>>>>>>> meaning > >>>>>>>>>>> all > >>>>>>>>>>> local variables are forgotten once apply method returns. > >>>>>>>>>>> > >>>>>>>>>>> That way, "spuriousness" is not an issue, once thread > >>>>>>>>>>> returns? > >>>>>>>>>>> > >>>>>>>>>>> dd > >>>>>>>>>>> > >>>>>>>>>>> On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: > >>>>>>>>>>>> They should get unlinked from the global ring. > >>>>>>>>>>>> > >>>>>>>>>>>> Antony Hosking | Associate Professor | Computer > >>>>>>>>>>>> Science | > >>>>>>>>>>>> Purdue > >>>>>>>>>>>> University > >>>>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | > >>>>>>>>>>>> USA > >>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> What happens to thread stacks once threads are > >>>>>>>>>>>>> Join-ed? > >>>>>>>>>>>>> > >>>>>>>>>>>>> dd > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> This is probably a result of conservatism for > >>>>>>>>>>>>>> references > >>>>>>>>>>>>>> from > >>>>>>>>>>>>>> thread > >>>>>>>>>>>>>> stacks which results in spurious retention. It > >>>>>>>>>>>>>> may be > >>>>>>>>>>>>>> necessary > >>>>>>>>>>>>>> to be > >>>>>>>>>>>>>> more careful about what references are held on the > >>>>>>>>>>>>>> stack > >>>>>>>>>>>>>> in > >>>>>>>>>>>>>> the > >>>>>>>>>>>>>> threads implementation (and elsewhere in your > >>>>>>>>>>>>>> code). > >>>>>>>>>>>>>> Until we > >>>>>>>>>>>>>> diagnose the place where objects are being > >>>>>>>>>>>>>> retained it > >>>>>>>>>>>>>> will be > >>>>>>>>>>>>>> hard to > >>>>>>>>>>>>>> pinpoint. You should realize that weak refs are a > >>>>>>>>>>>>>> problem > >>>>>>>>>>>>>> in > >>>>>>>>>>>>>> many > >>>>>>>>>>>>>> language implementations, not just Modula-3. I > >>>>>>>>>>>>>> look > >>>>>>>>>>>>>> forward > >>>>>>>>>>>>>> to > >>>>>>>>>>>>>> hearing more definitively from you. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Antony Hosking | Associate Professor | Computer > >>>>>>>>>>>>>> Science | > >>>>>>>>>>>>>> Purdue > >>>>>>>>>>>>>> University > >>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN > >>>>>>>>>>>>>> 47907 | > >>>>>>>>>>>>>> USA > >>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Problem in my case can be thoroughness of GC... > >>>>>>>>>>>>>>> Because > >>>>>>>>>>>>>>> it's > >>>>>>>>>>>>>>> linked > >>>>>>>>>>>>>>> list of WeakRef-ed objects, it's always first > >>>>>>>>>>>>>>> element in > >>>>>>>>>>>>>>> list to > >>>>>>>>>>>>>>> become > >>>>>>>>>>>>>>> unreachable first - all others are reachable at > >>>>>>>>>>>>>>> least > >>>>>>>>>>>>>>> through > >>>>>>>>>>>>>>> previous > >>>>>>>>>>>>>>> (in list) WeakRef-ed object. So, behaviour I > >>>>>>>>>>>>>>> have > >>>>>>>>>>>>>>> observed > >>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>> because of some thousands of freeable objects, > >>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>> one > >>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>> missed > >>>>>>>>>>>>>>> too > >>>>>>>>>>>>>>> often. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I have other object being cleaned and counted... > >>>>>>>>>>>>>>> There > >>>>>>>>>>>>>>> also > >>>>>>>>>>>>>>> cleanup > >>>>>>>>>>>>>>> invocation looks like it's lagging too much. > >>>>>>>>>>>>>>> I'll > >>>>>>>>>>>>>>> come > >>>>>>>>>>>>>>> with > >>>>>>>>>>>>>>> numbers > >>>>>>>>>>>>>>> after I have more tests. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> dd > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> Sounds like we need some sort of testcase. > >>>>>>>>>>>>>>>> Would > >>>>>>>>>>>>>>>> you > >>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>> able > >>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>> devise one? It will be hard to make it > >>>>>>>>>>>>>>>> deterministic, > >>>>>>>>>>>>>>>> but > >>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>> least we > >>>>>>>>>>>>>>>> should see a non-zero cleanup count. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Antony Hosking | Associate Professor | > >>>>>>>>>>>>>>>> Computer > >>>>>>>>>>>>>>>> Science | > >>>>>>>>>>>>>>>> Purdue > >>>>>>>>>>>>>>>> University > >>>>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN > >>>>>>>>>>>>>>>> 47907 | > >>>>>>>>>>>>>>>> USA > >>>>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 > >>>>>>>>>>>>>>>> 5484 > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? > >>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I have single linked list that I use to send > >>>>>>>>>>>>>>>>> messages to > >>>>>>>>>>>>>>>>> many > >>>>>>>>>>>>>>>>> threads. > >>>>>>>>>>>>>>>>> It's end is locked for addition, and current > >>>>>>>>>>>>>>>>> end > >>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>> given to > >>>>>>>>>>>>>>>>> each > >>>>>>>>>>>>>>>>> new > >>>>>>>>>>>>>>>>> client connecting. This way, all client are > >>>>>>>>>>>>>>>>> going > >>>>>>>>>>>>>>>>> towards > >>>>>>>>>>>>>>>>> end of > >>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>> list so it's head becomes unreferenced and > >>>>>>>>>>>>>>>>> goes > >>>>>>>>>>>>>>>>> away > >>>>>>>>>>>>>>>>> by > >>>>>>>>>>>>>>>>> GC. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Except it does not. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I've added WeakRef cleanup and sequential > >>>>>>>>>>>>>>>>> id's > >>>>>>>>>>>>>>>>> so I > >>>>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>> record > >>>>>>>>>>>>>>>>> maximum > >>>>>>>>>>>>>>>>> freed id for checking. No single cleanup > >>>>>>>>>>>>>>>>> happens. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I'll cross check my code and count my > >>>>>>>>>>>>>>>>> references > >>>>>>>>>>>>>>>>> from > >>>>>>>>>>>>>>>>> application > >>>>>>>>>>>>>>>>> side, > >>>>>>>>>>>>>>>>> but maybe someone else has similar problems? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> dd > >>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>> Dragi?a Duri? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>> Dragi?a Duri? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> -- > >>>>>>>>>>>>> Dragi?a Duri? > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> Dragi?a Duri? > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> Dragi?a Duri? > >>>>>>>>> > >>>>>>>> > >>>>>>> -- > >>>>>>> Dragi?a Duri? > >>>>>>> > >>>>>> > >>>>> -- > >>>>> Dragi?a Duri? > >>>>> > >>>> > >>> -- > >>> Dragi?a Duri? > >> > > -- > > Dragi?a Duri? > -- Dragi?a Duri? From hosking at cs.purdue.edu Thu Mar 13 23:35:00 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 13 Mar 2008 18:35:00 -0400 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <1205446902.32152.168.camel@hias.m3w.org> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> <1205425734.32152.144.camel@hias.m3w.org> <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> <1205430199.32152.152.camel@hias.m3w.org> <1205444774.32152.158.camel@hias.m3w.org> <5649BA54-5B2F-423C-AEF7-65FF09CF3763@cs.purdue.edu> <1205446178.32152.162.camel@hias.m3w.org> <4DD89DC7-FA3C-4B61-A2A1-5C8488CF3A3B@cs.purdue.edu> <1205446902.32152.168.camel@hias.m3w.org> Message-ID: <2EA2C202-F053-4AE8-BA60-E51A1A339C6B@cs.purdue.edu> Ah, the fact that you needed to be aggressive suggests that conservatism may be the culprit. I keep meaning one day to make sure that we do not can objects on pinned pages unless we know that object is definitely alive. On Mar 13, 2008, at 6:21 PM, Dragi?a Duri? wrote: > I see... What bugs me now is a fact about my flow... I am already > doing > this, and there is always few thousand objects uncleaned from of type > Message... I'll follow this through, to crosscheck everything again. > > What is nice, after I made this agressive RTCollector.Collect > "policy", > my memory footprint remains same for many hours... It was swelling > before. > > Thanks, > dd > > On Thu, 2008-03-13 at 18:15 -0400, Tony Hosking wrote: >> No, when a page is pinned because one of its objects is directly >> referenced from a thread stack then all objects on that page, both >> live and dead, are retained. The collector does not move objects to >> or from pinned pages. The objects on the page can only be freed when >> the page is no longer pinned by a direct reference from any thread >> stack. Moral of the story is to NIL references aggressively in local >> variables. >> >> On Mar 13, 2008, at 6:09 PM, Dragi?a Duri? wrote: >> >>> When object becomes unreachable, and it sits on page with other >>> reachable objects, it's pinned? >>> >>> What I asked is... As garbage collector moves objects around, would >>> it >>> pack reachable objects over unreachable ones in these pages? Thus >>> kicking them out of pages where they're pinned? >>> >>> dd >>> >>> On Thu, 2008-03-13 at 18:03 -0400, Tony Hosking wrote: >>>> I'm not sure I understand what you mean? >>>> >>>> On Mar 13, 2008, at 5:46 PM, Dragi?a Duri? wrote: >>>> >>>>> Are objects on pages prepacked/moved? >>>>> >>>>> On Thu, 2008-03-13 at 17:41 -0400, Tony Hosking wrote: >>>>>> The thing about conservatism is that even if you NIL *all* >>>>>> references >>>>>> to an object that just happens to lie on a pinned page, that >>>>>> object >>>>>> (and objects reachable from it) will be retained until the page >>>>>> is no >>>>>> longer pinned. >>>>>> >>>>>> Antony Hosking | Associate Professor | Computer Science | Purdue >>>>>> University >>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mar 13, 2008, at 1:43 PM, Dragi?a Duri? wrote: >>>>>> >>>>>>> I understand transitiveness :), but I've made big effort to >>>>>>> escape >>>>>>> that. >>>>>>> >>>>>>> Lifetime of thread equals lifetime of Client.T instance, and >>>>>>> that >>>>>>> instance is kept in thread's closure, and also in local >>>>>>> variable. >>>>>>> Both >>>>>>> are set to NIL before thread termination - all terminations and >>>>>>> joins >>>>>>> happen and are counted and shown. >>>>>>> >>>>>>> Pointers to Message objects are kept in two places. Hub.T keeps >>>>>>> pointer >>>>>>> to list tail, and each Client.T contains pointer to it's current >>>>>>> element. That pointer is forwarded as Message's are added to >>>>>>> end. No >>>>>>> temporary variable, not even WITH substitution is used. And >>>>>>> these >>>>>>> threads are ones dying with their Client.T's. >>>>>>> >>>>>>> "head" is new mechanism used to unlink all elements with id less >>>>>>> than >>>>>>> minimal id used in active Client.T instances. Unlinked totally >>>>>>> and >>>>>>> all >>>>>>> as one free for garbage collector to finish them. >>>>>>> >>>>>>> I see no transition. >>>>>>> >>>>>>> dd >>>>>>> >>>>>>> On Thu, 2008-03-13 at 12:45 -0400, Tony Hosking wrote: >>>>>>>> Realizing that conservative roots collectors like ours can >>>>>>>> retain >>>>>>>> data >>>>>>>> unnecessarily. If a thread stack refers to a page that holds a >>>>>>>> reference that (transitively) refers to some otherwise dead >>>>>>>> object >>>>>>>> then that object will be retained. We say that the page is >>>>>>>> pinned. >>>>>>>> Smartening up the collector to ignore dead objects in pinned >>>>>>>> pages >>>>>>>> is >>>>>>>> possible, but not currently implemented. >>>>>>>> >>>>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>>>> Purdue >>>>>>>> University >>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mar 13, 2008, at 12:28 PM, Dragi?a Duri? wrote: >>>>>>>> >>>>>>>>> [17:25:19] 117 users, 1 ops and 1 unresolved; >>>>>>>>> minId = >>>>>>>>> 18617; thrs = 120 >>>>>>>>> [17:25:19] started = 663; cleaned = 452; nilled = >>>>>>>>> 543; >>>>>>>>> joined = 543 >>>>>>>>> [17:25:19] Dangling thrs-a-b-n = 1 >>>>>>>>> [17:25:19] Live queue end is 18621 (head = 18616), >>>>>>>>> maxClean = 18611, cleaned = 16583 >>>>>>>>> >>>>>>>>> About one hour uptime, after I added RTCollector.Collect() >>>>>>>>> call >>>>>>>>> every 20 >>>>>>>>> seconds. >>>>>>>>> >>>>>>>>> dd >>>>>>>>> >>>>>>>>> On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking wrote: >>>>>>>>>> Yes. RTCollectorSRC.FinishCollection; >>>>>>>>>> RTCollectorSRC.StartCollection. This will finish the current >>>>>>>>>> (possibly minor) collection, and start a major collection. >>>>>>>>>> The >>>>>>>>>> major >>>>>>>>>> collection must finish before you can guarantee garbage has >>>>>>>>>> been >>>>>>>>>> freed, so you may want to RTCollectorSRC.FinishCollection. >>>>>>>>>> This >>>>>>>>>> sequence is packaged as RTCollector.Collect. >>>>>>>>>> >>>>>>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>>>>>> Purdue >>>>>>>>>> University >>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? wrote: >>>>>>>>>> >>>>>>>>>>> Can I force "major collection" ? >>>>>>>>>>> >>>>>>>>>>> dd >>>>>>>>>>> >>>>>>>>>>> On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: >>>>>>>>>>>> Indeed! Can you devise a testcase? Note that the >>>>>>>>>>>> generational >>>>>>>>>>>> collector will retain old objects for some time until a >>>>>>>>>>>> major >>>>>>>>>>>> collection occurs, >>>>>>>>>>>> >>>>>>>>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>>>>>>>> Purdue >>>>>>>>>>>> University >>>>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: >>>>>>>>>>>> >>>>>>>>>>>>> And not only that, their stackbase is also set to NIL, >>>>>>>>>>>>> meaning >>>>>>>>>>>>> their >>>>>>>>>>>>> stack is not regarded in any way during future >>>>>>>>>>>>> collections - >>>>>>>>>>>>> meaning >>>>>>>>>>>>> all >>>>>>>>>>>>> local variables are forgotten once apply method returns. >>>>>>>>>>>>> >>>>>>>>>>>>> That way, "spuriousness" is not an issue, once thread >>>>>>>>>>>>> returns? >>>>>>>>>>>>> >>>>>>>>>>>>> dd >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: >>>>>>>>>>>>>> They should get unlinked from the global ring. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Antony Hosking | Associate Professor | Computer >>>>>>>>>>>>>> Science | >>>>>>>>>>>>>> Purdue >>>>>>>>>>>>>> University >>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | >>>>>>>>>>>>>> USA >>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> What happens to thread stacks once threads are >>>>>>>>>>>>>>> Join-ed? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> dd >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> This is probably a result of conservatism for >>>>>>>>>>>>>>>> references >>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>> thread >>>>>>>>>>>>>>>> stacks which results in spurious retention. It >>>>>>>>>>>>>>>> may be >>>>>>>>>>>>>>>> necessary >>>>>>>>>>>>>>>> to be >>>>>>>>>>>>>>>> more careful about what references are held on the >>>>>>>>>>>>>>>> stack >>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> threads implementation (and elsewhere in your >>>>>>>>>>>>>>>> code). >>>>>>>>>>>>>>>> Until we >>>>>>>>>>>>>>>> diagnose the place where objects are being >>>>>>>>>>>>>>>> retained it >>>>>>>>>>>>>>>> will be >>>>>>>>>>>>>>>> hard to >>>>>>>>>>>>>>>> pinpoint. You should realize that weak refs are a >>>>>>>>>>>>>>>> problem >>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>> many >>>>>>>>>>>>>>>> language implementations, not just Modula-3. I >>>>>>>>>>>>>>>> look >>>>>>>>>>>>>>>> forward >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>> hearing more definitively from you. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Antony Hosking | Associate Professor | Computer >>>>>>>>>>>>>>>> Science | >>>>>>>>>>>>>>>> Purdue >>>>>>>>>>>>>>>> University >>>>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN >>>>>>>>>>>>>>>> 47907 | >>>>>>>>>>>>>>>> USA >>>>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Problem in my case can be thoroughness of GC... >>>>>>>>>>>>>>>>> Because >>>>>>>>>>>>>>>>> it's >>>>>>>>>>>>>>>>> linked >>>>>>>>>>>>>>>>> list of WeakRef-ed objects, it's always first >>>>>>>>>>>>>>>>> element in >>>>>>>>>>>>>>>>> list to >>>>>>>>>>>>>>>>> become >>>>>>>>>>>>>>>>> unreachable first - all others are reachable at >>>>>>>>>>>>>>>>> least >>>>>>>>>>>>>>>>> through >>>>>>>>>>>>>>>>> previous >>>>>>>>>>>>>>>>> (in list) WeakRef-ed object. So, behaviour I >>>>>>>>>>>>>>>>> have >>>>>>>>>>>>>>>>> observed >>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>> because of some thousands of freeable objects, >>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>> one >>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>> missed >>>>>>>>>>>>>>>>> too >>>>>>>>>>>>>>>>> often. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I have other object being cleaned and counted... >>>>>>>>>>>>>>>>> There >>>>>>>>>>>>>>>>> also >>>>>>>>>>>>>>>>> cleanup >>>>>>>>>>>>>>>>> invocation looks like it's lagging too much. >>>>>>>>>>>>>>>>> I'll >>>>>>>>>>>>>>>>> come >>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>> numbers >>>>>>>>>>>>>>>>> after I have more tests. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> dd >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> Sounds like we need some sort of testcase. >>>>>>>>>>>>>>>>>> Would >>>>>>>>>>>>>>>>>> you >>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>> able >>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>> devise one? It will be hard to make it >>>>>>>>>>>>>>>>>> deterministic, >>>>>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>> least we >>>>>>>>>>>>>>>>>> should see a non-zero cleanup count. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Antony Hosking | Associate Professor | >>>>>>>>>>>>>>>>>> Computer >>>>>>>>>>>>>>>>>> Science | >>>>>>>>>>>>>>>>>> Purdue >>>>>>>>>>>>>>>>>> University >>>>>>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN >>>>>>>>>>>>>>>>>> 47907 | >>>>>>>>>>>>>>>>>> USA >>>>>>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 >>>>>>>>>>>>>>>>>> 5484 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? >>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I have single linked list that I use to send >>>>>>>>>>>>>>>>>>> messages to >>>>>>>>>>>>>>>>>>> many >>>>>>>>>>>>>>>>>>> threads. >>>>>>>>>>>>>>>>>>> It's end is locked for addition, and current >>>>>>>>>>>>>>>>>>> end >>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>> given to >>>>>>>>>>>>>>>>>>> each >>>>>>>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>>>> client connecting. This way, all client are >>>>>>>>>>>>>>>>>>> going >>>>>>>>>>>>>>>>>>> towards >>>>>>>>>>>>>>>>>>> end of >>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> list so it's head becomes unreferenced and >>>>>>>>>>>>>>>>>>> goes >>>>>>>>>>>>>>>>>>> away >>>>>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>> GC. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Except it does not. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I've added WeakRef cleanup and sequential >>>>>>>>>>>>>>>>>>> id's >>>>>>>>>>>>>>>>>>> so I >>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>> record >>>>>>>>>>>>>>>>>>> maximum >>>>>>>>>>>>>>>>>>> freed id for checking. No single cleanup >>>>>>>>>>>>>>>>>>> happens. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I'll cross check my code and count my >>>>>>>>>>>>>>>>>>> references >>>>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>>>> application >>>>>>>>>>>>>>>>>>> side, >>>>>>>>>>>>>>>>>>> but maybe someone else has similar problems? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> dd >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Dragi?a Duri? >>>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> Dragi?a Duri? >>>>>>> >>>>>> >>>>> -- >>>>> Dragi?a Duri? >>>> >>> -- >>> Dragi?a Duri? >> > -- > Dragi?a Duri? From dragisha at m3w.org Fri Mar 14 00:05:10 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Fri, 14 Mar 2008 00:05:10 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <2EA2C202-F053-4AE8-BA60-E51A1A339C6B@cs.purdue.edu> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> <1205425734.32152.144.camel@hias.m3w.org> <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> <1205430199.32152.152.camel@hias.m3w.org> <1205444774.32152.158.camel@hias.m3w.org> <5649BA54-5B2F-423C-AEF7-65FF09CF3763@cs.purdue.edu> <1205446178.32152.162.camel@hias.m3w.org> <4DD89DC7-FA3C-4B61-A2A1-5C8488CF3A3B@cs.purdue.edu> <1205446902.32152.168.camel@hias.m3w.org> <2EA2C202-F053-4AE8-BA60-E51A1A339C6B@cs.purdue.edu> Message-ID: <1205449510.32152.171.camel@hias.m3w.org> Is your work on GC covered somewhere else except source, comments and this list? dd On Thu, 2008-03-13 at 18:35 -0400, Tony Hosking wrote: > Ah, the fact that you needed to be aggressive suggests that > conservatism may be the culprit. I keep meaning one day to make sure > that we do not can objects on pinned pages unless we know that object > is definitely alive. > > On Mar 13, 2008, at 6:21 PM, Dragi?a Duri? wrote: > > > I see... What bugs me now is a fact about my flow... I am already > > doing > > this, and there is always few thousand objects uncleaned from of type > > Message... I'll follow this through, to crosscheck everything again. > > > > What is nice, after I made this agressive RTCollector.Collect > > "policy", > > my memory footprint remains same for many hours... It was swelling > > before. > > > > Thanks, > > dd > > > > On Thu, 2008-03-13 at 18:15 -0400, Tony Hosking wrote: > >> No, when a page is pinned because one of its objects is directly > >> referenced from a thread stack then all objects on that page, both > >> live and dead, are retained. The collector does not move objects to > >> or from pinned pages. The objects on the page can only be freed when > >> the page is no longer pinned by a direct reference from any thread > >> stack. Moral of the story is to NIL references aggressively in local > >> variables. > >> > >> On Mar 13, 2008, at 6:09 PM, Dragi?a Duri? wrote: > >> > >>> When object becomes unreachable, and it sits on page with other > >>> reachable objects, it's pinned? > >>> > >>> What I asked is... As garbage collector moves objects around, would > >>> it > >>> pack reachable objects over unreachable ones in these pages? Thus > >>> kicking them out of pages where they're pinned? > >>> > >>> dd > >>> > >>> On Thu, 2008-03-13 at 18:03 -0400, Tony Hosking wrote: > >>>> I'm not sure I understand what you mean? > >>>> > >>>> On Mar 13, 2008, at 5:46 PM, Dragi?a Duri? wrote: > >>>> > >>>>> Are objects on pages prepacked/moved? > >>>>> > >>>>> On Thu, 2008-03-13 at 17:41 -0400, Tony Hosking wrote: > >>>>>> The thing about conservatism is that even if you NIL *all* > >>>>>> references > >>>>>> to an object that just happens to lie on a pinned page, that > >>>>>> object > >>>>>> (and objects reachable from it) will be retained until the page > >>>>>> is no > >>>>>> longer pinned. > >>>>>> > >>>>>> Antony Hosking | Associate Professor | Computer Science | Purdue > >>>>>> University > >>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> On Mar 13, 2008, at 1:43 PM, Dragi?a Duri? wrote: > >>>>>> > >>>>>>> I understand transitiveness :), but I've made big effort to > >>>>>>> escape > >>>>>>> that. > >>>>>>> > >>>>>>> Lifetime of thread equals lifetime of Client.T instance, and > >>>>>>> that > >>>>>>> instance is kept in thread's closure, and also in local > >>>>>>> variable. > >>>>>>> Both > >>>>>>> are set to NIL before thread termination - all terminations and > >>>>>>> joins > >>>>>>> happen and are counted and shown. > >>>>>>> > >>>>>>> Pointers to Message objects are kept in two places. Hub.T keeps > >>>>>>> pointer > >>>>>>> to list tail, and each Client.T contains pointer to it's current > >>>>>>> element. That pointer is forwarded as Message's are added to > >>>>>>> end. No > >>>>>>> temporary variable, not even WITH substitution is used. And > >>>>>>> these > >>>>>>> threads are ones dying with their Client.T's. > >>>>>>> > >>>>>>> "head" is new mechanism used to unlink all elements with id less > >>>>>>> than > >>>>>>> minimal id used in active Client.T instances. Unlinked totally > >>>>>>> and > >>>>>>> all > >>>>>>> as one free for garbage collector to finish them. > >>>>>>> > >>>>>>> I see no transition. > >>>>>>> > >>>>>>> dd > >>>>>>> > >>>>>>> On Thu, 2008-03-13 at 12:45 -0400, Tony Hosking wrote: > >>>>>>>> Realizing that conservative roots collectors like ours can > >>>>>>>> retain > >>>>>>>> data > >>>>>>>> unnecessarily. If a thread stack refers to a page that holds a > >>>>>>>> reference that (transitively) refers to some otherwise dead > >>>>>>>> object > >>>>>>>> then that object will be retained. We say that the page is > >>>>>>>> pinned. > >>>>>>>> Smartening up the collector to ignore dead objects in pinned > >>>>>>>> pages > >>>>>>>> is > >>>>>>>> possible, but not currently implemented. > >>>>>>>> > >>>>>>>> Antony Hosking | Associate Professor | Computer Science | > >>>>>>>> Purdue > >>>>>>>> University > >>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> On Mar 13, 2008, at 12:28 PM, Dragi?a Duri? wrote: > >>>>>>>> > >>>>>>>>> [17:25:19] 117 users, 1 ops and 1 unresolved; > >>>>>>>>> minId = > >>>>>>>>> 18617; thrs = 120 > >>>>>>>>> [17:25:19] started = 663; cleaned = 452; nilled = > >>>>>>>>> 543; > >>>>>>>>> joined = 543 > >>>>>>>>> [17:25:19] Dangling thrs-a-b-n = 1 > >>>>>>>>> [17:25:19] Live queue end is 18621 (head = 18616), > >>>>>>>>> maxClean = 18611, cleaned = 16583 > >>>>>>>>> > >>>>>>>>> About one hour uptime, after I added RTCollector.Collect() > >>>>>>>>> call > >>>>>>>>> every 20 > >>>>>>>>> seconds. > >>>>>>>>> > >>>>>>>>> dd > >>>>>>>>> > >>>>>>>>> On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking wrote: > >>>>>>>>>> Yes. RTCollectorSRC.FinishCollection; > >>>>>>>>>> RTCollectorSRC.StartCollection. This will finish the current > >>>>>>>>>> (possibly minor) collection, and start a major collection. > >>>>>>>>>> The > >>>>>>>>>> major > >>>>>>>>>> collection must finish before you can guarantee garbage has > >>>>>>>>>> been > >>>>>>>>>> freed, so you may want to RTCollectorSRC.FinishCollection. > >>>>>>>>>> This > >>>>>>>>>> sequence is packaged as RTCollector.Collect. > >>>>>>>>>> > >>>>>>>>>> Antony Hosking | Associate Professor | Computer Science | > >>>>>>>>>> Purdue > >>>>>>>>>> University > >>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? wrote: > >>>>>>>>>> > >>>>>>>>>>> Can I force "major collection" ? > >>>>>>>>>>> > >>>>>>>>>>> dd > >>>>>>>>>>> > >>>>>>>>>>> On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: > >>>>>>>>>>>> Indeed! Can you devise a testcase? Note that the > >>>>>>>>>>>> generational > >>>>>>>>>>>> collector will retain old objects for some time until a > >>>>>>>>>>>> major > >>>>>>>>>>>> collection occurs, > >>>>>>>>>>>> > >>>>>>>>>>>> Antony Hosking | Associate Professor | Computer Science | > >>>>>>>>>>>> Purdue > >>>>>>>>>>>> University > >>>>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> And not only that, their stackbase is also set to NIL, > >>>>>>>>>>>>> meaning > >>>>>>>>>>>>> their > >>>>>>>>>>>>> stack is not regarded in any way during future > >>>>>>>>>>>>> collections - > >>>>>>>>>>>>> meaning > >>>>>>>>>>>>> all > >>>>>>>>>>>>> local variables are forgotten once apply method returns. > >>>>>>>>>>>>> > >>>>>>>>>>>>> That way, "spuriousness" is not an issue, once thread > >>>>>>>>>>>>> returns? > >>>>>>>>>>>>> > >>>>>>>>>>>>> dd > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: > >>>>>>>>>>>>>> They should get unlinked from the global ring. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Antony Hosking | Associate Professor | Computer > >>>>>>>>>>>>>> Science | > >>>>>>>>>>>>>> Purdue > >>>>>>>>>>>>>> University > >>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | > >>>>>>>>>>>>>> USA > >>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> What happens to thread stacks once threads are > >>>>>>>>>>>>>>> Join-ed? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> dd > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> This is probably a result of conservatism for > >>>>>>>>>>>>>>>> references > >>>>>>>>>>>>>>>> from > >>>>>>>>>>>>>>>> thread > >>>>>>>>>>>>>>>> stacks which results in spurious retention. It > >>>>>>>>>>>>>>>> may be > >>>>>>>>>>>>>>>> necessary > >>>>>>>>>>>>>>>> to be > >>>>>>>>>>>>>>>> more careful about what references are held on the > >>>>>>>>>>>>>>>> stack > >>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>> threads implementation (and elsewhere in your > >>>>>>>>>>>>>>>> code). > >>>>>>>>>>>>>>>> Until we > >>>>>>>>>>>>>>>> diagnose the place where objects are being > >>>>>>>>>>>>>>>> retained it > >>>>>>>>>>>>>>>> will be > >>>>>>>>>>>>>>>> hard to > >>>>>>>>>>>>>>>> pinpoint. You should realize that weak refs are a > >>>>>>>>>>>>>>>> problem > >>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>> many > >>>>>>>>>>>>>>>> language implementations, not just Modula-3. I > >>>>>>>>>>>>>>>> look > >>>>>>>>>>>>>>>> forward > >>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>> hearing more definitively from you. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Antony Hosking | Associate Professor | Computer > >>>>>>>>>>>>>>>> Science | > >>>>>>>>>>>>>>>> Purdue > >>>>>>>>>>>>>>>> University > >>>>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN > >>>>>>>>>>>>>>>> 47907 | > >>>>>>>>>>>>>>>> USA > >>>>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Problem in my case can be thoroughness of GC... > >>>>>>>>>>>>>>>>> Because > >>>>>>>>>>>>>>>>> it's > >>>>>>>>>>>>>>>>> linked > >>>>>>>>>>>>>>>>> list of WeakRef-ed objects, it's always first > >>>>>>>>>>>>>>>>> element in > >>>>>>>>>>>>>>>>> list to > >>>>>>>>>>>>>>>>> become > >>>>>>>>>>>>>>>>> unreachable first - all others are reachable at > >>>>>>>>>>>>>>>>> least > >>>>>>>>>>>>>>>>> through > >>>>>>>>>>>>>>>>> previous > >>>>>>>>>>>>>>>>> (in list) WeakRef-ed object. So, behaviour I > >>>>>>>>>>>>>>>>> have > >>>>>>>>>>>>>>>>> observed > >>>>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>> because of some thousands of freeable objects, > >>>>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>> one > >>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>> missed > >>>>>>>>>>>>>>>>> too > >>>>>>>>>>>>>>>>> often. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I have other object being cleaned and counted... > >>>>>>>>>>>>>>>>> There > >>>>>>>>>>>>>>>>> also > >>>>>>>>>>>>>>>>> cleanup > >>>>>>>>>>>>>>>>> invocation looks like it's lagging too much. > >>>>>>>>>>>>>>>>> I'll > >>>>>>>>>>>>>>>>> come > >>>>>>>>>>>>>>>>> with > >>>>>>>>>>>>>>>>> numbers > >>>>>>>>>>>>>>>>> after I have more tests. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> dd > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking > >>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>> Sounds like we need some sort of testcase. > >>>>>>>>>>>>>>>>>> Would > >>>>>>>>>>>>>>>>>> you > >>>>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>> able > >>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>> devise one? It will be hard to make it > >>>>>>>>>>>>>>>>>> deterministic, > >>>>>>>>>>>>>>>>>> but > >>>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>>> least we > >>>>>>>>>>>>>>>>>> should see a non-zero cleanup count. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Antony Hosking | Associate Professor | > >>>>>>>>>>>>>>>>>> Computer > >>>>>>>>>>>>>>>>>> Science | > >>>>>>>>>>>>>>>>>> Purdue > >>>>>>>>>>>>>>>>>> University > >>>>>>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN > >>>>>>>>>>>>>>>>>> 47907 | > >>>>>>>>>>>>>>>>>> USA > >>>>>>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 > >>>>>>>>>>>>>>>>>> 5484 > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? > >>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> I have single linked list that I use to send > >>>>>>>>>>>>>>>>>>> messages to > >>>>>>>>>>>>>>>>>>> many > >>>>>>>>>>>>>>>>>>> threads. > >>>>>>>>>>>>>>>>>>> It's end is locked for addition, and current > >>>>>>>>>>>>>>>>>>> end > >>>>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>> given to > >>>>>>>>>>>>>>>>>>> each > >>>>>>>>>>>>>>>>>>> new > >>>>>>>>>>>>>>>>>>> client connecting. This way, all client are > >>>>>>>>>>>>>>>>>>> going > >>>>>>>>>>>>>>>>>>> towards > >>>>>>>>>>>>>>>>>>> end of > >>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>> list so it's head becomes unreferenced and > >>>>>>>>>>>>>>>>>>> goes > >>>>>>>>>>>>>>>>>>> away > >>>>>>>>>>>>>>>>>>> by > >>>>>>>>>>>>>>>>>>> GC. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Except it does not. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> I've added WeakRef cleanup and sequential > >>>>>>>>>>>>>>>>>>> id's > >>>>>>>>>>>>>>>>>>> so I > >>>>>>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>>>> record > >>>>>>>>>>>>>>>>>>> maximum > >>>>>>>>>>>>>>>>>>> freed id for checking. No single cleanup > >>>>>>>>>>>>>>>>>>> happens. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> I'll cross check my code and count my > >>>>>>>>>>>>>>>>>>> references > >>>>>>>>>>>>>>>>>>> from > >>>>>>>>>>>>>>>>>>> application > >>>>>>>>>>>>>>>>>>> side, > >>>>>>>>>>>>>>>>>>> but maybe someone else has similar problems? > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> dd > >>>>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>>>> Dragi?a Duri? > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>> Dragi?a Duri? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>> Dragi?a Duri? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> -- > >>>>>>>>>>>>> Dragi?a Duri? > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> Dragi?a Duri? > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> Dragi?a Duri? > >>>>>>>>> > >>>>>>>> > >>>>>>> -- > >>>>>>> Dragi?a Duri? > >>>>>>> > >>>>>> > >>>>> -- > >>>>> Dragi?a Duri? > >>>> > >>> -- > >>> Dragi?a Duri? > >> > > -- > > Dragi?a Duri? > -- Dragi?a Duri? From hosking at cs.purdue.edu Fri Mar 14 02:16:16 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 13 Mar 2008 21:16:16 -0400 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <1205449510.32152.171.camel@hias.m3w.org> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> <1205425734.32152.144.camel@hias.m3w.org> <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> <1205430199.32152.152.camel@hias.m3w.org> <1205444774.32152.158.camel@hias.m3w.org> <5649BA54-5B2F-423C-AEF7-65FF09CF3763@cs.purdue.edu> <1205446178.32152.162.camel@hias.m3w.org> <4DD89DC7-FA3C-4B61-A2A1-5C8488CF3A3B@cs.purdue.edu> <1205446902.32152.168.camel@hias.m3w.org> <2EA2C202-F053-4AE8-BA60-E51A1A339C6B@cs.purdue.edu> <1205449510.32152.171.camel@hi! as.m3w.org> Message-ID: <3F319520-7A7E-4DE3-A467-3712EBB8AA8F@cs.purdue.edu> http://www.cs.purdue.edu/homes/hosking/papers/ismm06.pdf On Mar 13, 2008, at 7:05 PM, Dragi?a Duri? wrote: > Is your work on GC covered somewhere else except source, comments and > this list? > > dd > > On Thu, 2008-03-13 at 18:35 -0400, Tony Hosking wrote: >> Ah, the fact that you needed to be aggressive suggests that >> conservatism may be the culprit. I keep meaning one day to make sure >> that we do not can objects on pinned pages unless we know that object >> is definitely alive. >> >> On Mar 13, 2008, at 6:21 PM, Dragi?a Duri? wrote: >> >>> I see... What bugs me now is a fact about my flow... I am already >>> doing >>> this, and there is always few thousand objects uncleaned from of >>> type >>> Message... I'll follow this through, to crosscheck everything again. >>> >>> What is nice, after I made this agressive RTCollector.Collect >>> "policy", >>> my memory footprint remains same for many hours... It was swelling >>> before. >>> >>> Thanks, >>> dd >>> >>> On Thu, 2008-03-13 at 18:15 -0400, Tony Hosking wrote: >>>> No, when a page is pinned because one of its objects is directly >>>> referenced from a thread stack then all objects on that page, both >>>> live and dead, are retained. The collector does not move objects >>>> to >>>> or from pinned pages. The objects on the page can only be freed >>>> when >>>> the page is no longer pinned by a direct reference from any thread >>>> stack. Moral of the story is to NIL references aggressively in >>>> local >>>> variables. >>>> >>>> On Mar 13, 2008, at 6:09 PM, Dragi?a Duri? wrote: >>>> >>>>> When object becomes unreachable, and it sits on page with other >>>>> reachable objects, it's pinned? >>>>> >>>>> What I asked is... As garbage collector moves objects around, >>>>> would >>>>> it >>>>> pack reachable objects over unreachable ones in these pages? Thus >>>>> kicking them out of pages where they're pinned? >>>>> >>>>> dd >>>>> >>>>> On Thu, 2008-03-13 at 18:03 -0400, Tony Hosking wrote: >>>>>> I'm not sure I understand what you mean? >>>>>> >>>>>> On Mar 13, 2008, at 5:46 PM, Dragi?a Duri? wrote: >>>>>> >>>>>>> Are objects on pages prepacked/moved? >>>>>>> >>>>>>> On Thu, 2008-03-13 at 17:41 -0400, Tony Hosking wrote: >>>>>>>> The thing about conservatism is that even if you NIL *all* >>>>>>>> references >>>>>>>> to an object that just happens to lie on a pinned page, that >>>>>>>> object >>>>>>>> (and objects reachable from it) will be retained until the page >>>>>>>> is no >>>>>>>> longer pinned. >>>>>>>> >>>>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>>>> Purdue >>>>>>>> University >>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mar 13, 2008, at 1:43 PM, Dragi?a Duri? wrote: >>>>>>>> >>>>>>>>> I understand transitiveness :), but I've made big effort to >>>>>>>>> escape >>>>>>>>> that. >>>>>>>>> >>>>>>>>> Lifetime of thread equals lifetime of Client.T instance, and >>>>>>>>> that >>>>>>>>> instance is kept in thread's closure, and also in local >>>>>>>>> variable. >>>>>>>>> Both >>>>>>>>> are set to NIL before thread termination - all terminations >>>>>>>>> and >>>>>>>>> joins >>>>>>>>> happen and are counted and shown. >>>>>>>>> >>>>>>>>> Pointers to Message objects are kept in two places. Hub.T >>>>>>>>> keeps >>>>>>>>> pointer >>>>>>>>> to list tail, and each Client.T contains pointer to it's >>>>>>>>> current >>>>>>>>> element. That pointer is forwarded as Message's are added to >>>>>>>>> end. No >>>>>>>>> temporary variable, not even WITH substitution is used. And >>>>>>>>> these >>>>>>>>> threads are ones dying with their Client.T's. >>>>>>>>> >>>>>>>>> "head" is new mechanism used to unlink all elements with id >>>>>>>>> less >>>>>>>>> than >>>>>>>>> minimal id used in active Client.T instances. Unlinked totally >>>>>>>>> and >>>>>>>>> all >>>>>>>>> as one free for garbage collector to finish them. >>>>>>>>> >>>>>>>>> I see no transition. >>>>>>>>> >>>>>>>>> dd >>>>>>>>> >>>>>>>>> On Thu, 2008-03-13 at 12:45 -0400, Tony Hosking wrote: >>>>>>>>>> Realizing that conservative roots collectors like ours can >>>>>>>>>> retain >>>>>>>>>> data >>>>>>>>>> unnecessarily. If a thread stack refers to a page that >>>>>>>>>> holds a >>>>>>>>>> reference that (transitively) refers to some otherwise dead >>>>>>>>>> object >>>>>>>>>> then that object will be retained. We say that the page is >>>>>>>>>> pinned. >>>>>>>>>> Smartening up the collector to ignore dead objects in pinned >>>>>>>>>> pages >>>>>>>>>> is >>>>>>>>>> possible, but not currently implemented. >>>>>>>>>> >>>>>>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>>>>>> Purdue >>>>>>>>>> University >>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mar 13, 2008, at 12:28 PM, Dragi?a Duri? wrote: >>>>>>>>>> >>>>>>>>>>> [17:25:19] 117 users, 1 ops and 1 unresolved; >>>>>>>>>>> minId = >>>>>>>>>>> 18617; thrs = 120 >>>>>>>>>>> [17:25:19] started = 663; cleaned = 452; nilled = >>>>>>>>>>> 543; >>>>>>>>>>> joined = 543 >>>>>>>>>>> [17:25:19] Dangling thrs-a-b-n = 1 >>>>>>>>>>> [17:25:19] Live queue end is 18621 (head = 18616), >>>>>>>>>>> maxClean = 18611, cleaned = 16583 >>>>>>>>>>> >>>>>>>>>>> About one hour uptime, after I added RTCollector.Collect() >>>>>>>>>>> call >>>>>>>>>>> every 20 >>>>>>>>>>> seconds. >>>>>>>>>>> >>>>>>>>>>> dd >>>>>>>>>>> >>>>>>>>>>> On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking wrote: >>>>>>>>>>>> Yes. RTCollectorSRC.FinishCollection; >>>>>>>>>>>> RTCollectorSRC.StartCollection. This will finish the >>>>>>>>>>>> current >>>>>>>>>>>> (possibly minor) collection, and start a major collection. >>>>>>>>>>>> The >>>>>>>>>>>> major >>>>>>>>>>>> collection must finish before you can guarantee garbage has >>>>>>>>>>>> been >>>>>>>>>>>> freed, so you may want to RTCollectorSRC.FinishCollection. >>>>>>>>>>>> This >>>>>>>>>>>> sequence is packaged as RTCollector.Collect. >>>>>>>>>>>> >>>>>>>>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>>>>>>>> Purdue >>>>>>>>>>>> University >>>>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Can I force "major collection" ? >>>>>>>>>>>>> >>>>>>>>>>>>> dd >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: >>>>>>>>>>>>>> Indeed! Can you devise a testcase? Note that the >>>>>>>>>>>>>> generational >>>>>>>>>>>>>> collector will retain old objects for some time until a >>>>>>>>>>>>>> major >>>>>>>>>>>>>> collection occurs, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>>>>>>>>>> Purdue >>>>>>>>>>>>>> University >>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | >>>>>>>>>>>>>> USA >>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> And not only that, their stackbase is also set to NIL, >>>>>>>>>>>>>>> meaning >>>>>>>>>>>>>>> their >>>>>>>>>>>>>>> stack is not regarded in any way during future >>>>>>>>>>>>>>> collections - >>>>>>>>>>>>>>> meaning >>>>>>>>>>>>>>> all >>>>>>>>>>>>>>> local variables are forgotten once apply method returns. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> That way, "spuriousness" is not an issue, once thread >>>>>>>>>>>>>>> returns? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> dd >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: >>>>>>>>>>>>>>>> They should get unlinked from the global ring. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Antony Hosking | Associate Professor | Computer >>>>>>>>>>>>>>>> Science | >>>>>>>>>>>>>>>> Purdue >>>>>>>>>>>>>>>> University >>>>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | >>>>>>>>>>>>>>>> USA >>>>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> What happens to thread stacks once threads are >>>>>>>>>>>>>>>>> Join-ed? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> dd >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> This is probably a result of conservatism for >>>>>>>>>>>>>>>>>> references >>>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>>> thread >>>>>>>>>>>>>>>>>> stacks which results in spurious retention. It >>>>>>>>>>>>>>>>>> may be >>>>>>>>>>>>>>>>>> necessary >>>>>>>>>>>>>>>>>> to be >>>>>>>>>>>>>>>>>> more careful about what references are held on the >>>>>>>>>>>>>>>>>> stack >>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> threads implementation (and elsewhere in your >>>>>>>>>>>>>>>>>> code). >>>>>>>>>>>>>>>>>> Until we >>>>>>>>>>>>>>>>>> diagnose the place where objects are being >>>>>>>>>>>>>>>>>> retained it >>>>>>>>>>>>>>>>>> will be >>>>>>>>>>>>>>>>>> hard to >>>>>>>>>>>>>>>>>> pinpoint. You should realize that weak refs are a >>>>>>>>>>>>>>>>>> problem >>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>> many >>>>>>>>>>>>>>>>>> language implementations, not just Modula-3. I >>>>>>>>>>>>>>>>>> look >>>>>>>>>>>>>>>>>> forward >>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>> hearing more definitively from you. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Antony Hosking | Associate Professor | Computer >>>>>>>>>>>>>>>>>> Science | >>>>>>>>>>>>>>>>>> Purdue >>>>>>>>>>>>>>>>>> University >>>>>>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN >>>>>>>>>>>>>>>>>> 47907 | >>>>>>>>>>>>>>>>>> USA >>>>>>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Problem in my case can be thoroughness of GC... >>>>>>>>>>>>>>>>>>> Because >>>>>>>>>>>>>>>>>>> it's >>>>>>>>>>>>>>>>>>> linked >>>>>>>>>>>>>>>>>>> list of WeakRef-ed objects, it's always first >>>>>>>>>>>>>>>>>>> element in >>>>>>>>>>>>>>>>>>> list to >>>>>>>>>>>>>>>>>>> become >>>>>>>>>>>>>>>>>>> unreachable first - all others are reachable at >>>>>>>>>>>>>>>>>>> least >>>>>>>>>>>>>>>>>>> through >>>>>>>>>>>>>>>>>>> previous >>>>>>>>>>>>>>>>>>> (in list) WeakRef-ed object. So, behaviour I >>>>>>>>>>>>>>>>>>> have >>>>>>>>>>>>>>>>>>> observed >>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>> because of some thousands of freeable objects, >>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>> one >>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>> missed >>>>>>>>>>>>>>>>>>> too >>>>>>>>>>>>>>>>>>> often. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I have other object being cleaned and counted... >>>>>>>>>>>>>>>>>>> There >>>>>>>>>>>>>>>>>>> also >>>>>>>>>>>>>>>>>>> cleanup >>>>>>>>>>>>>>>>>>> invocation looks like it's lagging too much. >>>>>>>>>>>>>>>>>>> I'll >>>>>>>>>>>>>>>>>>> come >>>>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>>>> numbers >>>>>>>>>>>>>>>>>>> after I have more tests. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> dd >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>> Sounds like we need some sort of testcase. >>>>>>>>>>>>>>>>>>>> Would >>>>>>>>>>>>>>>>>>>> you >>>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>> able >>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>> devise one? It will be hard to make it >>>>>>>>>>>>>>>>>>>> deterministic, >>>>>>>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>> least we >>>>>>>>>>>>>>>>>>>> should see a non-zero cleanup count. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Antony Hosking | Associate Professor | >>>>>>>>>>>>>>>>>>>> Computer >>>>>>>>>>>>>>>>>>>> Science | >>>>>>>>>>>>>>>>>>>> Purdue >>>>>>>>>>>>>>>>>>>> University >>>>>>>>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN >>>>>>>>>>>>>>>>>>>> 47907 | >>>>>>>>>>>>>>>>>>>> USA >>>>>>>>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 >>>>>>>>>>>>>>>>>>>> 5484 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? >>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I have single linked list that I use to send >>>>>>>>>>>>>>>>>>>>> messages to >>>>>>>>>>>>>>>>>>>>> many >>>>>>>>>>>>>>>>>>>>> threads. >>>>>>>>>>>>>>>>>>>>> It's end is locked for addition, and current >>>>>>>>>>>>>>>>>>>>> end >>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>> given to >>>>>>>>>>>>>>>>>>>>> each >>>>>>>>>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>>>>>> client connecting. This way, all client are >>>>>>>>>>>>>>>>>>>>> going >>>>>>>>>>>>>>>>>>>>> towards >>>>>>>>>>>>>>>>>>>>> end of >>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> list so it's head becomes unreferenced and >>>>>>>>>>>>>>>>>>>>> goes >>>>>>>>>>>>>>>>>>>>> away >>>>>>>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>>>> GC. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Except it does not. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I've added WeakRef cleanup and sequential >>>>>>>>>>>>>>>>>>>>> id's >>>>>>>>>>>>>>>>>>>>> so I >>>>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>>> record >>>>>>>>>>>>>>>>>>>>> maximum >>>>>>>>>>>>>>>>>>>>> freed id for checking. No single cleanup >>>>>>>>>>>>>>>>>>>>> happens. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I'll cross check my code and count my >>>>>>>>>>>>>>>>>>>>> references >>>>>>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>>>>>> application >>>>>>>>>>>>>>>>>>>>> side, >>>>>>>>>>>>>>>>>>>>> but maybe someone else has similar problems? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> dd >>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Dragi?a Duri? >>>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> Dragi?a Duri? >>>>>> >>>>> -- >>>>> Dragi?a Duri? >>>> >>> -- >>> Dragi?a Duri? >> > -- > Dragi?a Duri? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Fri Mar 14 07:45:57 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Fri, 14 Mar 2008 07:45:57 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <3F319520-7A7E-4DE3-A467-3712EBB8AA8F@cs.purdue.edu> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> <1205425734.32152.144.camel@hias.m3w.org> <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> <1205430199.32152.152.camel@hias.m3w.org> <1205444774.32152.158.camel@hias.m3w.org> <5649BA54-5B2F-423C-AEF7-65FF09CF3763@cs.purdue.edu> <1205446178.32152.162.camel@hias.m3w.org> <4DD89DC7-FA3C-4B61-A2A1-5C8488CF3A3B@cs.purdue.edu> <1205446902.32152.168.camel@hias.m3w.org> <2EA2C202-F053-4AE8-BA60-E51A1A339C6B@cs.purdue.edu> <1205449510.32152.171.camel@hi! as.m3w.org> <3F319520-7A7E-4DE3-A467-3712EBB8AA8F@cs.purdue.edu> Message-ID: <1205477157.32152.189.camel@hias.m3w.org> [07:29:28] 0 users, 1 ops and 0 unresolved; minId = 44906; thrs = 1 [07:29:28] started = 2201; cleaned = 2199; nilled = 2200; joined = 2200 [07:29:28] Dangling thrs-a-b-n = 0 [07:29:28] Live queue end is 44909 (head = 44906), maxClean = 44897, cleaned = 44897 After all keep-pointers-off-stack, one thing that remained was long living closures... And I killed them all and what happened was total confirmation of your explanation... One Client.T that remained is most probably pinned on page with listener thread's closure object. My understanding of conservativeness of GC with regard to register and stack content was something else... I was thinking it's like this: ProcessStacks will process registers and stacks; each time address-sized, correctly-aligned bit pattern which looks like address of some object GC knows occurs, GC accepts it as it's address - it maybe is not, but conservative thinking will not make risks there. This way, ProcessStacks will know exactly what objects are (maybe, but no risks...) referenced from stack. I know I am missing something here, and that's why I ask, why don't you do it just like this? dd On Thu, 2008-03-13 at 21:16 -0400, Tony Hosking wrote: > http://www.cs.purdue.edu/homes/hosking/papers/ismm06.pdf > > > On Mar 13, 2008, at 7:05 PM, Dragi?a Duri? wrote: > > > Is your work on GC covered somewhere else except source, comments > > and > > this list? > > > > dd > > > > On Thu, 2008-03-13 at 18:35 -0400, Tony Hosking wrote: > > > Ah, the fact that you needed to be aggressive suggests that > > > conservatism may be the culprit. I keep meaning one day to make > > > sure > > > that we do not can objects on pinned pages unless we know that > > > object > > > is definitely alive. > > > > > > On Mar 13, 2008, at 6:21 PM, Dragi?a Duri? wrote: > > > > > > > I see... What bugs me now is a fact about my flow... I am > > > > already > > > > doing > > > > this, and there is always few thousand objects uncleaned from of > > > > type > > > > Message... I'll follow this through, to crosscheck everything > > > > again. > > > > > > > > What is nice, after I made this agressive RTCollector.Collect > > > > "policy", > > > > my memory footprint remains same for many hours... It was > > > > swelling > > > > before. > > > > > > > > Thanks, > > > > dd > > > > > > > > On Thu, 2008-03-13 at 18:15 -0400, Tony Hosking wrote: > > > > > No, when a page is pinned because one of its objects is > > > > > directly > > > > > referenced from a thread stack then all objects on that page, > > > > > both > > > > > live and dead, are retained. The collector does not move > > > > > objects to > > > > > or from pinned pages. The objects on the page can only be > > > > > freed when > > > > > the page is no longer pinned by a direct reference from any > > > > > thread > > > > > stack. Moral of the story is to NIL references aggressively > > > > > in local > > > > > variables. > > > > > > > > > > On Mar 13, 2008, at 6:09 PM, Dragi?a Duri? wrote: > > > > > > > > > > > When object becomes unreachable, and it sits on page with > > > > > > other > > > > > > reachable objects, it's pinned? > > > > > > > > > > > > What I asked is... As garbage collector moves objects > > > > > > around, would > > > > > > it > > > > > > pack reachable objects over unreachable ones in these pages? > > > > > > Thus > > > > > > kicking them out of pages where they're pinned? > > > > > > > > > > > > dd > > > > > > > > > > > > On Thu, 2008-03-13 at 18:03 -0400, Tony Hosking wrote: > > > > > > > I'm not sure I understand what you mean? > > > > > > > > > > > > > > On Mar 13, 2008, at 5:46 PM, Dragi?a Duri? wrote: > > > > > > > > > > > > > > > Are objects on pages prepacked/moved? > > > > > > > > > > > > > > > > On Thu, 2008-03-13 at 17:41 -0400, Tony Hosking wrote: > > > > > > > > > The thing about conservatism is that even if you NIL > > > > > > > > > *all* > > > > > > > > > references > > > > > > > > > to an object that just happens to lie on a pinned > > > > > > > > > page, that > > > > > > > > > object > > > > > > > > > (and objects reachable from it) will be retained until > > > > > > > > > the page > > > > > > > > > is no > > > > > > > > > longer pinned. > > > > > > > > > > > > > > > > > > Antony Hosking | Associate Professor | Computer > > > > > > > > > Science | Purdue > > > > > > > > > University > > > > > > > > > 305 N. University Street | West Lafayette | IN 47907 | > > > > > > > > > USA > > > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 1:43 PM, Dragi?a Duri? wrote: > > > > > > > > > > > > > > > > > > > I understand transitiveness :), but I've made big > > > > > > > > > > effort to > > > > > > > > > > escape > > > > > > > > > > that. > > > > > > > > > > > > > > > > > > > > Lifetime of thread equals lifetime of Client.T > > > > > > > > > > instance, and > > > > > > > > > > that > > > > > > > > > > instance is kept in thread's closure, and also in > > > > > > > > > > local > > > > > > > > > > variable. > > > > > > > > > > Both > > > > > > > > > > are set to NIL before thread termination - all > > > > > > > > > > terminations and > > > > > > > > > > joins > > > > > > > > > > happen and are counted and shown. > > > > > > > > > > > > > > > > > > > > Pointers to Message objects are kept in two places. > > > > > > > > > > Hub.T keeps > > > > > > > > > > pointer > > > > > > > > > > to list tail, and each Client.T contains pointer to > > > > > > > > > > it's current > > > > > > > > > > element. That pointer is forwarded as Message's are > > > > > > > > > > added to > > > > > > > > > > end. No > > > > > > > > > > temporary variable, not even WITH substitution is > > > > > > > > > > used. And > > > > > > > > > > these > > > > > > > > > > threads are ones dying with their Client.T's. > > > > > > > > > > > > > > > > > > > > "head" is new mechanism used to unlink all elements > > > > > > > > > > with id less > > > > > > > > > > than > > > > > > > > > > minimal id used in active Client.T instances. > > > > > > > > > > Unlinked totally > > > > > > > > > > and > > > > > > > > > > all > > > > > > > > > > as one free for garbage collector to finish them. > > > > > > > > > > > > > > > > > > > > I see no transition. > > > > > > > > > > > > > > > > > > > > dd > > > > > > > > > > > > > > > > > > > > On Thu, 2008-03-13 at 12:45 -0400, Tony Hosking > > > > > > > > > > wrote: > > > > > > > > > > > Realizing that conservative roots collectors like > > > > > > > > > > > ours can > > > > > > > > > > > retain > > > > > > > > > > > data > > > > > > > > > > > unnecessarily. If a thread stack refers to a page > > > > > > > > > > > that holds a > > > > > > > > > > > reference that (transitively) refers to some > > > > > > > > > > > otherwise dead > > > > > > > > > > > object > > > > > > > > > > > then that object will be retained. We say that > > > > > > > > > > > the page is > > > > > > > > > > > pinned. > > > > > > > > > > > Smartening up the collector to ignore dead objects > > > > > > > > > > > in pinned > > > > > > > > > > > pages > > > > > > > > > > > is > > > > > > > > > > > possible, but not currently implemented. > > > > > > > > > > > > > > > > > > > > > > Antony Hosking | Associate Professor | Computer > > > > > > > > > > > Science | > > > > > > > > > > > Purdue > > > > > > > > > > > University > > > > > > > > > > > 305 N. University Street | West Lafayette | IN > > > > > > > > > > > 47907 | USA > > > > > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 12:28 PM, Dragi?a Duri? wrote: > > > > > > > > > > > > > > > > > > > > > > > [17:25:19] 117 users, 1 ops and 1 > > > > > > > > > > > > unresolved; > > > > > > > > > > > > minId = > > > > > > > > > > > > 18617; thrs = 120 > > > > > > > > > > > > [17:25:19] started = 663; cleaned = > > > > > > > > > > > > 452; nilled = > > > > > > > > > > > > 543; > > > > > > > > > > > > joined = 543 > > > > > > > > > > > > [17:25:19] Dangling thrs-a-b-n = 1 > > > > > > > > > > > > [17:25:19] Live queue end is 18621 > > > > > > > > > > > > (head = 18616), > > > > > > > > > > > > maxClean = 18611, cleaned = 16583 > > > > > > > > > > > > > > > > > > > > > > > > About one hour uptime, after I added > > > > > > > > > > > > RTCollector.Collect() > > > > > > > > > > > > call > > > > > > > > > > > > every 20 > > > > > > > > > > > > seconds. > > > > > > > > > > > > > > > > > > > > > > > > dd > > > > > > > > > > > > > > > > > > > > > > > > On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Yes. RTCollectorSRC.FinishCollection; > > > > > > > > > > > > > RTCollectorSRC.StartCollection. This will > > > > > > > > > > > > > finish the current > > > > > > > > > > > > > (possibly minor) collection, and start a major > > > > > > > > > > > > > collection. > > > > > > > > > > > > > The > > > > > > > > > > > > > major > > > > > > > > > > > > > collection must finish before you can > > > > > > > > > > > > > guarantee garbage has > > > > > > > > > > > > > been > > > > > > > > > > > > > freed, so you may want to > > > > > > > > > > > > > RTCollectorSRC.FinishCollection. > > > > > > > > > > > > > This > > > > > > > > > > > > > sequence is packaged as RTCollector.Collect. > > > > > > > > > > > > > > > > > > > > > > > > > > Antony Hosking | Associate Professor | > > > > > > > > > > > > > Computer Science | > > > > > > > > > > > > > Purdue > > > > > > > > > > > > > University > > > > > > > > > > > > > 305 N. University Street | West Lafayette | IN > > > > > > > > > > > > > 47907 | USA > > > > > > > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 > > > > > > > > > > > > > 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Can I force "major collection" ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > dd > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, 2008-03-13 at 10:08 -0400, Tony > > > > > > > > > > > > > > Hosking wrote: > > > > > > > > > > > > > > > Indeed! Can you devise a testcase? Note > > > > > > > > > > > > > > > that the > > > > > > > > > > > > > > > generational > > > > > > > > > > > > > > > collector will retain old objects for some > > > > > > > > > > > > > > > time until a > > > > > > > > > > > > > > > major > > > > > > > > > > > > > > > collection occurs, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Antony Hosking | Associate Professor | > > > > > > > > > > > > > > > Computer Science | > > > > > > > > > > > > > > > Purdue > > > > > > > > > > > > > > > University > > > > > > > > > > > > > > > 305 N. University Street | West Lafayette > > > > > > > > > > > > > > > | IN 47907 | USA > > > > > > > > > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 > > > > > > > > > > > > > > > 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > And not only that, their stackbase is > > > > > > > > > > > > > > > > also set to NIL, > > > > > > > > > > > > > > > > meaning > > > > > > > > > > > > > > > > their > > > > > > > > > > > > > > > > stack is not regarded in any way during > > > > > > > > > > > > > > > > future > > > > > > > > > > > > > > > > collections - > > > > > > > > > > > > > > > > meaning > > > > > > > > > > > > > > > > all > > > > > > > > > > > > > > > > local variables are forgotten once apply > > > > > > > > > > > > > > > > method returns. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > That way, "spuriousness" is not an > > > > > > > > > > > > > > > > issue, once thread > > > > > > > > > > > > > > > > returns? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > dd > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, 2008-03-13 at 09:46 -0400, Tony > > > > > > > > > > > > > > > > Hosking wrote: > > > > > > > > > > > > > > > > > They should get unlinked from the > > > > > > > > > > > > > > > > > global ring. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Antony Hosking | Associate Professor | > > > > > > > > > > > > > > > > > Computer > > > > > > > > > > > > > > > > > Science | > > > > > > > > > > > > > > > > > Purdue > > > > > > > > > > > > > > > > > University > > > > > > > > > > > > > > > > > 305 N. University Street | West > > > > > > > > > > > > > > > > > Lafayette | IN 47907 | > > > > > > > > > > > > > > > > > USA > > > > > > > > > > > > > > > > > Office +1 765 494 6001 | Mobile +1 765 > > > > > > > > > > > > > > > > > 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 6:16 AM, Dragi?a > > > > > > > > > > > > > > > > > Duri? wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > What happens to thread stacks once > > > > > > > > > > > > > > > > > > threads are > > > > > > > > > > > > > > > > > > Join-ed? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > dd > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, 2008-03-12 at 11:12 -0400, > > > > > > > > > > > > > > > > > > Tony Hosking > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > This is probably a result of > > > > > > > > > > > > > > > > > > > conservatism for > > > > > > > > > > > > > > > > > > > references > > > > > > > > > > > > > > > > > > > from > > > > > > > > > > > > > > > > > > > thread > > > > > > > > > > > > > > > > > > > stacks which results in spurious > > > > > > > > > > > > > > > > > > > retention. It > > > > > > > > > > > > > > > > > > > may be > > > > > > > > > > > > > > > > > > > necessary > > > > > > > > > > > > > > > > > > > to be > > > > > > > > > > > > > > > > > > > more careful about what references > > > > > > > > > > > > > > > > > > > are held on the > > > > > > > > > > > > > > > > > > > stack > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > threads implementation (and > > > > > > > > > > > > > > > > > > > elsewhere in your > > > > > > > > > > > > > > > > > > > code). > > > > > > > > > > > > > > > > > > > Until we > > > > > > > > > > > > > > > > > > > diagnose the place where objects > > > > > > > > > > > > > > > > > > > are being > > > > > > > > > > > > > > > > > > > retained it > > > > > > > > > > > > > > > > > > > will be > > > > > > > > > > > > > > > > > > > hard to > > > > > > > > > > > > > > > > > > > pinpoint. You should realize that > > > > > > > > > > > > > > > > > > > weak refs are a > > > > > > > > > > > > > > > > > > > problem > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > many > > > > > > > > > > > > > > > > > > > language implementations, not just > > > > > > > > > > > > > > > > > > > Modula-3. I > > > > > > > > > > > > > > > > > > > look > > > > > > > > > > > > > > > > > > > forward > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > hearing more definitively from > > > > > > > > > > > > > > > > > > > you. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Antony Hosking | Associate > > > > > > > > > > > > > > > > > > > Professor | Computer > > > > > > > > > > > > > > > > > > > Science | > > > > > > > > > > > > > > > > > > > Purdue > > > > > > > > > > > > > > > > > > > University > > > > > > > > > > > > > > > > > > > 305 N. University Street | West > > > > > > > > > > > > > > > > > > > Lafayette | IN > > > > > > > > > > > > > > > > > > > 47907 | > > > > > > > > > > > > > > > > > > > USA > > > > > > > > > > > > > > > > > > > Office +1 765 494 6001 | Mobile +1 > > > > > > > > > > > > > > > > > > > 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 12, 2008, at 4:21 AM, > > > > > > > > > > > > > > > > > > > Dragi?a Duri? wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Problem in my case can be > > > > > > > > > > > > > > > > > > > > thoroughness of GC... > > > > > > > > > > > > > > > > > > > > Because > > > > > > > > > > > > > > > > > > > > it's > > > > > > > > > > > > > > > > > > > > linked > > > > > > > > > > > > > > > > > > > > list of WeakRef-ed objects, it's > > > > > > > > > > > > > > > > > > > > always first > > > > > > > > > > > > > > > > > > > > element in > > > > > > > > > > > > > > > > > > > > list to > > > > > > > > > > > > > > > > > > > > become > > > > > > > > > > > > > > > > > > > > unreachable first - all others > > > > > > > > > > > > > > > > > > > > are reachable at > > > > > > > > > > > > > > > > > > > > least > > > > > > > > > > > > > > > > > > > > through > > > > > > > > > > > > > > > > > > > > previous > > > > > > > > > > > > > > > > > > > > (in list) WeakRef-ed object. So, > > > > > > > > > > > > > > > > > > > > behaviour I > > > > > > > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > > > > > observed > > > > > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > because of some thousands of > > > > > > > > > > > > > > > > > > > > freeable objects, > > > > > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > > one > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > missed > > > > > > > > > > > > > > > > > > > > too > > > > > > > > > > > > > > > > > > > > often. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I have other object being > > > > > > > > > > > > > > > > > > > > cleaned and counted... > > > > > > > > > > > > > > > > > > > > There > > > > > > > > > > > > > > > > > > > > also > > > > > > > > > > > > > > > > > > > > cleanup > > > > > > > > > > > > > > > > > > > > invocation looks like it's > > > > > > > > > > > > > > > > > > > > lagging too much. > > > > > > > > > > > > > > > > > > > > I'll > > > > > > > > > > > > > > > > > > > > come > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > numbers > > > > > > > > > > > > > > > > > > > > after I have more tests. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > dd > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, 2008-03-11 at 09:50 > > > > > > > > > > > > > > > > > > > > -0400, Tony Hosking > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Sounds like we need some sort > > > > > > > > > > > > > > > > > > > > > of testcase. > > > > > > > > > > > > > > > > > > > > > Would > > > > > > > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > able > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > devise one? It will be hard > > > > > > > > > > > > > > > > > > > > > to make it > > > > > > > > > > > > > > > > > > > > > deterministic, > > > > > > > > > > > > > > > > > > > > > but > > > > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > least we > > > > > > > > > > > > > > > > > > > > > should see a non-zero cleanup > > > > > > > > > > > > > > > > > > > > > count. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Antony Hosking | Associate > > > > > > > > > > > > > > > > > > > > > Professor | > > > > > > > > > > > > > > > > > > > > > Computer > > > > > > > > > > > > > > > > > > > > > Science | > > > > > > > > > > > > > > > > > > > > > Purdue > > > > > > > > > > > > > > > > > > > > > University > > > > > > > > > > > > > > > > > > > > > 305 N. University Street | > > > > > > > > > > > > > > > > > > > > > West Lafayette | IN > > > > > > > > > > > > > > > > > > > > > 47907 | > > > > > > > > > > > > > > > > > > > > > USA > > > > > > > > > > > > > > > > > > > > > Office +1 765 494 6001 | > > > > > > > > > > > > > > > > > > > > > Mobile +1 765 427 > > > > > > > > > > > > > > > > > > > > > 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 11, 2008, at 4:03 AM, > > > > > > > > > > > > > > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I have single linked list > > > > > > > > > > > > > > > > > > > > > > that I use to send > > > > > > > > > > > > > > > > > > > > > > messages to > > > > > > > > > > > > > > > > > > > > > > many > > > > > > > > > > > > > > > > > > > > > > threads. > > > > > > > > > > > > > > > > > > > > > > It's end is locked for > > > > > > > > > > > > > > > > > > > > > > addition, and current > > > > > > > > > > > > > > > > > > > > > > end > > > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > given to > > > > > > > > > > > > > > > > > > > > > > each > > > > > > > > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > > > > > > client connecting. This way, > > > > > > > > > > > > > > > > > > > > > > all client are > > > > > > > > > > > > > > > > > > > > > > going > > > > > > > > > > > > > > > > > > > > > > towards > > > > > > > > > > > > > > > > > > > > > > end of > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > list so it's head becomes > > > > > > > > > > > > > > > > > > > > > > unreferenced and > > > > > > > > > > > > > > > > > > > > > > goes > > > > > > > > > > > > > > > > > > > > > > away > > > > > > > > > > > > > > > > > > > > > > by > > > > > > > > > > > > > > > > > > > > > > GC. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Except it does not. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I've added WeakRef cleanup > > > > > > > > > > > > > > > > > > > > > > and sequential > > > > > > > > > > > > > > > > > > > > > > id's > > > > > > > > > > > > > > > > > > > > > > so I > > > > > > > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > > > > record > > > > > > > > > > > > > > > > > > > > > > maximum > > > > > > > > > > > > > > > > > > > > > > freed id for checking. No > > > > > > > > > > > > > > > > > > > > > > single cleanup > > > > > > > > > > > > > > > > > > > > > > happens. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'll cross check my code and > > > > > > > > > > > > > > > > > > > > > > count my > > > > > > > > > > > > > > > > > > > > > > references > > > > > > > > > > > > > > > > > > > > > > from > > > > > > > > > > > > > > > > > > > > > > application > > > > > > > > > > > > > > > > > > > > > > side, > > > > > > > > > > > > > > > > > > > > > > but maybe someone else has > > > > > > > > > > > > > > > > > > > > > > similar problems? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > dd > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > -- > > > > > > Dragi?a Duri? > > > > > > > > > -- > > > > Dragi?a Duri? > > > > > -- > > Dragi?a Duri? > > > -- Dragi?a Duri? From dragisha at m3w.org Fri Mar 14 09:25:39 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Fri, 14 Mar 2008 09:25:39 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <1205477157.32152.189.camel@hias.m3w.org> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> <1205425734.32152.144.camel@hias.m3w.org> <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> <1205430199.32152.152.camel@hias.m3w.org> <1205444774.32152.158.camel@hias.m3w.org> <5649BA54-5B2F-423C-AEF7-65FF09CF3763@cs.purdue.edu> <1205446178.32152.162.camel@hias.m3w.org> <4DD89DC7-FA3C-4B61-A2A1-5C8488CF3A3B@cs.purdue.edu> <1205446902.32152.168.camel@hias.m3w.org> <2EA2C202-F053-4AE8-BA60-E51A1A339C6B@cs.purdue.edu> <1205449510.32152.171.camel@hi! as.m3w.org> <3F319520-7A7E-4DE3-A467-3712EBB8AA8F@cs.purdue.edu> <1205477157.32152.189.camel@hias.m3w.org> Message-ID: <1205483139.32152.194.camel@hias.m3w.org> (timezones, :) On Fri, 2008-03-14 at 07:45 +0100, Dragi?a Duri? wrote: ... > My understanding of conservativeness of GC with regard to register and > stack content was something else... I was thinking it's like this: > ProcessStacks will process registers and stacks; each time > address-sized, correctly-aligned bit pattern which looks like address of > some object GC knows occurs, GC accepts it as it's address - it maybe is > not, but conservative thinking will not make risks there. This way, > ProcessStacks will know exactly what objects are (maybe, but no > risks...) referenced from stack. > > I know I am missing something here, and that's why I ask, why don't you > do it just like this? I saw why... And I think I have some ideas here. When I finish reading your RTCollector.m3 I'll come back to this subject. -- Dragi?a Duri? From hosking at cs.purdue.edu Fri Mar 14 16:08:04 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 14 Mar 2008 11:08:04 -0400 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <1205477157.32152.189.camel@hias.m3w.org> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> <1205425734.32152.144.camel@hias.m3w.org> <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> <1205430199.32152.152.camel@hias.m3w.org> <1205444774.32152.158.camel@hias.m3w.org> <5649BA54-5B2F-423C-AEF7-65FF09CF3763@cs.purdue.edu> <1205446178.32152.162.camel@hias.m3w.org> <4DD89DC7-FA3C-4B61-A2A1-5C8488CF3A3B@cs.purdue.edu> <1205446902.32152.168.camel@hias.m3w.org> <2EA2C202-F053-4AE8-BA60-E51A1A339C6B@cs.purdue.edu> <1205449510.32152.171.camel@hi! ! as.m3w.org> <3F319520-7A7E-4DE3-A467-3712EBB8AA8F@cs.purdue.edu> <1205477157.32152.189.camel@hias.m3w.org> Message-ID: On Mar 14, 2008, at 2:45 AM, Dragi?a Duri? wrote: > [07:29:28] 0 users, 1 ops and 0 unresolved; minId = 44906; > thrs = 1 > [07:29:28] started = 2201; cleaned = 2199; nilled = 2200; > joined = 2200 > [07:29:28] Dangling thrs-a-b-n = 0 > [07:29:28] Live queue end is 44909 (head = 44906), > maxClean = 44897, cleaned = 44897 > > After all keep-pointers-off-stack, one thing that remained was long > living closures... And I killed them all and what happened was total > confirmation of your explanation... One Client.T that remained is most > probably pinned on page with listener thread's closure object. Great! > My understanding of conservativeness of GC with regard to register and > stack content was something else... I was thinking it's like this: > ProcessStacks will process registers and stacks; each time > address-sized, correctly-aligned bit pattern which looks like > address of > some object GC knows occurs, GC accepts it as it's address - it > maybe is > not, but conservative thinking will not make risks there. This way, > ProcessStacks will know exactly what objects are (maybe, but no > risks...) referenced from stack. The problem can arise that the compiler optimizes the code so that you get a pointer from the stack that may refer one word past the object it refers to (e.g., for looping through array elements). I'm not sure if it actually occurs, but it might. > I know I am missing something here, and that's why I ask, why don't > you > do it just like this? > > dd > > On Thu, 2008-03-13 at 21:16 -0400, Tony Hosking wrote: >> http://www.cs.purdue.edu/homes/hosking/papers/ismm06.pdf >> >> >> On Mar 13, 2008, at 7:05 PM, Dragi?a Duri? wrote: >> >>> Is your work on GC covered somewhere else except source, comments >>> and >>> this list? >>> >>> dd >>> >>> On Thu, 2008-03-13 at 18:35 -0400, Tony Hosking wrote: >>>> Ah, the fact that you needed to be aggressive suggests that >>>> conservatism may be the culprit. I keep meaning one day to make >>>> sure >>>> that we do not can objects on pinned pages unless we know that >>>> object >>>> is definitely alive. >>>> >>>> On Mar 13, 2008, at 6:21 PM, Dragi?a Duri? wrote: >>>> >>>>> I see... What bugs me now is a fact about my flow... I am >>>>> already >>>>> doing >>>>> this, and there is always few thousand objects uncleaned from of >>>>> type >>>>> Message... I'll follow this through, to crosscheck everything >>>>> again. >>>>> >>>>> What is nice, after I made this agressive RTCollector.Collect >>>>> "policy", >>>>> my memory footprint remains same for many hours... It was >>>>> swelling >>>>> before. >>>>> >>>>> Thanks, >>>>> dd >>>>> >>>>> On Thu, 2008-03-13 at 18:15 -0400, Tony Hosking wrote: >>>>>> No, when a page is pinned because one of its objects is >>>>>> directly >>>>>> referenced from a thread stack then all objects on that page, >>>>>> both >>>>>> live and dead, are retained. The collector does not move >>>>>> objects to >>>>>> or from pinned pages. The objects on the page can only be >>>>>> freed when >>>>>> the page is no longer pinned by a direct reference from any >>>>>> thread >>>>>> stack. Moral of the story is to NIL references aggressively >>>>>> in local >>>>>> variables. >>>>>> >>>>>> On Mar 13, 2008, at 6:09 PM, Dragi?a Duri? wrote: >>>>>> >>>>>>> When object becomes unreachable, and it sits on page with >>>>>>> other >>>>>>> reachable objects, it's pinned? >>>>>>> >>>>>>> What I asked is... As garbage collector moves objects >>>>>>> around, would >>>>>>> it >>>>>>> pack reachable objects over unreachable ones in these pages? >>>>>>> Thus >>>>>>> kicking them out of pages where they're pinned? >>>>>>> >>>>>>> dd >>>>>>> >>>>>>> On Thu, 2008-03-13 at 18:03 -0400, Tony Hosking wrote: >>>>>>>> I'm not sure I understand what you mean? >>>>>>>> >>>>>>>> On Mar 13, 2008, at 5:46 PM, Dragi?a Duri? wrote: >>>>>>>> >>>>>>>>> Are objects on pages prepacked/moved? >>>>>>>>> >>>>>>>>> On Thu, 2008-03-13 at 17:41 -0400, Tony Hosking wrote: >>>>>>>>>> The thing about conservatism is that even if you NIL >>>>>>>>>> *all* >>>>>>>>>> references >>>>>>>>>> to an object that just happens to lie on a pinned >>>>>>>>>> page, that >>>>>>>>>> object >>>>>>>>>> (and objects reachable from it) will be retained until >>>>>>>>>> the page >>>>>>>>>> is no >>>>>>>>>> longer pinned. >>>>>>>>>> >>>>>>>>>> Antony Hosking | Associate Professor | Computer >>>>>>>>>> Science | Purdue >>>>>>>>>> University >>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | >>>>>>>>>> USA >>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mar 13, 2008, at 1:43 PM, Dragi?a Duri? wrote: >>>>>>>>>> >>>>>>>>>>> I understand transitiveness :), but I've made big >>>>>>>>>>> effort to >>>>>>>>>>> escape >>>>>>>>>>> that. >>>>>>>>>>> >>>>>>>>>>> Lifetime of thread equals lifetime of Client.T >>>>>>>>>>> instance, and >>>>>>>>>>> that >>>>>>>>>>> instance is kept in thread's closure, and also in >>>>>>>>>>> local >>>>>>>>>>> variable. >>>>>>>>>>> Both >>>>>>>>>>> are set to NIL before thread termination - all >>>>>>>>>>> terminations and >>>>>>>>>>> joins >>>>>>>>>>> happen and are counted and shown. >>>>>>>>>>> >>>>>>>>>>> Pointers to Message objects are kept in two places. >>>>>>>>>>> Hub.T keeps >>>>>>>>>>> pointer >>>>>>>>>>> to list tail, and each Client.T contains pointer to >>>>>>>>>>> it's current >>>>>>>>>>> element. That pointer is forwarded as Message's are >>>>>>>>>>> added to >>>>>>>>>>> end. No >>>>>>>>>>> temporary variable, not even WITH substitution is >>>>>>>>>>> used. And >>>>>>>>>>> these >>>>>>>>>>> threads are ones dying with their Client.T's. >>>>>>>>>>> >>>>>>>>>>> "head" is new mechanism used to unlink all elements >>>>>>>>>>> with id less >>>>>>>>>>> than >>>>>>>>>>> minimal id used in active Client.T instances. >>>>>>>>>>> Unlinked totally >>>>>>>>>>> and >>>>>>>>>>> all >>>>>>>>>>> as one free for garbage collector to finish them. >>>>>>>>>>> >>>>>>>>>>> I see no transition. >>>>>>>>>>> >>>>>>>>>>> dd >>>>>>>>>>> >>>>>>>>>>> On Thu, 2008-03-13 at 12:45 -0400, Tony Hosking >>>>>>>>>>> wrote: >>>>>>>>>>>> Realizing that conservative roots collectors like >>>>>>>>>>>> ours can >>>>>>>>>>>> retain >>>>>>>>>>>> data >>>>>>>>>>>> unnecessarily. If a thread stack refers to a page >>>>>>>>>>>> that holds a >>>>>>>>>>>> reference that (transitively) refers to some >>>>>>>>>>>> otherwise dead >>>>>>>>>>>> object >>>>>>>>>>>> then that object will be retained. We say that >>>>>>>>>>>> the page is >>>>>>>>>>>> pinned. >>>>>>>>>>>> Smartening up the collector to ignore dead objects >>>>>>>>>>>> in pinned >>>>>>>>>>>> pages >>>>>>>>>>>> is >>>>>>>>>>>> possible, but not currently implemented. >>>>>>>>>>>> >>>>>>>>>>>> Antony Hosking | Associate Professor | Computer >>>>>>>>>>>> Science | >>>>>>>>>>>> Purdue >>>>>>>>>>>> University >>>>>>>>>>>> 305 N. University Street | West Lafayette | IN >>>>>>>>>>>> 47907 | USA >>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mar 13, 2008, at 12:28 PM, Dragi?a Duri? wrote: >>>>>>>>>>>> >>>>>>>>>>>>> [17:25:19] 117 users, 1 ops and 1 >>>>>>>>>>>>> unresolved; >>>>>>>>>>>>> minId = >>>>>>>>>>>>> 18617; thrs = 120 >>>>>>>>>>>>> [17:25:19] started = 663; cleaned = >>>>>>>>>>>>> 452; nilled = >>>>>>>>>>>>> 543; >>>>>>>>>>>>> joined = 543 >>>>>>>>>>>>> [17:25:19] Dangling thrs-a-b-n = 1 >>>>>>>>>>>>> [17:25:19] Live queue end is 18621 >>>>>>>>>>>>> (head = 18616), >>>>>>>>>>>>> maxClean = 18611, cleaned = 16583 >>>>>>>>>>>>> >>>>>>>>>>>>> About one hour uptime, after I added >>>>>>>>>>>>> RTCollector.Collect() >>>>>>>>>>>>> call >>>>>>>>>>>>> every 20 >>>>>>>>>>>>> seconds. >>>>>>>>>>>>> >>>>>>>>>>>>> dd >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> Yes. RTCollectorSRC.FinishCollection; >>>>>>>>>>>>>> RTCollectorSRC.StartCollection. This will >>>>>>>>>>>>>> finish the current >>>>>>>>>>>>>> (possibly minor) collection, and start a major >>>>>>>>>>>>>> collection. >>>>>>>>>>>>>> The >>>>>>>>>>>>>> major >>>>>>>>>>>>>> collection must finish before you can >>>>>>>>>>>>>> guarantee garbage has >>>>>>>>>>>>>> been >>>>>>>>>>>>>> freed, so you may want to >>>>>>>>>>>>>> RTCollectorSRC.FinishCollection. >>>>>>>>>>>>>> This >>>>>>>>>>>>>> sequence is packaged as RTCollector.Collect. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Antony Hosking | Associate Professor | >>>>>>>>>>>>>> Computer Science | >>>>>>>>>>>>>> Purdue >>>>>>>>>>>>>> University >>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN >>>>>>>>>>>>>> 47907 | USA >>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 >>>>>>>>>>>>>> 5484 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Can I force "major collection" ? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> dd >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, 2008-03-13 at 10:08 -0400, Tony >>>>>>>>>>>>>>> Hosking wrote: >>>>>>>>>>>>>>>> Indeed! Can you devise a testcase? Note >>>>>>>>>>>>>>>> that the >>>>>>>>>>>>>>>> generational >>>>>>>>>>>>>>>> collector will retain old objects for some >>>>>>>>>>>>>>>> time until a >>>>>>>>>>>>>>>> major >>>>>>>>>>>>>>>> collection occurs, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Antony Hosking | Associate Professor | >>>>>>>>>>>>>>>> Computer Science | >>>>>>>>>>>>>>>> Purdue >>>>>>>>>>>>>>>> University >>>>>>>>>>>>>>>> 305 N. University Street | West Lafayette >>>>>>>>>>>>>>>> | IN 47907 | USA >>>>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 >>>>>>>>>>>>>>>> 5484 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> And not only that, their stackbase is >>>>>>>>>>>>>>>>> also set to NIL, >>>>>>>>>>>>>>>>> meaning >>>>>>>>>>>>>>>>> their >>>>>>>>>>>>>>>>> stack is not regarded in any way during >>>>>>>>>>>>>>>>> future >>>>>>>>>>>>>>>>> collections - >>>>>>>>>>>>>>>>> meaning >>>>>>>>>>>>>>>>> all >>>>>>>>>>>>>>>>> local variables are forgotten once apply >>>>>>>>>>>>>>>>> method returns. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> That way, "spuriousness" is not an >>>>>>>>>>>>>>>>> issue, once thread >>>>>>>>>>>>>>>>> returns? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> dd >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Thu, 2008-03-13 at 09:46 -0400, Tony >>>>>>>>>>>>>>>>> Hosking wrote: >>>>>>>>>>>>>>>>>> They should get unlinked from the >>>>>>>>>>>>>>>>>> global ring. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Antony Hosking | Associate Professor | >>>>>>>>>>>>>>>>>> Computer >>>>>>>>>>>>>>>>>> Science | >>>>>>>>>>>>>>>>>> Purdue >>>>>>>>>>>>>>>>>> University >>>>>>>>>>>>>>>>>> 305 N. University Street | West >>>>>>>>>>>>>>>>>> Lafayette | IN 47907 | >>>>>>>>>>>>>>>>>> USA >>>>>>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 >>>>>>>>>>>>>>>>>> 427 5484 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Mar 13, 2008, at 6:16 AM, Dragi?a >>>>>>>>>>>>>>>>>> Duri? wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> What happens to thread stacks once >>>>>>>>>>>>>>>>>>> threads are >>>>>>>>>>>>>>>>>>> Join-ed? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> dd >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Wed, 2008-03-12 at 11:12 -0400, >>>>>>>>>>>>>>>>>>> Tony Hosking >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>> This is probably a result of >>>>>>>>>>>>>>>>>>>> conservatism for >>>>>>>>>>>>>>>>>>>> references >>>>>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>>>>> thread >>>>>>>>>>>>>>>>>>>> stacks which results in spurious >>>>>>>>>>>>>>>>>>>> retention. It >>>>>>>>>>>>>>>>>>>> may be >>>>>>>>>>>>>>>>>>>> necessary >>>>>>>>>>>>>>>>>>>> to be >>>>>>>>>>>>>>>>>>>> more careful about what references >>>>>>>>>>>>>>>>>>>> are held on the >>>>>>>>>>>>>>>>>>>> stack >>>>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>> threads implementation (and >>>>>>>>>>>>>>>>>>>> elsewhere in your >>>>>>>>>>>>>>>>>>>> code). >>>>>>>>>>>>>>>>>>>> Until we >>>>>>>>>>>>>>>>>>>> diagnose the place where objects >>>>>>>>>>>>>>>>>>>> are being >>>>>>>>>>>>>>>>>>>> retained it >>>>>>>>>>>>>>>>>>>> will be >>>>>>>>>>>>>>>>>>>> hard to >>>>>>>>>>>>>>>>>>>> pinpoint. You should realize that >>>>>>>>>>>>>>>>>>>> weak refs are a >>>>>>>>>>>>>>>>>>>> problem >>>>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>> many >>>>>>>>>>>>>>>>>>>> language implementations, not just >>>>>>>>>>>>>>>>>>>> Modula-3. I >>>>>>>>>>>>>>>>>>>> look >>>>>>>>>>>>>>>>>>>> forward >>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>> hearing more definitively from >>>>>>>>>>>>>>>>>>>> you. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Antony Hosking | Associate >>>>>>>>>>>>>>>>>>>> Professor | Computer >>>>>>>>>>>>>>>>>>>> Science | >>>>>>>>>>>>>>>>>>>> Purdue >>>>>>>>>>>>>>>>>>>> University >>>>>>>>>>>>>>>>>>>> 305 N. University Street | West >>>>>>>>>>>>>>>>>>>> Lafayette | IN >>>>>>>>>>>>>>>>>>>> 47907 | >>>>>>>>>>>>>>>>>>>> USA >>>>>>>>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 >>>>>>>>>>>>>>>>>>>> 765 427 5484 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Mar 12, 2008, at 4:21 AM, >>>>>>>>>>>>>>>>>>>> Dragi?a Duri? wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Problem in my case can be >>>>>>>>>>>>>>>>>>>>> thoroughness of GC... >>>>>>>>>>>>>>>>>>>>> Because >>>>>>>>>>>>>>>>>>>>> it's >>>>>>>>>>>>>>>>>>>>> linked >>>>>>>>>>>>>>>>>>>>> list of WeakRef-ed objects, it's >>>>>>>>>>>>>>>>>>>>> always first >>>>>>>>>>>>>>>>>>>>> element in >>>>>>>>>>>>>>>>>>>>> list to >>>>>>>>>>>>>>>>>>>>> become >>>>>>>>>>>>>>>>>>>>> unreachable first - all others >>>>>>>>>>>>>>>>>>>>> are reachable at >>>>>>>>>>>>>>>>>>>>> least >>>>>>>>>>>>>>>>>>>>> through >>>>>>>>>>>>>>>>>>>>> previous >>>>>>>>>>>>>>>>>>>>> (in list) WeakRef-ed object. So, >>>>>>>>>>>>>>>>>>>>> behaviour I >>>>>>>>>>>>>>>>>>>>> have >>>>>>>>>>>>>>>>>>>>> observed >>>>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>> because of some thousands of >>>>>>>>>>>>>>>>>>>>> freeable objects, >>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>> one >>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>> missed >>>>>>>>>>>>>>>>>>>>> too >>>>>>>>>>>>>>>>>>>>> often. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I have other object being >>>>>>>>>>>>>>>>>>>>> cleaned and counted... >>>>>>>>>>>>>>>>>>>>> There >>>>>>>>>>>>>>>>>>>>> also >>>>>>>>>>>>>>>>>>>>> cleanup >>>>>>>>>>>>>>>>>>>>> invocation looks like it's >>>>>>>>>>>>>>>>>>>>> lagging too much. >>>>>>>>>>>>>>>>>>>>> I'll >>>>>>>>>>>>>>>>>>>>> come >>>>>>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>>>>>> numbers >>>>>>>>>>>>>>>>>>>>> after I have more tests. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> dd >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Tue, 2008-03-11 at 09:50 >>>>>>>>>>>>>>>>>>>>> -0400, Tony Hosking >>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>> Sounds like we need some sort >>>>>>>>>>>>>>>>>>>>>> of testcase. >>>>>>>>>>>>>>>>>>>>>> Would >>>>>>>>>>>>>>>>>>>>>> you >>>>>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>> able >>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>> devise one? It will be hard >>>>>>>>>>>>>>>>>>>>>> to make it >>>>>>>>>>>>>>>>>>>>>> deterministic, >>>>>>>>>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>> least we >>>>>>>>>>>>>>>>>>>>>> should see a non-zero cleanup >>>>>>>>>>>>>>>>>>>>>> count. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Antony Hosking | Associate >>>>>>>>>>>>>>>>>>>>>> Professor | >>>>>>>>>>>>>>>>>>>>>> Computer >>>>>>>>>>>>>>>>>>>>>> Science | >>>>>>>>>>>>>>>>>>>>>> Purdue >>>>>>>>>>>>>>>>>>>>>> University >>>>>>>>>>>>>>>>>>>>>> 305 N. University Street | >>>>>>>>>>>>>>>>>>>>>> West Lafayette | IN >>>>>>>>>>>>>>>>>>>>>> 47907 | >>>>>>>>>>>>>>>>>>>>>> USA >>>>>>>>>>>>>>>>>>>>>> Office +1 765 494 6001 | >>>>>>>>>>>>>>>>>>>>>> Mobile +1 765 427 >>>>>>>>>>>>>>>>>>>>>> 5484 >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Mar 11, 2008, at 4:03 AM, >>>>>>>>>>>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I have single linked list >>>>>>>>>>>>>>>>>>>>>>> that I use to send >>>>>>>>>>>>>>>>>>>>>>> messages to >>>>>>>>>>>>>>>>>>>>>>> many >>>>>>>>>>>>>>>>>>>>>>> threads. >>>>>>>>>>>>>>>>>>>>>>> It's end is locked for >>>>>>>>>>>>>>>>>>>>>>> addition, and current >>>>>>>>>>>>>>>>>>>>>>> end >>>>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>> given to >>>>>>>>>>>>>>>>>>>>>>> each >>>>>>>>>>>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>>>>>>>> client connecting. This way, >>>>>>>>>>>>>>>>>>>>>>> all client are >>>>>>>>>>>>>>>>>>>>>>> going >>>>>>>>>>>>>>>>>>>>>>> towards >>>>>>>>>>>>>>>>>>>>>>> end of >>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>> list so it's head becomes >>>>>>>>>>>>>>>>>>>>>>> unreferenced and >>>>>>>>>>>>>>>>>>>>>>> goes >>>>>>>>>>>>>>>>>>>>>>> away >>>>>>>>>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>>>>>> GC. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Except it does not. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I've added WeakRef cleanup >>>>>>>>>>>>>>>>>>>>>>> and sequential >>>>>>>>>>>>>>>>>>>>>>> id's >>>>>>>>>>>>>>>>>>>>>>> so I >>>>>>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>>>>> record >>>>>>>>>>>>>>>>>>>>>>> maximum >>>>>>>>>>>>>>>>>>>>>>> freed id for checking. No >>>>>>>>>>>>>>>>>>>>>>> single cleanup >>>>>>>>>>>>>>>>>>>>>>> happens. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I'll cross check my code and >>>>>>>>>>>>>>>>>>>>>>> count my >>>>>>>>>>>>>>>>>>>>>>> references >>>>>>>>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>>>>>>>> application >>>>>>>>>>>>>>>>>>>>>>> side, >>>>>>>>>>>>>>>>>>>>>>> but maybe someone else has >>>>>>>>>>>>>>>>>>>>>>> similar problems? >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> dd >>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Dragi?a Duri? >>>>>>>> >>>>>>> -- >>>>>>> Dragi?a Duri? >>>>>> >>>>> -- >>>>> Dragi?a Duri? >>>> >>> -- >>> Dragi?a Duri? >>> >> > -- > Dragi?a Duri? From dragisha at m3w.org Sat Mar 15 10:29:39 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sat, 15 Mar 2008 10:29:39 +0100 Subject: [M3devel] WeakRef mechanism... In-Reply-To: References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> <1205425734.32152.144.camel@hias.m3w.org> <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> <1205430199.32152.152.camel@hias.m3w.org> <1205444774.32152.158.camel@hias.m3w.org> <5649BA54-5B2F-423C-AEF7-65FF09CF3763@cs.purdue.edu> <1205446178.32152.162.camel@hias.m3w.org> <4DD89DC7-FA3C-4B61-A2A1-5C8488CF3A3B@cs.purdue.edu> <1205446902.32152.168.camel@hias.m3w.org> <2EA2C202-F053-4AE8-BA60-E51A1A339C6B@cs.purdue.edu> <1205449510.32152.171.camel@hi! ! as.m3w.org> <3F319520-7A7E-4DE3-A467-3712EBB8AA8F@cs.purdue.edu> <1205477157.32152.189.camel@hias.m3w.org> Message-ID: <1205573379.32152.207.camel@hias.m3w.org> On Fri, 2008-03-14 at 11:08 -0400, Tony Hosking wrote: > On Mar 14, 2008, at 2:45 AM, Dragi?a Duri? wrote: > > > My understanding of conservativeness of GC with regard to register and > > stack content was something else... I was thinking it's like this: > > ProcessStacks will process registers and stacks; each time > > address-sized, correctly-aligned bit pattern which looks like > > address of > > some object GC knows occurs, GC accepts it as it's address - it > > maybe is > > not, but conservative thinking will not make risks there. This way, > > ProcessStacks will know exactly what objects are (maybe, but no > > risks...) referenced from stack. > > The problem can arise that the compiler optimizes the code so that you > get a pointer from the stack that may refer one word past the object > it refers to (e.g., for looping through array elements). I'm not sure > if it actually occurs, but it might. I understand this, and also point from your GC paper about fields as VAR args... Address in regs/stack can be one inside our object, not only object's address. But, this still means we can know which object is pointed by ambiguous root. It's not only ambiguous root heap page identification (as is now), there are some address compares. IIRC, in your paper you stated ambiguously referenced pages are only a couple per round of GC, so it probably will not be performance hit at all, and we'll get nearly perfect WeakRef situation. dd -- Dragi?a Duri? From dragisha at m3w.org Sat Mar 15 15:46:16 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sat, 15 Mar 2008 15:46:16 +0100 Subject: [M3devel] SVN migration Message-ID: <1205592376.29245.11.camel@hias.m3w.org> Why don't we migrate to SVN? It's piece of cake to convert CVS repo, and IIRC it can be done incrementally. What I think would be real benefit is trac - we can do some real cooperation and documenting using it. CVSup is real nice tool, but whoever needs local repo can have it using rsync. I don't think it would make big oberhead anywhere, not these days and this hardware/bandwidth we usually have today. dd -- Dragi?a Duri? From wagner at elegosoft.com Sat Mar 15 19:03:00 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Sat, 15 Mar 2008 19:03:00 +0100 Subject: [M3devel] SVN migration In-Reply-To: <1205592376.29245.11.camel@hias.m3w.org> References: <1205592376.29245.11.camel@hias.m3w.org> Message-ID: <20080315190300.xc13iviopws0ocoo@mail.elegosoft.com> Quoting Dragi?a Duri? : > Why don't we migrate to SVN? > > It's piece of cake to convert CVS repo, and IIRC it can be done > incrementally. See https://mail.elegosoft.com/pipermail/m3devel/2008-January/000803.html and https://mail.elegosoft.com/pipermail/m3devel/2008-January/000855.html > What I think would be real benefit is trac - we can do > some real cooperation and documenting using it. There already is a trac installation on www.elegosoft.com to which some old PRs from our GNATs system have been copied. It's not really integrated to CVS and tinderbox yet; I haven't had time to really test it since our system administrator set it up. So there's still some work to do. I'll talk to Ronny on monday if we can open trac write access for all M3 users. It's at https://bugs.elego.de/cm3/. > CVSup is real nice tool, but whoever needs local repo can have it using > rsync. I don't think it would make big oberhead anywhere, not these days > and this hardware/bandwidth we usually have today. But it would break existing infrastructure. I for one haven't got time to spent on setting up something different again. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From dragisha at m3w.org Sun Mar 16 00:29:50 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 16 Mar 2008 00:29:50 +0100 Subject: [M3devel] SVN migration In-Reply-To: <20080315190300.xc13iviopws0ocoo@mail.elegosoft.com> References: <1205592376.29245.11.camel@hias.m3w.org> <20080315190300.xc13iviopws0ocoo@mail.elegosoft.com> Message-ID: <1205623790.29245.19.camel@hias.m3w.org> Saw. Read. SVN is sort of mature, finished CVS. I am 100% sure I don't have your SCM experience, but if SVN's only advantage was trac integration I think it would be more than enough reason to move to it. Trac's got wiki for documentation, very granular access control, nice hyperlinking mechanism for changesets-wiki-tickets-... Whoever gets a bit of free time and inspiration, with trac he can contribute to cm3 in some way. We definitely can have better flow of information with it. I don't know much what you mean under "existing infrastructure". Building/testing insfrastructure? Access to CVS repo? dd On Sat, 2008-03-15 at 19:03 +0100, Olaf Wagner wrote: > Quoting Dragi?a Duri? : > > > Why don't we migrate to SVN? > > > > It's piece of cake to convert CVS repo, and IIRC it can be done > > incrementally. > > See https://mail.elegosoft.com/pipermail/m3devel/2008-January/000803.html > and https://mail.elegosoft.com/pipermail/m3devel/2008-January/000855.html > > > What I think would be real benefit is trac - we can do > > some real cooperation and documenting using it. > > There already is a trac installation on www.elegosoft.com to > which some old PRs from our GNATs system have been copied. It's > not really integrated to CVS and tinderbox yet; I haven't had time > to really test it since our system administrator set it up. So > there's still some work to do. > > I'll talk to Ronny on monday if we can open trac write access > for all M3 users. > > It's at https://bugs.elego.de/cm3/. > > > CVSup is real nice tool, but whoever needs local repo can have it using > > rsync. I don't think it would make big oberhead anywhere, not these days > > and this hardware/bandwidth we usually have today. > > But it would break existing infrastructure. I for one haven't got time > to spent on setting up something different again. > > Olaf -- Dragi?a Duri? From dragisha at m3w.org Sun Mar 16 09:04:53 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 16 Mar 2008 09:04:53 +0100 Subject: [M3devel] SVN migration, also promotion In-Reply-To: <1205623790.29245.19.camel@hias.m3w.org> References: <1205592376.29245.11.camel@hias.m3w.org> <20080315190300.xc13iviopws0ocoo@mail.elegosoft.com> <1205623790.29245.19.camel@hias.m3w.org> Message-ID: <1205654693.29245.34.camel@hias.m3w.org> And, I forgot to mention probable impact on public with more organized development of CM3. Most people, and I have few friends from C++/D world who look regularly at Modula-3, don't see our little community as too organized force. OSS world moved on since CVS became widespread. Separated web, no wiki, no tickets. Source history surfable with one tool, web (best looking in lynx, don't even try something else:) built with other, no real effort to document in depth. Few weeks ago there was mention about possible promotion time. With state of Modula-3 system we are all aware of, it is excellent idea. Only problem is - other people must collect plethora of information from plethora of sources. 99% of them will stop at first obstacle, and we have a lot of them ready. One example.. Yesterday, I came over Mr. Hosking's commit log on m3devel archive. All nice - except text went off right side of my screen some two meters. Most people will pass on - one more unreadable mail archive on the web. I've used CVS to read it, of course. But most people will not know alternatives, or not care - why would they if we don't care?? Similiar communities are spoiling their "followers", but we Modula-3 people are "it's all out there, you are not getting it? Pity!". dd On Sun, 2008-03-16 at 00:29 +0100, Dragi?a Duri? wrote: > Saw. Read. > > SVN is sort of mature, finished CVS. I am 100% sure I don't have your > SCM experience, but if SVN's only advantage was trac integration I think > it would be more than enough reason to move to it. > > Trac's got wiki for documentation, very granular access control, nice > hyperlinking mechanism for changesets-wiki-tickets-... Whoever gets a > bit of free time and inspiration, with trac he can contribute to cm3 in > some way. We definitely can have better flow of information with it. > > I don't know much what you mean under "existing infrastructure". > Building/testing insfrastructure? Access to CVS repo? > > dd > > On Sat, 2008-03-15 at 19:03 +0100, Olaf Wagner wrote: > > Quoting Dragi?a Duri? : > > > > > Why don't we migrate to SVN? > > > > > > It's piece of cake to convert CVS repo, and IIRC it can be done > > > incrementally. > > > > See https://mail.elegosoft.com/pipermail/m3devel/2008-January/000803.html > > and https://mail.elegosoft.com/pipermail/m3devel/2008-January/000855.html > > > > > What I think would be real benefit is trac - we can do > > > some real cooperation and documenting using it. > > > > There already is a trac installation on www.elegosoft.com to > > which some old PRs from our GNATs system have been copied. It's > > not really integrated to CVS and tinderbox yet; I haven't had time > > to really test it since our system administrator set it up. So > > there's still some work to do. > > > > I'll talk to Ronny on monday if we can open trac write access > > for all M3 users. > > > > It's at https://bugs.elego.de/cm3/. > > > > > CVSup is real nice tool, but whoever needs local repo can have it using > > > rsync. I don't think it would make big oberhead anywhere, not these days > > > and this hardware/bandwidth we usually have today. > > > > But it would break existing infrastructure. I for one haven't got time > > to spent on setting up something different again. > > > > Olaf -- Dragi?a Duri? From stsp at elego.de Sun Mar 16 12:56:47 2008 From: stsp at elego.de (Stefan Sperling) Date: Sun, 16 Mar 2008 12:56:47 +0100 Subject: [M3devel] SVN migration, also promotion In-Reply-To: <1205654693.29245.34.camel@hias.m3w.org> References: <1205592376.29245.11.camel@hias.m3w.org> <20080315190300.xc13iviopws0ocoo@mail.elegosoft.com> <1205623790.29245.19.camel@hias.m3w.org> <1205654693.29245.34.camel@hias.m3w.org> Message-ID: <20080316115647.GA2870@ted.stsp.name> On Sun, Mar 16, 2008 at 09:04:53AM +0100, Dragi?a Duri? wrote: > And, I forgot to mention probable impact on public with more organized > development of CM3. Hey Dragi?a, if your only concern is having something like trac available for cm3, (and you don't really want SVN for all its other advantages over CVS) have you checked out http://www.cvstrac.org/ ? I have not looked at it closely, but if it matches your needs, I guess it could be added to the current infrastructure without major headaches (I would volunteer to help elego admins set this up if necessary). The installation instructions don't look too complicated (oh, if only this was the case for cm3 as well :) > Most people, and I have few friends from C++/D world > who look regularly at Modula-3, don't see our little community as too > organized force. OSS world moved on since CVS became widespread. I concur, that is part of the problem of low cm3 adaption imho. Other parts are the arcane installation system and requirements, even though the "everything goes in one folder"-install seems to get more-and-more widespread again in the unix world these days (see mac os and pcbsd). But it's very hard to package cm3 for a current Linux distribution or BSD system while complying with their packaging guidelines. cm3 is still showing its non-FOSS roots here. -- Stefan Sperling Software Developer elego Software Solutions GmbH HRB 77719 Gustav-Meyer-Allee 25, Gebaeude 12 Tel: +49 30 23 45 86 96 13355 Berlin Fax: +49 30 23 45 86 95 http://www.elego.de Geschaeftsfuehrer: Olaf Wagner -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available URL: From wagner at elegosoft.com Sun Mar 16 13:44:25 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Sun, 16 Mar 2008 13:44:25 +0100 Subject: [M3devel] SVN migration, also promotion In-Reply-To: <1205654693.29245.34.camel@hias.m3w.org> References: <1205592376.29245.11.camel@hias.m3w.org> <20080315190300.xc13iviopws0ocoo@mail.elegosoft.com> <1205623790.29245.19.camel@hias.m3w.org> <1205654693.29245.34.camel@hias.m3w.org> Message-ID: <20080316134425.bog0v1yz48o8ckss@mail.elegosoft.com> Quoting Dragi?a Duri? : > And, I forgot to mention probable impact on public with more organized > development of CM3. Most people, and I have few friends from C++/D world > who look regularly at Modula-3, don't see our little community as too > organized force. OSS world moved on since CVS became widespread. > Separated web, no wiki, no tickets. Source history surfable with one > tool, web (best looking in lynx, don't even try something else:) built > with other, no real effort to document in depth. > > Few weeks ago there was mention about possible promotion time. With > state of Modula-3 system we are all aware of, it is excellent idea. Only > problem is - other people must collect plethora of information from > plethora of sources. 99% of them will stop at first obstacle, and we > have a lot of them ready. > > One example.. Yesterday, I came over Mr. Hosking's commit log on m3devel > archive. All nice - except text went off right side of my screen some > two meters. Most people will pass on - one more unreadable mail archive > on the web. I've used CVS to read it, of course. But most people will > not know alternatives, or not care - why would they if we don't care?? > > Similiar communities are spoiling their "followers", but we Modula-3 > people are "it's all out there, you are not getting it? Pity!". I think you don't really realize how much effort has been put into getting everything into its current state and keeping the quality standard as best as possible. This is because many people have contributed their time and work and other resources. Elego, for example, has contributed its machines, network bandwidth, administration and support, and quite a few months of student's work; I myself have contributed months of time to get the 5th release of CM3 running first on one and then on more and more platforms since we got the sources from Critical Mass. In recent months I have been working on improving both the test infrastructure and documentation/presentation of CM3, while trying to support all other contributors as best as possible and coordinating their achievements. I don't think there are a `plethora of sources'; as a matter of fact, all resources are available via www.opencm3.net and being integrated more and more. There is the web presentation, the CVS repository with anonymous and ssh access, CVSWeb, mailing lists, tinderbox regression testing, trac (still in testing), etc. Soon coming are contributions of CM3_IDE (which is a development environment that really integrates different kinds of resources, and already running on some systems, but not yet being approved for release by the owner) and packages for CM3 integration with advanced version control and `project' management. All this needs time and a lot of work. I don't think it will be furthered by switching from CVS to SVN or any other tool change. I'm a bit tired of these discussions, as I don't think they help the project on. What is needed is concrete work on documentation and infrastructure. I'd also doubt that switching to SVN would bring about a group of motivated and competent contributors who have just been waiting for the CM3 folks to change their version control to participate in the development of Modula-3. If there really is the wish within the community to switch from CVS to SVN, then those interested should build a team, investigate the impacts and propose a migration plan. Elego and me for one would need new server setup, the actual repository migration (which should preserve as much information as possible), setup of repository replication to two machines, integration of SVN into the regression test framework, setup of SVN clients, update / rewrite of all the repository access information and CM guidelines, and a migration strategy for all the existing HTTP links in search engines etc. If I should make an estimation for this project in a commercial environment, I'd say it will take 3-6 months and cost a client between 40,000 and 80,000 Euros (that's currently 60,000 to 120,000$). I'd rather invest these efforts for other goals. If other CM3 users think it is worthwhile to do, they should just do it; after all, this is an open source project. No tool will be able to bring about more organized development of CM3 simply by its use though. It needs willing and competent humans to do this. I should have used the time it took me to write this mail for working on CM3 code and infrastructure. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From hosking at cs.purdue.edu Sun Mar 16 16:40:49 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 16 Mar 2008 11:40:49 -0400 Subject: [M3devel] SVN migration, also promotion In-Reply-To: <20080316134425.bog0v1yz48o8ckss@mail.elegosoft.com> References: <1205592376.29245.11.camel@hias.m3w.org> <20080315190300.xc13iviopws0ocoo@mail.elegosoft.com> <1205623790.29245.19.camel@hias.m3w.org> <1205654693.29245.34.camel@hias.m3w.org> <20080316134425.bog0v1yz48o8ckss@mail.elegosoft.com> Message-ID: <3EFCE04A-FA5D-4EC5-97F5-9AF333D51D48@cs.purdue.edu> For a small community of volunteers (I don't think any of us are making any money actually maintaining CM3), I don't think we are doing too badly. We are much smaller than other projects I am involved in (jikesrvm.org for example), which make about the same degree of forward progress as the CM3 community is doing. So, I applaud the efforts of everyone involved. The new regression framework is a *huge* improvement in preserving, maintaining, and enhancing the reliability of CM3 on many more platforms than most other projects support. If someone (I don't have the time) would like to explore migration to SVN and trac I would not object (I happily use both these systems in other projects), but the biggest piece of work in any CM3 transition will be to preserve what we have without too much turmoil. I am leery of making any move that will fatally disturb the fine work everyone is doing. I am sure Dragi?a meant his suggestions to be constructive criticism rather than any kind of attack on individuals or decisions made over the long history of the CM3 project. Remember, this project has its roots in a system that was started almost 20 years ago, before anyone knew what open-source or the internet or Web really were. It is a testament to the staying power of the language and implementation that we are still up and running and I think as healthy as we have ever been! I know of very few open-source projects that can claim the length of history that Modula-3 and CM3 have. I encourage everyone to stay involved, channel their enthusiasm, and make progress happen. Just my 2 cents... :-) Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 16, 2008, at 8:44 AM, Olaf Wagner wrote: > Quoting Dragi?a Duri? : > >> And, I forgot to mention probable impact on public with more >> organized >> development of CM3. Most people, and I have few friends from C++/D >> world >> who look regularly at Modula-3, don't see our little community as too >> organized force. OSS world moved on since CVS became widespread. >> Separated web, no wiki, no tickets. Source history surfable with one >> tool, web (best looking in lynx, don't even try something else:) >> built >> with other, no real effort to document in depth. >> >> Few weeks ago there was mention about possible promotion time. With >> state of Modula-3 system we are all aware of, it is excellent idea. >> Only >> problem is - other people must collect plethora of information from >> plethora of sources. 99% of them will stop at first obstacle, and we >> have a lot of them ready. >> >> One example.. Yesterday, I came over Mr. Hosking's commit log on >> m3devel >> archive. All nice - except text went off right side of my screen some >> two meters. Most people will pass on - one more unreadable mail >> archive >> on the web. I've used CVS to read it, of course. But most people will >> not know alternatives, or not care - why would they if we don't >> care?? >> >> Similiar communities are spoiling their "followers", but we Modula-3 >> people are "it's all out there, you are not getting it? Pity!". > > I think you don't really realize how much effort has been put into > getting everything into its current state and keeping the quality > standard as best as possible. This is because many people have > contributed their time and work and other resources. Elego, for > example, has contributed its machines, network bandwidth, > administration > and support, and quite a few months of student's work; I myself have > contributed months of time to get the 5th release of CM3 running first > on one and then on more and more platforms since we got the sources > from Critical Mass. > > In recent months I have been working on improving both the test > infrastructure and documentation/presentation of CM3, while trying to > support all other contributors as best as possible and coordinating > their achievements. > > I don't think there are a `plethora of sources'; as a matter of fact, > all resources are available via www.opencm3.net and being integrated > more and more. There is the web presentation, the CVS repository with > anonymous and ssh access, CVSWeb, mailing lists, tinderbox regression > testing, trac (still in testing), etc. Soon coming are contributions > of CM3_IDE (which is a development environment that really integrates > different kinds of resources, and already running on some systems, > but not yet being approved for release by the owner) and packages > for CM3 integration with advanced version control and `project' > management. All this needs time and a lot of work. > > I don't think it will be furthered by switching from CVS to SVN > or any other tool change. > > I'm a bit tired of these discussions, as I don't think they help > the project on. What is needed is concrete work on documentation > and infrastructure. I'd also doubt that switching to SVN would > bring about a group of motivated and competent contributors > who have just been waiting for the CM3 folks to change their > version control to participate in the development of Modula-3. > > If there really is the wish within the community to switch from > CVS to SVN, then those interested should build a team, investigate > the impacts and propose a migration plan. Elego and me for one > would need new server setup, the actual repository migration (which > should preserve as much information as possible), setup of repository > replication to two machines, integration of SVN into the regression > test framework, setup of SVN clients, update / rewrite of all the > repository access information and CM guidelines, and a migration > strategy for all the existing HTTP links in search engines etc. > > If I should make an estimation for this project in a commercial > environment, I'd say it will take 3-6 months and cost a client > between 40,000 and 80,000 Euros (that's currently 60,000 to 120,000$). > I'd rather invest these efforts for other goals. If other CM3 users > think it is worthwhile to do, they should just do it; after all, this > is an open source project. No tool will be able to bring about more > organized development of CM3 simply by its use though. It needs > willing and competent humans to do this. > > I should have used the time it took me to write this mail for working > on CM3 code and infrastructure. > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, > Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 > 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: > Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: > DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Sun Mar 16 16:53:11 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 16 Mar 2008 11:53:11 -0400 Subject: [M3devel] Recent tinderbox problems Message-ID: <29D7616A-22D8-47BE-A71F-B4216406D2F8@cs.purdue.edu> There has been a big rise in tinderbox errors since the LONGINT formatting routines were added to Fmt.i3. This is because stubgen does not recognize LONGINT. I will investigate, but I suspect it is because the m3 utilities don't know about LONGINT (this was on the list of things TODO when I checked in the LONGINT support in the core system). Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Sun Mar 16 18:45:00 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Sun, 16 Mar 2008 18:45:00 +0100 Subject: [M3devel] SVN migration, also promotion In-Reply-To: <3EFCE04A-FA5D-4EC5-97F5-9AF333D51D48@cs.purdue.edu> References: <1205592376.29245.11.camel@hias.m3w.org> <20080315190300.xc13iviopws0ocoo@mail.elegosoft.com> <1205623790.29245.19.camel@hias.m3w.org> <1205654693.29245.34.camel@hias.m3w.org> <20080316134425.bog0v1yz48o8ckss@mail.elegosoft.com> <3EFCE04A-FA5D-4EC5-97F5-9AF333D51D48@cs.purdue.edu> Message-ID: <20080316184500.bcui1d3v4c48skw8@mail.elegosoft.com> Quoting Tony Hosking : > For a small community of volunteers (I don't think any of us are > making any money actually maintaining CM3), I don't think we are doing > too badly. We are much smaller than other projects I am involved in > (jikesrvm.org for example), which make about the same degree of > forward progress as the CM3 community is doing. So, I applaud the > efforts of everyone involved. The new regression framework is a > *huge* improvement in preserving, maintaining, and enhancing the > reliability of CM3 on many more platforms than most other projects > support. If someone (I don't have the time) would like to explore > migration to SVN and trac I would not object (I happily use both these > systems in other projects), but the biggest piece of work in any CM3 > transition will be to preserve what we have without too much turmoil. > I am leery of making any move that will fatally disturb the fine work > everyone is doing. I am sure Dragi?a meant his suggestions to be > constructive criticism rather than any kind of attack on individuals > or decisions made over the long history of the CM3 project. Remember, > this project has its roots in a system that was started almost 20 > years ago, before anyone knew what open-source or the internet or Web > really were. It is a testament to the staying power of the language > and implementation that we are still up and running and I think as > healthy as we have ever been! I know of very few open-source projects > that can claim the length of history that Modula-3 and CM3 have. I > encourage everyone to stay involved, channel their enthusiasm, and > make progress happen. Indeed! Rereading my mail, it sounds a bit too negative; I didn't want to discourage anyone. I'm also very satisfied with the current thriving of the project. Keep up all the good work! Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From wagner at elegosoft.com Sun Mar 16 18:55:54 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Sun, 16 Mar 2008 18:55:54 +0100 Subject: [M3devel] Recent tinderbox problems In-Reply-To: <29D7616A-22D8-47BE-A71F-B4216406D2F8@cs.purdue.edu> References: <29D7616A-22D8-47BE-A71F-B4216406D2F8@cs.purdue.edu> Message-ID: <20080316185554.4azh6claqs8ogco8@mail.elegosoft.com> Quoting Tony Hosking : > There has been a big rise in tinderbox errors since the LONGINT > formatting routines were added to Fmt.i3. This is because stubgen > does not recognize LONGINT. I will investigate, but I suspect it is > because the m3 utilities don't know about LONGINT (this was on the > list of things TODO when I checked in the LONGINT support in the core > system). Yes, all obliq-related packages are broken. The main problem are indeed the new Fmt functions for LONGINT: "/pub/users/m3/work/cm3-inst/new.elego.de/current//pkg/libm3/src /fmtlex/Fmt.i3", line 39,21: Identifier 'LONGINT' not declared "/pub/users/m3/work/cm3-inst/new.elego.de/current//pkg/netobj/sr c/netobj.tmpl", line 37: quake runtime error: exit 1: /pub/users/m3/work/cm3-inst/new.elego.de/current//bin/stubgen -v1 -sno ObValue.RemVar -T.M3IMPTAB Fatal Error: package build failed BTW, the HTML construction for the package status result page does not seem to work properly on SOLgnu. Any ideas what could cause this? It's not high priority to fix though. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From stsp at elego.de Sun Mar 16 19:01:18 2008 From: stsp at elego.de (Stefan Sperling) Date: Sun, 16 Mar 2008 19:01:18 +0100 Subject: [M3devel] SVN migration, also promotion In-Reply-To: <20080316115647.GA2870@ted.stsp.name> References: <1205592376.29245.11.camel@hias.m3w.org> <20080315190300.xc13iviopws0ocoo@mail.elegosoft.com> <1205623790.29245.19.camel@hias.m3w.org> <1205654693.29245.34.camel@hias.m3w.org> <20080316115647.GA2870@ted.stsp.name> Message-ID: <20080316180118.GN2870@ted.stsp.name> On Sun, Mar 16, 2008 at 12:56:47PM +0100, Stefan Sperling wrote: > On Sun, Mar 16, 2008 at 09:04:53AM +0100, Dragi?a Duri? wrote: > > And, I forgot to mention probable impact on public with more organized > > development of CM3. > > Hey Dragi?a, > > if your only concern is having something like trac available for cm3, > (and you don't really want SVN for all its other advantages over CVS) > have you checked out http://www.cvstrac.org/ ? I've been told that several people at elego have already evaluated this option and that it was rejected. I guess Olaf knows the reasons why this tool doesn't meet the requirements. -- Stefan Sperling Software Developer elego Software Solutions GmbH HRB 77719 Gustav-Meyer-Allee 25, Gebaeude 12 Tel: +49 30 23 45 86 96 13355 Berlin Fax: +49 30 23 45 86 95 http://www.elego.de Geschaeftsfuehrer: Olaf Wagner -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available URL: From hosking at cs.purdue.edu Sun Mar 16 19:03:27 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 16 Mar 2008 14:03:27 -0400 Subject: [M3devel] Recent tinderbox problems In-Reply-To: <20080316185554.4azh6claqs8ogco8@mail.elegosoft.com> References: <29D7616A-22D8-47BE-A71F-B4216406D2F8@cs.purdue.edu> <20080316185554.4azh6claqs8ogco8@mail.elegosoft.com> Message-ID: <588C2B24-2254-4C95-B6C9-303D8924D1FC@cs.purdue.edu> I am working on the m3tk support now, and should have a solution in a day or so. I don't know what the HTML script problem is on Solaris, but I am guessing some non-standard sh usage as in the other scripts I fixed. On Mar 16, 2008, at 1:55 PM, Olaf Wagner wrote: > Quoting Tony Hosking : > >> There has been a big rise in tinderbox errors since the LONGINT >> formatting routines were added to Fmt.i3. This is because stubgen >> does not recognize LONGINT. I will investigate, but I suspect it is >> because the m3 utilities don't know about LONGINT (this was on the >> list of things TODO when I checked in the LONGINT support in the core >> system). > > Yes, all obliq-related packages are broken. The main problem are > indeed the new Fmt functions for LONGINT: > > "/pub/users/m3/work/cm3-inst/new.elego.de/current//pkg/libm3/src > /fmtlex/Fmt.i3", line 39,21: Identifier 'LONGINT' not declared > "/pub/users/m3/work/cm3-inst/new.elego.de/current//pkg/netobj/sr > c/netobj.tmpl", line 37: quake runtime error: exit 1: > /pub/users/m3/work/cm3-inst/new.elego.de/current//bin/stubgen > -v1 -sno ObValue.RemVar -T.M3IMPTAB > Fatal Error: package build failed > > BTW, the HTML construction for the package status result page does > not seem to work properly on SOLgnu. Any ideas what could cause this? > It's not high priority to fix though. > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, > Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 > 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: > Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: > DE163214194 > From dragisha at m3w.org Sun Mar 16 19:31:28 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 16 Mar 2008 19:31:28 +0100 Subject: [M3devel] SVN migration, also promotion In-Reply-To: <20080316134425.bog0v1yz48o8ckss@mail.elegosoft.com> References: <1205592376.29245.11.camel@hias.m3w.org> <20080315190300.xc13iviopws0ocoo@mail.elegosoft.com> <1205623790.29245.19.camel@hias.m3w.org> <1205654693.29245.34.camel@hias.m3w.org> <20080316134425.bog0v1yz48o8ckss@mail.elegosoft.com> Message-ID: <1205692288.29245.71.camel@hias.m3w.org> On Sun, 2008-03-16 at 13:44 +0100, Olaf Wagner wrote: > Quoting Dragi?a Duri? : ... > > Similiar communities are spoiling their "followers", but we Modula-3 > > people are "it's all out there, you are not getting it? Pity!". > > I think you don't really realize how much effort has been put into > getting everything into its current state and keeping the quality > standard as best as possible. This is because many people have > contributed their time and work and other resources. Elego, for > example, has contributed its machines, network bandwidth, administration > and support, and quite a few months of student's work; I myself have > contributed months of time to get the 5th release of CM3 running first > on one and then on more and more platforms since we got the sources > from Critical Mass. If criticism produces "you don't realize how much we work" then it would be best for me to stop writing about this and keep it quiet and happy? > > In recent months I have been working on improving both the test > infrastructure and documentation/presentation of CM3, while trying to > support all other contributors as best as possible and coordinating > their achievements. Also, if you don't think I follow M3 closely, then I don't know what to say. Try google. Or check your mailboxes. :) > I don't think there are a `plethora of sources'; as a matter of fact, > all resources are available via www.opencm3.net and being integrated > more and more. Look at any good trac using project, lighttpd for example, and you'll see much less information organized in much better way. You'll see trivial projects, compared to CM3, and see good flow and easy ways to jump in, find info and support, and what not. I am Modula-2 programmer since 1987, Pascal since 1982. Sold my first Pascal software in 1984. Discovered Modula-3 in 1990, and tried to use it for what I live from for years. My company started selling Modula-3 software 10yrs ago. I know very well who's and what's and where's of Modula-3. But try to imagine someone coming outside, and refer him to hard science which makes at least half of Modula-3 docs. > There is the web presentation, the CVS repository with > anonymous and ssh access, CVSWeb, mailing lists, Mailing lists archive was one of thing that trigerred me. I am very glad we have them, but that archive look, feel and usability is as outdated as many other things we have. I am glad we have all of them, and I am thankful to Elego and Mr. Hosking and others for preserving Modula-3's life, but many of our resources are like that kid whose face only mother can think beautiful. ... > I don't think it will be furthered by switching from CVS to SVN > or any other tool change. Ok. > I'm a bit tired of these discussions, as I don't think they help > the project on. What is needed is concrete work on documentation > and infrastructure. I'd also doubt that switching to SVN would > bring about a group of motivated and competent contributors > who have just been waiting for the CM3 folks to change their > version control to participate in the development of Modula-3. I can understand you, as someone from SCM company, are a bit touchy on these topics. What I say, as a longtime OSS user/follower and sometimes contributor, is - Modula-3 train is pretty hard one to jump on. > If there really is the wish within the community to switch from > CVS to SVN, then those interested should build a team, investigate > the impacts and propose a migration plan. Elego and me for one > would need new server setup, the actual repository migration (which > should preserve as much information as possible), setup of repository > replication to two machines, integration of SVN into the regression > test framework, setup of SVN clients, update / rewrite of all the > repository access information and CM guidelines, and a migration > strategy for all the existing HTTP links in search engines etc. > > If I should make an estimation for this project in a commercial > environment, I'd say it will take 3-6 months and cost a client > between 40,000 and 80,000 Euros (that's currently 60,000 to 120,000$). Next time your company gets such job, outsource it to mine - I'll do it for half that money and 1/6 the time :). > I'd rather invest these efforts for other goals. If other CM3 users > think it is worthwhile to do, they should just do it; after all, this > is an open source project. No tool will be able to bring about more > organized development of CM3 simply by its use though. It needs > willing and competent humans to do this. It also can be done other way. As SVN from CVS can be done incrementally, we can make a sync-only SVN repo so locally installed trac can browse/link CM3's SVN. Voila. If I produced this trick in commercial environment, I would be hard pressed to charge it 5,000-10,000 EUR. Too bad nobody'll pay :). > I should have used the time it took me to write this mail for working > on CM3 code and infrastructure. I am sorry you are receiving this so hard. I think it would be really good idea to look a bit what other OSS people are doing and use some fresh ideas. > > Olaf dd -- Dragi?a Duri? From dragisha at m3w.org Sun Mar 16 19:35:54 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 16 Mar 2008 19:35:54 +0100 Subject: [M3devel] Recent tinderbox problems In-Reply-To: <29D7616A-22D8-47BE-A71F-B4216406D2F8@cs.purdue.edu> References: <29D7616A-22D8-47BE-A71F-B4216406D2F8@cs.purdue.edu> Message-ID: <1205692554.29245.74.camel@hias.m3w.org> Ouch. That's all I can say about that little submission of mine. What is solution? Revert submission and kill Fmt.LongReal or wait for stubgen changes? dd On Sun, 2008-03-16 at 11:53 -0400, Tony Hosking wrote: > There has been a big rise in tinderbox errors since the LONGINT > formatting routines were added to Fmt.i3. This is because stubgen > does not recognize LONGINT. I will investigate, but I suspect it is > because the m3 utilities don't know about LONGINT (this was on the > list of things TODO when I checked in the LONGINT support in the core > system). > > > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > -- Dragi?a Duri? From hosking at cs.purdue.edu Sun Mar 16 19:46:40 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 16 Mar 2008 14:46:40 -0400 Subject: [M3devel] Recent tinderbox problems In-Reply-To: <1205692554.29245.74.camel@hias.m3w.org> References: <29D7616A-22D8-47BE-A71F-B4216406D2F8@cs.purdue.edu> <1205692554.29245.74.camel@hias.m3w.org> Message-ID: I'm working on it. It was a known deficiency from my addition of LONGINT to the compiler, neglecting adding support to m3tk, but this is the first time it bit us. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 16, 2008, at 2:35 PM, Dragi?a Duri? wrote: > Ouch. > > That's all I can say about that little submission of mine. > > What is solution? Revert submission and kill Fmt.LongReal or wait for > stubgen changes? > > dd > > On Sun, 2008-03-16 at 11:53 -0400, Tony Hosking wrote: >> There has been a big rise in tinderbox errors since the LONGINT >> formatting routines were added to Fmt.i3. This is because stubgen >> does not recognize LONGINT. I will investigate, but I suspect it is >> because the m3 utilities don't know about LONGINT (this was on the >> list of things TODO when I checked in the LONGINT support in the core >> system). >> >> >> >> Antony Hosking | Associate Professor | Computer Science | Purdue >> University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> >> > -- > Dragi?a Duri? -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Sun Mar 16 20:34:16 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Sun, 16 Mar 2008 20:34:16 +0100 Subject: [M3devel] SVN migration, also promotion In-Reply-To: <1205692288.29245.71.camel@hias.m3w.org> References: <1205592376.29245.11.camel@hias.m3w.org> <20080315190300.xc13iviopws0ocoo@mail.elegosoft.com> <1205623790.29245.19.camel@hias.m3w.org> <1205654693.29245.34.camel@hias.m3w.org> <20080316134425.bog0v1yz48o8ckss@mail.elegosoft.com> <1205692288.29245.71.camel@hias.m3w.org> Message-ID: <20080316203416.kpc6cze4cgsww8c0@mail.elegosoft.com> Quoting Dragi?a Duri? : > If criticism produces "you don't realize how much we work" then it > would be best for me to stop writing about this and keep it quiet and > happy? There is some misunderstanding here. I didn't want to point out how much we have worked, but how much effort any such change as replacing the underlying version control system will involve. >> I'm a bit tired of these discussions, as I don't think they help >> the project on. What is needed is concrete work on documentation >> and infrastructure. I'd also doubt that switching to SVN would >> bring about a group of motivated and competent contributors >> who have just been waiting for the CM3 folks to change their >> version control to participate in the development of Modula-3. > > I can understand you, as someone from SCM company, are a bit touchy on > these topics. What I say, as a longtime OSS user/follower and sometimes > contributor, is - Modula-3 train is pretty hard one to jump on. I'll grant you that, but as others have pointed out, this is rather due to the completely different roots of the project. We'll need to work on it to make CM3 more user-friendly of course. >> If there really is the wish within the community to switch from >> CVS to SVN, then those interested should build a team, investigate >> the impacts and propose a migration plan. Elego and me for one >> would need new server setup, the actual repository migration (which >> should preserve as much information as possible), setup of repository >> replication to two machines, integration of SVN into the regression >> test framework, setup of SVN clients, update / rewrite of all the >> repository access information and CM guidelines, and a migration >> strategy for all the existing HTTP links in search engines etc. >> >> If I should make an estimation for this project in a commercial >> environment, I'd say it will take 3-6 months and cost a client >> between 40,000 and 80,000 Euros (that's currently 60,000 to 120,000$). > > Next time your company gets such job, outsource it to mine - I'll do > it for half that money and 1/6 the time :). If you can really do these projects in 1/6th of our time, we will surely be interested in working with you :-) You should not underestimate the time needed for system analysis, meetings, convincing people and writing documents though ;-) The technical aspects are often marginal compared to the human factor. (Where exactly are you located? Would you be willing to travel? :-)) >> I'd rather invest these efforts for other goals. If other CM3 users >> think it is worthwhile to do, they should just do it; after all, this >> is an open source project. No tool will be able to bring about more >> organized development of CM3 simply by its use though. It needs >> willing and competent humans to do this. > > It also can be done other way. As SVN from CVS can be done > incrementally, we can make a sync-only SVN repo so locally installed > trac can browse/link CM3's SVN. Voila. This sounds like a good idea. Would you be willing to provide a prototype setup for the read-only svn mirror? > If I produced this trick in commercial environment, I would be hard > pressed to charge it 5,000-10,000 EUR. Too bad nobody'll pay :). Not for Modula-3 :-) If such a setup is really needed in an enterprise, they'll certainly pay this price if they haven't got the knowledge to do it on their own. >> I should have used the time it took me to write this mail for working >> on CM3 code and infrastructure. > > I am sorry you are receiving this so hard. I think it would be really > good idea to look a bit what other OSS people are doing and use some > fresh ideas. Of course you are right, it is always a good idea to look what others are doing. And I'm not personally annoyed in any way; just a bit bit tired, as I said. So I hope I haven't quenched your enthusiasm and contributions, Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From dragisha at m3w.org Sun Mar 16 21:22:17 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 16 Mar 2008 21:22:17 +0100 Subject: [M3devel] SVN migration, also promotion In-Reply-To: <20080316203416.kpc6cze4cgsww8c0@mail.elegosoft.com> References: <1205592376.29245.11.camel@hias.m3w.org> <20080315190300.xc13iviopws0ocoo@mail.elegosoft.com> <1205623790.29245.19.camel@hias.m3w.org> <1205654693.29245.34.camel@hias.m3w.org> <20080316134425.bog0v1yz48o8ckss@mail.elegosoft.com> <1205692288.29245.71.camel@hias.m3w.org> <20080316203416.kpc6cze4cgsww8c0@mail.elegosoft.com> Message-ID: <1205698937.29245.93.camel@hias.m3w.org> On Sun, 2008-03-16 at 20:34 +0100, Olaf Wagner wrote: > Quoting Dragi?a Duri? : ... > > > > I can understand you, as someone from SCM company, are a bit touchy on > > these topics. What I say, as a longtime OSS user/follower and sometimes > > contributor, is - Modula-3 train is pretty hard one to jump on. > > I'll grant you that, but as others have pointed out, this is > rather due to the completely different roots of the project. > We'll need to work on it to make CM3 more user-friendly of course. And we're all on same task here. Few years ago, I was telling people my big wish is to see Modula-3 as first class citizen of modern programming language world. Modula-3 is that, now. And every day, more so. ... > > Next time your company gets such job, outsource it to mine - I'll do > > it for half that money and 1/6 the time :). > > If you can really do these projects in 1/6th of our time, we will > surely be interested in working with you :-) You should not underestimate > the time needed for system analysis, meetings, convincing people and > writing documents though ;-) The technical aspects are often marginal > compared to the human factor. (Where exactly are you located? Would > you be willing to travel? :-)) Technical and convicing people is second nature to me :). As for travel, I'll call you when I learn German :). > > >> I'd rather invest these efforts for other goals. If other CM3 users > >> think it is worthwhile to do, they should just do it; after all, this > >> is an open source project. No tool will be able to bring about more > >> organized development of CM3 simply by its use though. It needs > >> willing and competent humans to do this. > > > > It also can be done other way. As SVN from CVS can be done > > incrementally, we can make a sync-only SVN repo so locally installed > > trac can browse/link CM3's SVN. Voila. > > This sounds like a good idea. Would you be willing to provide a > prototype setup for the read-only svn mirror? I'll work on whole thing. Trac setup is easy and it's self contained we can copy it once I have it moving. > > > If I produced this trick in commercial environment, I would be hard > > pressed to charge it 5,000-10,000 EUR. Too bad nobody'll pay :). > > Not for Modula-3 :-) If such a setup is really needed in an enterprise, > they'll certainly pay this price if they haven't got the knowledge to > do it on their own. Of course. I usually don't sell such services, but I know the drill. > > >> I should have used the time it took me to write this mail for working > >> on CM3 code and infrastructure. > > > > I am sorry you are receiving this so hard. I think it would be really > > good idea to look a bit what other OSS people are doing and use some > > fresh ideas. > > Of course you are right, it is always a good idea to look what others > are doing. And I'm not personally annoyed in any way; just a bit > bit tired, as I said. > > So I hope I haven't quenched your enthusiasm and contributions, You are not. Did you read that bit on convincing people? I am pretty used to reactions like yours. They all react like that, at first. :) > > Olaf dd > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -- Dragi?a Duri? From wagner at elegosoft.com Sun Mar 16 22:33:15 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Sun, 16 Mar 2008 22:33:15 +0100 Subject: [M3devel] SVN migration, also promotion In-Reply-To: <1205698937.29245.93.camel@hias.m3w.org> References: <1205592376.29245.11.camel@hias.m3w.org> <20080315190300.xc13iviopws0ocoo@mail.elegosoft.com> <1205623790.29245.19.camel@hias.m3w.org> <1205654693.29245.34.camel@hias.m3w.org> <20080316134425.bog0v1yz48o8ckss@mail.elegosoft.com> <1205692288.29245.71.camel@hias.m3w.org> <20080316203416.kpc6cze4cgsww8c0@mail.elegosoft.com> <1205698937.29245.93.camel@hias.m3w.org> Message-ID: <20080316223315.iu656uoxs0g084ws@mail.elegosoft.com> Quoting Dragi?a Duri? : > On Sun, 2008-03-16 at 20:34 +0100, Olaf Wagner wrote: >> Quoting Dragi?a Duri? : >> >> I'd rather invest these efforts for other goals. If other CM3 users >> >> think it is worthwhile to do, they should just do it; after all, this >> >> is an open source project. No tool will be able to bring about more >> >> organized development of CM3 simply by its use though. It needs >> >> willing and competent humans to do this. >> > >> > It also can be done other way. As SVN from CVS can be done >> > incrementally, we can make a sync-only SVN repo so locally installed >> > trac can browse/link CM3's SVN. Voila. >> >> This sounds like a good idea. Would you be willing to provide a >> prototype setup for the read-only svn mirror? > > I'll work on whole thing. Trac setup is easy and it's self contained we > can copy it once I have it moving. Trac is already installed and working on. You can get access via our system admin if you want to work on it. It's the CVS integration that is lacking. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From hendrik at topoi.pooq.com Mon Mar 17 04:10:40 2008 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Sun, 16 Mar 2008 23:10:40 -0400 Subject: [M3devel] SVN migration, also promotion In-Reply-To: <20080316134425.bog0v1yz48o8ckss@mail.elegosoft.com> References: <1205592376.29245.11.camel@hias.m3w.org> <20080315190300.xc13iviopws0ocoo@mail.elegosoft.com> <1205623790.29245.19.camel@hias.m3w.org> <1205654693.29245.34.camel@hias.m3w.org> <20080316134425.bog0v1yz48o8ckss@mail.elegosoft.com> Message-ID: <20080317031040.GA30645@topoi.pooq.com> On Sun, Mar 16, 2008 at 01:44:25PM +0100, Olaf Wagner wrote: > > I don't think it will be furthered by switching from CVS to SVN > or any other tool change. ... ... > > If there really is the wish within the community to switch from > CVS to SVN, then those interested should build a team, investigate > the impacts and propose a migration plan. Elego and me for one > would need new server setup, the actual repository migration (which > should preserve as much information as possible), setup of repository > replication to two machines, integration of SVN into the regression > test framework, setup of SVN clients, update / rewrite of all the > repository access information and CM guidelines, and a migration > strategy for all the existing HTTP links in search engines etc. > > If I should make an estimation for this project in a commercial > environment, I'd say it will take 3-6 months and cost a client > between 40,000 and 80,000 Euros (that's currently 60,000 to 120,000$). > I'd rather invest these efforts for other goals. If other CM3 users > think it is worthwhile to do, they should just do it; after all, this > is an open source project. If such a group does reorganise the Modula 3 sources into a new revision control system, though, I'd strongly advise them to look into a distributed one. -- hendrik From hosking at cs.purdue.edu Mon Mar 17 08:43:09 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 17 Mar 2008 03:43:09 -0400 Subject: [M3devel] Whitespace checkins Message-ID: <73FCF91E-BF90-425C-BBF0-D0746EE44C5F@cs.purdue.edu> Jay, I notice you just checked in a slew of whitespace removal. I hate extra whitespace as much as anyone, but these sorts of checkins play havoc with those of us who track changes via diff (and derivatives). Please restrain your whitespace editing to files where you make meaningful changes. Thanks, Tony Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Mon Mar 17 09:32:38 2008 From: jayk123 at hotmail.com (Jay) Date: Mon, 17 Mar 2008 08:32:38 +0000 Subject: [M3devel] host vs. target naming? In-Reply-To: <788881.63404.qm@web58511.mail.re3.yahoo.com> References: <788881.63404.qm@web58511.mail.re3.yahoo.com> Message-ID: host vs. target naming I think this stuff is mildly messed up.Currently slashes and foo.lib vs. libfoo.a "go together". I contend that this is suspicous. The "slash" character is pretty much always related to the host.If a cross build stops at some point and gets copied from one machineto another and resumed, and the slash character changes en route,that is interesting. However, in this case, I think the pathswritten to .m3ship etc. should probably be host and target-independent.Either use arrays of path elements, or declare that one or the otheror both slash characters are the "abstract" representation, and convertas needed at runtime. Or, just always use a forward slash. They always work.But avoid presenting to them to the Win32 user. The "naming" conventions: kernel32.lib vs. libkernel32.a foo.exe vs. foo foo.dll vs. foo.so The last two are strictly a function of target.A Linux-hosted NT386-targeted config should write foo.exe and foo.dll. Never (by default) foo and foo.so. A NT386-hosted Linux-targeted config should write foo and foo.so. Never (by default) foo.exe and foo.dll. The first is a function of toolset, and tendencies go both ways I think.I think mainly they go with the host. That is..almost none of this stuffvaries in any build, but in one example I have, NT386 targeting NT386GNU,the cm3 "host" is sort of "cross", but the backend is really "native",and the files are libkernel32.a, etc. I suggest that in M3Path.m3, Prefix and Suffix always be indexed by "target" (i.e. FALSE).Directory and volume separator code unchanged. Something like the following, which also switches NT386GNU to Unix naming conventions. Probably Target.m3 should hardcode the target naming conventions, andthe value be exposed perhaps to Quake (for the likes of my NT386.common). That is not done here.Using something other than numbers might be good too -- "Unix", "GrumpyUnix", "Win32". Index: src/Builder.m3===================================================================RCS file: /usr/cvs/cm3/m3-sys/cm3/src/Builder.m3,vretrieving revision 1.18diff -c -r1.18 Builder.m3*** src/Builder.m3 24 Feb 2008 16:36:44 -0000 1.18--- src/Builder.m3 17 Mar 2008 08:11:01 -0000****************** 12,17 ****--- 12,18 ---- IMPORT Msg, Arg, Utils, M3Path, M3Backend, M3Compiler; IMPORT Quake, QMachine, QValue, QVal, QVSeq; IMPORT M3Loc, M3Unit, M3Options, MxConfig AS M3Config;+ IMPORT QIdent; FROM Target IMPORT M3BackendMode_t, BackendAssembly; TYPE****************** 101,108 **** main : M3ID.T; (* "Main" *) m3env : Env; (* the compiler's environment closure *) target : TEXT; (* target machine *)! host_os : M3Path.OSKind; (* host system *)! target_os : M3Path.OSKind; (* target os *) m3backend_mode: M3BackendMode_t; (* tells how to turn M3CG -> object *) m3backend : ConfigProc; (* translate M3CG -> ASM or OBJ *) c_compiler : ConfigProc; (* compile C code *)--- 102,108 ---- main : M3ID.T; (* "Main" *) m3env : Env; (* the compiler's environment closure *) target : TEXT; (* target machine *)! target_os := M3Path.OSKind.Unix; (* target os *) m3backend_mode: M3BackendMode_t; (* tells how to turn M3CG -> object *) m3backend : ConfigProc; (* translate M3CG -> ASM or OBJ *) c_compiler : ConfigProc; (* compile C code *)****************** 139,153 **** END; PROCEDURE SetupNamingConventionsInternal (VAR s : State; mach : Quake.Machine) = BEGIN s.machine := mach;! s.host_os := GetOSType (s, "NAMING_CONVENTIONS");! IF (GetDefn (s, "TARGET_NAMING") = NIL)! THEN s.target_os := s.host_os;! ELSE s.target_os := GetOSType (s, "TARGET_NAMING"); END;! M3Path.SetOS (s.host_os, host := TRUE);! M3Path.SetOS (s.target_os, host := FALSE); END SetupNamingConventionsInternal; PROCEDURE SetupNamingConventions (mach : Quake.Machine) =--- 139,166 ---- END; PROCEDURE SetupNamingConventionsInternal (VAR s : State; mach : Quake.Machine) =+ VAR+ value : QValue.Binding; BEGIN s.machine := mach;! ! value := GetDefn (s, "NAMING_CONVENTIONS");! IF value # NIL THEN! WITH host_os = GetOSType (s, value) DO! s.target_os := host_os;! M3Path.SetOS (host_os, host := TRUE);! M3Path.SetOS (host_os, host := FALSE);! END;! END;! ! value := GetDefn (s, "TARGET_NAMING");! IF value # NIL THEN! WITH target_os = GetOSType (s, value) DO! s.target_os := target_os;! M3Path.SetOS (target_os, host := FALSE);! END; END;! END SetupNamingConventionsInternal; PROCEDURE SetupNamingConventions (mach : Quake.Machine) =****************** 230,261 **** RETURN s; END CompileUnits; ! PROCEDURE GetOSType (s: State; sym: TEXT): M3Path.OSKind =! VAR val := GetConfigItem (s, sym); BEGIN! IF Text.Equal (val, "0") THEN RETURN M3Path.OSKind.Unix;! ELSIF Text.Equal (val, "1") THEN RETURN M3Path.OSKind.GrumpyUnix;! ELSIF Text.Equal (val, "2") THEN RETURN M3Path.OSKind.Win32; END;! ConfigErr (s, sym, "unrecognized naming convention: " & val); RETURN M3Path.OSKind.Unix; END GetOSType; ! PROCEDURE GetConfigItem (s: State; symbol: TEXT; default: TEXT := NIL): TEXT =! VAR bind := GetDefn (s, symbol); BEGIN IF bind = NIL THEN IF default # NIL THEN RETURN default; END; END;- IF (bind = NIL) THEN ConfigErr (s, symbol, "not defined"); END; TRY RETURN QVal.ToText (s.machine, bind.value); EXCEPT Quake.Error (msg) =>! ConfigErr (s, symbol, msg); END; RETURN NIL; END GetConfigItem; PROCEDURE GetConfigProc (s: State; symbol: TEXT;--- 243,282 ---- RETURN s; END CompileUnits; ! PROCEDURE GetOSType (s: State; bind: QValue.Binding): M3Path.OSKind =! VAR val := BindingToText (s, bind); BEGIN! IF Text.Length (val) = 1 THEN! CASE (Text.GetChar (val, 0)) OF! | '0' => RETURN M3Path.OSKind.Unix;! | '1' => RETURN M3Path.OSKind.GrumpyUnix;! | '2' => RETURN M3Path.OSKind.Win32;! ELSE! END; END;! ConfigErr (s, s.machine.map.id2txt(bind.name), "unrecognized naming convention: " & val); RETURN M3Path.OSKind.Unix; END GetOSType; ! PROCEDURE BindingToText (s: State; bind: QValue.Binding; default: TEXT := NIL): TEXT = BEGIN IF bind = NIL THEN IF default # NIL THEN RETURN default; END;+ ConfigErr (s, s.machine.map.id2txt(bind.name), "not defined"); END; TRY RETURN QVal.ToText (s.machine, bind.value); EXCEPT Quake.Error (msg) =>! ConfigErr (s, s.machine.map.id2txt(bind.name), msg); END; RETURN NIL;+ END BindingToText;+ + PROCEDURE GetConfigItem (s: State; symbol: TEXT; default: TEXT := NIL): TEXT =+ BEGIN+ RETURN BindingToText (s, GetDefn (s, symbol), default); END GetConfigItem; PROCEDURE GetConfigProc (s: State; symbol: TEXT;Index: src/M3Path.i3===================================================================RCS file: /usr/cvs/cm3/m3-sys/cm3/src/M3Path.i3,vretrieving revision 1.2diff -c -r1.2 M3Path.i3*** src/M3Path.i3 18 Feb 2008 03:53:19 -0000 1.2--- src/M3Path.i3 17 Mar 2008 08:11:02 -0000****************** 14,20 **** END; VAR (* possibly changed at initialization and then CONST *)- SlashChar := '/'; SlashText := "/"; TYPE--- 14,19 ----Index: src/M3Path.m3===================================================================RCS file: /usr/cvs/cm3/m3-sys/cm3/src/M3Path.m3,vretrieving revision 1.28diff -c -r1.28 M3Path.m3*** src/M3Path.m3 25 Feb 2008 18:31:18 -0000 1.28--- src/M3Path.m3 17 Mar 2008 08:11:02 -0000****************** 9,15 **** MODULE M3Path; ! IMPORT Pathname, Text, Env; IMPORT RTIO, Process; CONST--- 9,15 ---- MODULE M3Path; ! IMPORT Pathname, Text; IMPORT RTIO, Process; CONST****************** 114,124 **** PROCEDURE Join (dir, base: TEXT; k: Kind; host: BOOLEAN): TEXT = VAR! os := os_map [host];! pre := Prefix [os][k];! ext := Suffix [os][k];! d_sep := DirSep [os];! v_sep := VolSep [os]; ch : CHAR; buf : ARRAY [0..255] OF CHAR; dir_len := 0;--- 114,125 ---- PROCEDURE Join (dir, base: TEXT; k: Kind; host: BOOLEAN): TEXT = VAR! host_os := os_map [host];! target_os := os_map [FALSE];! pre := Prefix [target_os][k];! ext := Suffix [target_os][k];! d_sep := DirSep [host_os];! v_sep := VolSep [host_os]; ch : CHAR; buf : ARRAY [0..255] OF CHAR; dir_len := 0;****************** 191,199 **** d_index := -1; v_index := -1; start := 0;! os := os_map [host];! d_sep := DirSep [os];! v_sep := VolSep [os]; ext : TEXT; ext_len : INTEGER; pre : TEXT;--- 192,201 ---- d_index := -1; v_index := -1; start := 0;! host_os := os_map [host];! target_os := os_map [FALSE];! d_sep := DirSep [host_os];! v_sep := VolSep [host_os]; ext : TEXT; ext_len : INTEGER; pre : TEXT;****************** 222,228 **** t.dir := Text.FromChars (SUBARRAY (nm, 0, v_index+1)); start := v_index + 1; ELSIF (d_index = 0) THEN! t.dir := DirSepText [os]; start := 1; ELSE t.dir := Text.FromChars (SUBARRAY (nm, 0, d_index));--- 224,230 ---- t.dir := Text.FromChars (SUBARRAY (nm, 0, v_index+1)); start := v_index + 1; ELSIF (d_index = 0) THEN! t.dir := DirSepText [host_os]; start := 1; ELSE t.dir := Text.FromChars (SUBARRAY (nm, 0, d_index));****************** 234,241 **** t.kind := Kind.Unknown; ext_len := 0; FOR k := FIRST (Kind) TO LAST (Kind) DO! ext := Suffix [os][k];! IF ExtMatch (nm_txt, ext, os) THEN ext_len := Text.Length (ext); t.kind := k; EXIT;--- 236,243 ---- t.kind := Kind.Unknown; ext_len := 0; FOR k := FIRST (Kind) TO LAST (Kind) DO! ext := Suffix [target_os][k];! IF ExtMatch (nm_txt, ext, host_os) THEN ext_len := Text.Length (ext); t.kind := k; EXIT;****************** 245,252 **** (* extract the base component *) t.base := Text.FromChars (SUBARRAY (nm, start, base_len - ext_len)); ! pre := Prefix[os][t.kind];! IF (Text.Length (pre) > 0) AND PrefixMatch (t.base, pre, os) THEN t.base := Text.Sub (t.base, Text.Length (pre)); END; --- 247,254 ---- (* extract the base component *) t.base := Text.FromChars (SUBARRAY (nm, start, base_len - ext_len)); ! pre := Prefix[target_os][t.kind];! IF (Text.Length (pre) > 0) AND PrefixMatch (t.base, pre, host_os) THEN t.base := Text.Sub (t.base, Text.Length (pre)); END; ****************** 321,337 ****--- 323,343 ---- PROCEDURE DefaultProgram (host: BOOLEAN): TEXT = BEGIN+ host := FALSE; RETURN Default_pgm [os_map [host]]; END DefaultProgram; PROCEDURE ProgramName (base: TEXT; host: BOOLEAN): TEXT = BEGIN+ host := FALSE; RETURN base & Suffix[os_map [host]][Kind.PGM]; END ProgramName; PROCEDURE LibraryName (base: TEXT; host: BOOLEAN): TEXT = VAR os := os_map [host]; BEGIN+ host := FALSE;+ os := os_map [host]; RETURN Prefix[os][Kind.LIB] & base & Suffix[os][Kind.LIB]; END LibraryName; ****************** 764,786 **** lcase[i] := VAL (ORD (i) - ORD ('A') + ORD ('a'), CHAR); END; ! (* OSKind is determined from "naming conventions"! which are mostly about foo.lib vs. libfoo.a, foo.dll vs. libfoo.so, foo vs. foo.exe.! "Naming conventions" also determines the slash, however NT386GNU uses! Win32 "naming conventions" (currently) but forward slashes. In reality,! forward slashes work almost anywhere, but at this time we are not pushing! through the change to use forward slashes more on regular NT386.! What we do here is probe our runtime for its slash as was already being! done to set the Quake SL variable, but applying the result to more code.! ! In general this run-time/compile-time determinations probably need better abstraction.! At least in other places, where the way time works is used to determine if the runtime is Unix.! In this case, the slash could be fed in at build time, or set in Quake, however! how to get his data from Quake efficiently (and early enough?), is to be determined. *) IF Text.GetChar (Pathname.Join ("a", "b"), 1) = BackSlash THEN - SlashChar := BackSlash; SlashText := "\\"; os_map [TRUE] := OSKind.Win32; os_map [FALSE] := OSKind.Win32;--- 770,780 ---- lcase[i] := VAL (ORD (i) - ORD ('A') + ORD ('a'), CHAR); END; ! (* Probe the host for what slash it uses, and default host and! target "naming conventions" based on that. *) IF Text.GetChar (Pathname.Join ("a", "b"), 1) = BackSlash THEN SlashText := "\\"; os_map [TRUE] := OSKind.Win32; os_map [FALSE] := OSKind.Win32;****************** 789,809 **** lcase [Slash] := BackSlash; - ELSE- - WITH OS = Env.Get("OS") DO- IF OS # NIL AND Text.Equal (OS, "Windows_NT") THEN- - (* NT386GNU uses foo.lib instead of libfoo.a (at least for now), and forward slashes. *)- - os_map [TRUE] := OSKind.Win32;- os_map [FALSE] := OSKind.Win32;- DirSep [OSKind.Win32] := DirSep [OSKind.Unix]; (* Slash *)- DirSepText [OSKind.Win32] := DirSepText [OSKind.Unix]; (* "//" *)- VolSep [OSKind.Win32] := VolSep [OSKind.Unix]; (* Null *)- - END;- END; END; (* Test (); *)--- 783,788 ---- - Jay _________________________________________________________________ Need to know the score, the latest news, or you need your Hotmail?-get your "fix". http://www.msnmobilefix.com/Default.aspx -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Mon Mar 17 09:35:12 2008 From: jayk123 at hotmail.com (Jay) Date: Mon, 17 Mar 2008 08:35:12 +0000 Subject: [M3devel] Whitespace checkins In-Reply-To: <73FCF91E-BF90-425C-BBF0-D0746EE44C5F@cs.purdue.edu> References: <73FCF91E-BF90-425C-BBF0-D0746EE44C5F@cs.purdue.edu> Message-ID: Tony, Can you use diff -w or diff -b? Can the derivatives be cleaned up?I'll volunteer. It'd be nice to get rid of whitespace at ends of lines all tabs overly indented "END", like I sometimes see on records. The Modula-3 codebase isn't so large as to make this infeasible,even if that extends to the PM3 copy. My editor uses 4 space tabs and a fair amount of code with tabsassumes 8. It'd be nice just to get rid of them. ok either way. A lot stuff bugs me and I /often/ but not always ignore it.. - Jay From: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Whitespace checkinsDate: Mon, 17 Mar 2008 03:43:09 -0400CC: m3devel at elegosoft.com Jay, I notice you just checked in a slew of whitespace removal. I hate extra whitespace as much as anyone, but these sorts of checkins play havoc with those of us who track changes via diff (and derivatives). Please restrain your whitespace editing to files where you make meaningful changes. Thanks, Tony Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 _________________________________________________________________ Shed those extra pounds with MSN and The Biggest Loser! http://biggestloser.msn.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Mar 17 11:11:31 2008 From: jay.krell at cornell.edu (Jay) Date: Mon, 17 Mar 2008 10:11:31 +0000 Subject: [M3devel] dtoa.h? Message-ID: any advise to for debuging..I think I'll just turn off optimization..that fixed it...no patience to debug codegen bugs or anything in dtoa..confusing stuff... This wasn't occuring..but now I can repro it consistently compiling some of m3core/extended*, m3core/real*, m3core/longreal*. The rest of m3core compiles ok.. Program received signal SIGSEGV, Segmentation fault.mult (a=0x1426860, b=0x1426880) at ..//src//Csupport//Common/dtoa.h:535535 *x = 0;(gdb) bt#0 mult (a=0x1426860, b=0x1426880) at ..//src//Csupport//Common/dtoa.h:535#1 0x00626721 in pow5mult (b=0x1426880, k=1185) at ..//src//Csupport//Common/dtoa.h:620#2 0x00628011 in m3_dtoa (d=4.9406564584124654e-324, mode=2, ndigits=14, decpt=0x22bc74, sign=0x22bc60, rve=0x22bc68) at ..//src//Csupport//Common/dtoa.h:2224#3 0x0061534d in Convert__InternalFromLongFloat (M3_CKMnXU_buf=0x22be58, M3_CtKayy_v=4.9406564584124654e-324, M3_AcxOUs_p=13, M3_BTpO8m_style=2 '\002', M3_Bkn9rd_exponentChar=88 'X') at ..\src\convert\Convert.m3:367#4 0x00614c1f in Convert__FromExtended (M3_CKMnXU_buf=0x22be58, M3_C4YHmT_v=4.9406564584124654e-324, M3_AcxOUs_p=13, M3_BTpO8m_style=2 '\002') at ..\src\convert\Convert.m3:282#5 0x005bb6ca in M3Buf__PutFloat (M3_DmgyYA_t=0x7fe9d290, M3_DYj8sa_f=0x7fed4b5c) at ..\src\M3Buf.m3:156#6 0x005760c9 in ReelExpr__GenFPLiteral (M3_DopDyP_p=0x7fed4b4c, M3_DmgyYA_buf=0x7fe9d290) at ..\src\exprs\ReelExpr.m3:278#7 0x0054bf68 in Expr__GenFPLiteral (M3_ES44mX_t=0x7fed4b4c, M3_DmgyYA_mbuf=0x7fe9d290) at ..\src\exprs\Expr.m3:224#8 0x005301b2 in Constant__AddFPTag (M3_AjdbKy_t=0x7fed4abc, M3_BlO1Kl_x=0x22bf68) at ..\src\values\Constant.m3:225#9 0x004cb94c in Value__AddFPTag (M3_EjfEr4_t=0x7fed4abc, M3_BlO1Kl_x=0x22bf68) at ..\src\values\Value.m3:143#10 0x004cb6e1 in Value__ToFP (M3_D9M1dT__result=0x22bff4, M3_EjfEr4_t=0x7fed4abc) at ..\src\values\Value.m3:120#11 0x004cbec3 in Value__Declare (M3_EjfEr4_t=0x7fed4abc) at ..\src\values\Value.m3:202#12 0x004bb7dd in Scope__Enter (M3_DurAC9_t=0x7fed48e0) at ..\src\misc\Scope.m3:239#13 0x004b2d70 in Module__CompileInterface (M3_DZ1mTg_t=0x7fecf77c) at ..\src\values\Module.m3:870#14 0x004b2985 in Module__Compile (M3_DZ1mTg_t=0x7fecf77c) at ..\src\values\Module.m3:836#15 0x0049fd4d in M3Front__DoCompile () at ..\src\misc\M3Front.m3:196#16 0x0049f37a in M3Front__Compile (M3_AT7GN4_input=0x22c1ec, M3_AUoUsm_env=0x7fec084c, M3_BySCJy_options=0x7ff6415c) at ..\src\misc\M3Front.m3:64#17 0x0040aad8 in Builder__RunM3 (M3_Bd9FNt_s=0x7ff3de1c, M3_AXLf8B_u=0x7fecc1dc, M3_Bd56fi_object=0x7ff640e8) at ..\src\Builder.m3:1454#18 0x0040927b in Builder__PushOneM3 (M3_Bd9FNt_s=0x7ff3de1c, M3_AXLf8B_u=0x7fecc1dc) at ..\src\Builder.m3:1125#19 0x00408e94 in Builder__CompileM3 (M3_Bd9FNt_s=0x7ff3de1c, M3_AXLf8B_u=0x7fecc1dc) at ..\src\Builder.m3:1076#20 0x00407b15 in Builder__CompileOne (M3_Bd9FNt_s=0x7ff3de1c,---Type to continue, or q to quit---qM3_AXLf8B_u=0x7fQuit) at ..\src\Builder.m3:911#21 0x00409f46 in Builder__CheckImp (M3_Bd9FNt_s=0x7ff3de1c, M3_A7mgsK_u=0x7ff79d54, M3_EIVwTa_z=0x7ff79d68, M3_A2QN6Z_kind=1 '\001') at ..\src\Builder.m3:1292#22 0x00409e45 in Builder__CheckImports (M3_Bd9FNt_s=0x7ff3de1c, M3_A7mgsK_u=0x7ff79d54) at ..\src\Builder.m3:1279#23 0x00409acf in Builder__M3isStale (M3_Bd9FNt_s=0x7ff3de1c, M3_AXLf8B_u=0x7fe5c270) at ..\src\Builder.m3:1229#24 0x00408e4b in Builder__CompileM3 (M3_Bd9FNt_s=0x7ff3de1c, M3_AXLf8B_u=0x7fe5c270) at ..\src\Builder.m3:1071#25 0x00407b15 in Builder__CompileOne (M3_Bd9FNt_s=0x7ff3de1c, M3_AXLf8B_u=0x7fe5c270) at ..\src\Builder.m3:911#26 0x004077e7 in Builder__CompileEverything (M3_Bd9FNt_s=0x7ff3de1c, M3_Cw4bpV_schedule=0x7ff247f0) at ..\src\Builder.m3:871#27 0x0040391d in Builder__CompileUnits (M3_Bd56fi_main=0x7ff67014, M3_EEuw3X_units=0x7ff7e73c, M3_C1FTrk_sys_libs=0x7ff3dd60, M3_A2QN6Z_info_kind=17 '\021', M3_An02H2_mach=0x7ff7e63c) at ..\src\Builder.m3:239#28 0x0040250a in Builder__BuildLib (M3_Bd56fi_lib=0x7ff67014, M3_EEuw3X_units=0x7ff7e73c, M3_C1FTrk_sys_libs=0x7ff3dd60, M3_AicXUJ_shared=1 '\001', M3_An02H2_m=0x7ff7e63c) at ..\src\Builder.m3:36#29 0x0041afd0 in M3Build__DoLibrary (M3_An02H2_m=0x7ff7e63c, M3_AcxOUs_n_args=1) at ..\src\M3Build.m3:1365#30 0x00477f27 in QMachine__DoCall (M3_An02H2_t=0x7ff7e63c, M3_AcxOUs_n_args=1, M3_AicXUJ_isFunc=0 '\0', M3_AicXUJ_outer=0 '\0') at ..\src\QMachine.m3:529#31 0x00476f1d in QMachine__Eval (M3_An02H2_t=0x7ff7e63c) at ..\src\QMachine.m3:405#32 0x0047f1f3 in QMachine (M3_An02H2_t=0x7ff7e63c, M3_Bd56fi_path=0x7ff939b0, M3_AicXUJ_from_code=0 '\0') at ..\src\QMachine.m3:1660#33 0x0047f038 in QMachine (M3_An02H2_t=0x7ff7e63c, M3_Bd56fi_path=0x7ff939b0) at ..\src\QMachine.m3:1631#34 0x00417959 in M3Build (M3_ABp1Zk_t=0x7ff7e63c, M3_Bd56fi_file=0x7ff939b0) at ..\src\M3Build.m3:579#35 0x0041935f in M3Build__DoIncludeDir (M3_An02H2_m=0x7ff7e63c, M3_AcxOUs_n_args=1) at ..\src\M3Build.m3:954#36 0x00477f27 in QMachine__DoCall (M3_An02H2_t=0x7ff7e63c, M3_AcxOUs_n_args=1, M3_AicXUJ_isFunc=0 '\0', M3_AicXUJ_outer=0 '\0') at ..\src\QMachine.m3:529#37 0x00476f1d in QMachine__Eval (M3_An02H2_t=0x7ff7e63c) at ..\src\QMachine.m3:405#38 0x004756ef in QMachine__Evaluate (M3_An02H2_t=0x7ff7e63c, M3_CYwAos_s=0x7ff93568) at ..\src\QMachine.m3:154#39 0x00486ad2 in Quake__Run (M3_An02H2_m=0x7ff7e63c,---Type to continue, or q to quit---qM3_Bd56fi_source_file=0x7Quit) at ..\src\Quake.m3:19#40 0x00415ee7 in M3Build__Run (M3_ABp1Zk_t=0x7ff7e63c, M3_Bd56fi_makefile=0x7ff9243c) at ..\src\M3Build.m3:183#41 0x0042c706 in Main__DoIt () at ..\src\Main.m3:81#42 0x0042c9dc in Main_M3 (M3_AcxOUs_mode=1) at ..\src\Main.m3:195#43 0x005fa1ee in RTLinker__RunMainBody (M3_DjPxE3_m=0x63c020) at ..\src\runtime\common\RTLinker.m3:399#44 0x005f9719 in RTLinker__AddUnitI (M3_DjPxE3_m=0x63c020) at ..\src\runtime\common\RTLinker.m3:113#45 0x005f97a0 in RTLinker__AddUnit (M3_DjPxE5_b=0x42c9c7) at ..\src\runtime\common\RTLinker.m3:122#46 0x00401085 in main (argc=1, argv=0x14218e8, envp=0x1420090) at _m3main.mc:4 _________________________________________________________________ Helping your favorite cause is as easy as instant messaging.?You IM, we give. http://im.live.com/Messenger/IM/Home/?source=text_hotmail_join -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Mon Mar 17 11:14:50 2008 From: jayk123 at hotmail.com (Jay) Date: Mon, 17 Mar 2008 10:14:50 +0000 Subject: [M3devel] possible cygwin createprocess/fork cost measurements.. Message-ID: I ran some mostly scientific measures of Cygwin. On one machine, no reboots, one OS, one set of files. x86, single proc, Windows 2000 (I'll go back to XP soon). It shows that..well, at least that wrapping Cygwin processes with sh is VERY expensive. Like, the data isn't yet complete, but this could cut building Cygwin libm3 from around 100 seconds to around 20 seconds. Not counting the Modula-3 front end time. Just cm3cg+as. cd libm3\NT386GNU having already built successfully, all the *.ic *.mc files are present cm3cg not wrapped with sh (F1) Repeated runs. 28 seconds (other stuff running on machine) 16 seconds 13 seconds (13.?) 13.8 seconds 14.01 seconds 13.3 seconds now add the -o flag 13.64 seconds 14.07 seconds now without echoing 13.22 seconds 13.18 seconds cm3cg wrapped with sh (F2) 51 seconds 51.35 seconds 51.19 seconds 50.88 seconds now add the -o flag 51.76 seconds now without echoing 51.05 seconds These runs did NOT have -o flags, but subsequent runs with -o were about the same. I added -o so I could run the as variations. now the same with .s note that due to the way the above worked, I just have *.s files, and not the usual *.is and *.ms as not wrapped with sh (F3) 5.6 seconds 5.28 seconds now remove echo 5.08 seconds 5.08 seconds 5.04 seconds forgot -o flag, oh well, enough data as wrapped with sh (F4) 43 seconds 43.56 seconds forgot -o flag, oh well, enough data What is not yet confirmed is: 1) Does cm3 wrap everything with sh? 2) Does calling m3cg/as from cm3 have these costs? Very clear: Wrapping stuff with sh on Cygwin is expensive! Actions: Confirm this cost is being paid by cm3. Either: 1) implement some "batch modes" in cm3 and/or cm3cg 2) or maybe, um, just make sure that cm3 does not wrap with sh, and if cm3 itself causes this slowdown, because of how Cygwin works, try interposing a small Win32 helper app. I think Cygwin handles runnig Win32 apps and being run from Win32 apps differently than Cygwin running Cygwin -- i.e. not slowly. I'll see. Could be that creating twice the number of processes, in order to avoid Cygwin running Cygwin, could be faster. Not yet known. Maybe use system() instead vfork() + exec()? Odd, though, vfork instead of fork is supposed to help. Here is the test code, you edit it to run one case or another: @if not "%1" == "" goto :%1 at rem \cygwin\bin\time cmd /c %~f0 F1 at rem \cygwin\bin\time cmd /c %~f0 F2 at rem \cygwin\bin\time cmd /c %~f0 F3@\cygwin\bin\time cmd /c %~f0 F4 at goto :eof :F1 at echo off at del *s *ofor %%a in (*.ic *.mc) do cm3cg -quiet %%a -o %%a.s"goto :eof :F2 at echo off at del *s *ofor %%a in (*.ic *.mc) do sh -c "cm3cg -quiet %%a -o %%a.s"goto :eof :F3 at del *o at echo offfor %%a in (*.s) do as %%agoto :eof :F4 at del *ofor %%a in (*.s) do sh -c "as %%a"goto :eof - Jay _________________________________________________________________ Helping your favorite cause is as easy as instant messaging.?You IM, we give. http://im.live.com/Messenger/IM/Home/?source=text_hotmail_join -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Mon Mar 17 15:15:20 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 17 Mar 2008 10:15:20 -0400 Subject: [M3devel] Whitespace checkins In-Reply-To: References: <73FCF91E-BF90-425C-BBF0-D0746EE44C5F@cs.purdue.edu> Message-ID: In principle I agree with everything you say for code *as it is edited* for meaningful improvements, but removing whitespace just for the fun (?) of it makes tracking meaningful changes a little more difficult. On Mar 17, 2008, at 4:35 AM, Jay wrote: > Tony, Can you use diff -w or diff -b? Sure, I can, but I have to type an extra "-w". ;-) > Can the derivatives be cleaned up? Derivatives? > I'll volunteer. > > It'd be nice to get rid of > whitespace at ends of lines > all tabs > overly indented "END", like I sometimes see on records. Other than white-space It's all a matter of style I guess. Most of this code was written in emacs which defaults tabs equivalent to 8 spaces. Yes, I know other editors do 4-space tabs. Modula-3 mode in emacs indents using spaces instead of tabs. > The Modula-3 codebase isn't so large as to make this infeasible, > even if that extends to the PM3 copy. True. > My editor uses 4 space tabs and a fair amount of code with tabs > assumes 8. It'd be nice just to get rid of them. Agreed. > ok either way. A lot stuff bugs me and I /often/ but not always > ignore it.. > > - Jay > > From: hosking at cs.purdue.edu > To: jayk123 at hotmail.com > Subject: Whitespace checkins > Date: Mon, 17 Mar 2008 03:43:09 -0400 > CC: m3devel at elegosoft.com > > Jay, > > I notice you just checked in a slew of whitespace removal. I hate > extra whitespace as much as anyone, but these sorts of checkins play > havoc with those of us who track changes via diff (and > derivatives). Please restrain your whitespace editing to files > where you make meaningful changes. > > Thanks, > > Tony > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > Shed those extra pounds with MSN and The Biggest Loser! Learn more. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Mon Mar 17 15:19:04 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 17 Mar 2008 10:19:04 -0400 Subject: [M3devel] dtoa.h? In-Reply-To: References: Message-ID: How was dtoa compiled? That code is a fragile part of the run-time system that uses C that breaks with some C compilers. (See why I hate non-portable C!). Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 17, 2008, at 6:11 AM, Jay wrote: > any advise to for debuging..I think I'll just turn off > optimization..that fixed it...no patience to debug codegen bugs or > anything in dtoa..confusing stuff... > This wasn't occuring..but now I can repro it consistently compiling > some of m3core/extended*, m3core/real*, m3core/longreal*. The rest > of m3core compiles ok.. > > Program received signal SIGSEGV, Segmentation fault. > mult (a=0x1426860, b=0x1426880) at ..//src//Csupport//Common/dtoa.h: > 535 > 535 *x = 0; > (gdb) bt > #0 mult (a=0x1426860, b=0x1426880) at ..//src//Csupport//Common/ > dtoa.h:535 > #1 0x00626721 in pow5mult (b=0x1426880, k=1185) > at ..//src//Csupport//Common/dtoa.h:620 > #2 0x00628011 in m3_dtoa (d=4.9406564584124654e-324, mode=2, > ndigits=14, > decpt=0x22bc74, sign=0x22bc60, rve=0x22bc68) > at ..//src//Csupport//Common/dtoa.h:2224 > #3 0x0061534d in Convert__InternalFromLongFloat > (M3_CKMnXU_buf=0x22be58, > M3_CtKayy_v=4.9406564584124654e-324, M3_AcxOUs_p=13, > M3_BTpO8m_style=2 '\002', M3_Bkn9rd_exponentChar=88 'X') > at ..\src\convert\Convert.m3:367 > #4 0x00614c1f in Convert__FromExtended (M3_CKMnXU_buf=0x22be58, > M3_C4YHmT_v=4.9406564584124654e-324, M3_AcxOUs_p=13, > M3_BTpO8m_style=2 '\002') at ..\src\convert\Convert.m3:282 > #5 0x005bb6ca in M3Buf__PutFloat (M3_DmgyYA_t=0x7fe9d290, > M3_DYj8sa_f=0x7fed4b5c) at ..\src\M3Buf.m3:156 > #6 0x005760c9 in ReelExpr__GenFPLiteral (M3_DopDyP_p=0x7fed4b4c, > M3_DmgyYA_buf=0x7fe9d290) at ..\src\exprs\ReelExpr.m3:278 > #7 0x0054bf68 in Expr__GenFPLiteral (M3_ES44mX_t=0x7fed4b4c, > M3_DmgyYA_mbuf=0x7fe9d290) at ..\src\exprs\Expr.m3:224 > #8 0x005301b2 in Constant__AddFPTag (M3_AjdbKy_t=0x7fed4abc, > M3_BlO1Kl_x=0x22bf68) at ..\src\values\Constant.m3:225 > #9 0x004cb94c in Value__AddFPTag (M3_EjfEr4_t=0x7fed4abc, > M3_BlO1Kl_x=0x22bf68) at ..\src\values\Value.m3:143 > #10 0x004cb6e1 in Value__ToFP (M3_D9M1dT__result=0x22bff4, > M3_EjfEr4_t=0x7fed4abc) at ..\src\values\Value.m3:120 > #11 0x004cbec3 in Value__Declare (M3_EjfEr4_t=0x7fed4abc) > at ..\src\values\Value.m3:202 > #12 0x004bb7dd in Scope__Enter (M3_DurAC9_t=0x7fed48e0) > at ..\src\misc\Scope.m3:239 > #13 0x004b2d70 in Module__CompileInterface (M3_DZ1mTg_t=0x7fecf77c) > at ..\src\values\Module.m3:870 > #14 0x004b2985 in Module__Compile (M3_DZ1mTg_t=0x7fecf77c) > at ..\src\values\Module.m3:836 > #15 0x0049fd4d in M3Front__DoCompile () at ..\src\misc\M3Front.m3:196 > #16 0x0049f37a in M3Front__Compile (M3_AT7GN4_input=0x22c1ec, > M3_AUoUsm_env=0x7fec084c, M3_BySCJy_options=0x7ff6415c) > at ..\src\misc\M3Front.m3:64 > #17 0x0040aad8 in Builder__RunM3 (M3_Bd9FNt_s=0x7ff3de1c, > M3_AXLf8B_u=0x7fecc1dc, M3_Bd56fi_object=0x7ff640e8) > at ..\src\Builder.m3:1454 > #18 0x0040927b in Builder__PushOneM3 (M3_Bd9FNt_s=0x7ff3de1c, > M3_AXLf8B_u=0x7fecc1dc) at ..\src\Builder.m3:1125 > #19 0x00408e94 in Builder__CompileM3 (M3_Bd9FNt_s=0x7ff3de1c, > M3_AXLf8B_u=0x7fecc1dc) at ..\src\Builder.m3:1076 > #20 0x00407b15 in Builder__CompileOne (M3_Bd9FNt_s=0x7ff3de1c, > ---Type to continue, or q to quit---q > M3_AXLf8B_u=0x7fQuit > ) at ..\src\Builder.m3:911 > #21 0x00409f46 in Builder__CheckImp (M3_Bd9FNt_s=0x7ff3de1c, > M3_A7mgsK_u=0x7ff79d54, M3_EIVwTa_z=0x7ff79d68, M3_A2QN6Z_kind=1 > '\001') > at ..\src\Builder.m3:1292 > #22 0x00409e45 in Builder__CheckImports (M3_Bd9FNt_s=0x7ff3de1c, > M3_A7mgsK_u=0x7ff79d54) at ..\src\Builder.m3:1279 > #23 0x00409acf in Builder__M3isStale (M3_Bd9FNt_s=0x7ff3de1c, > M3_AXLf8B_u=0x7fe5c270) at ..\src\Builder.m3:1229 > #24 0x00408e4b in Builder__CompileM3 (M3_Bd9FNt_s=0x7ff3de1c, > M3_AXLf8B_u=0x7fe5c270) at ..\src\Builder.m3:1071 > #25 0x00407b15 in Builder__CompileOne (M3_Bd9FNt_s=0x7ff3de1c, > M3_AXLf8B_u=0x7fe5c270) at ..\src\Builder.m3:911 > #26 0x004077e7 in Builder__CompileEverything (M3_Bd9FNt_s=0x7ff3de1c, > M3_Cw4bpV_schedule=0x7ff247f0) at ..\src\Builder.m3:871 > #27 0x0040391d in Builder__CompileUnits (M3_Bd56fi_main=0x7ff67014, > M3_EEuw3X_units=0x7ff7e73c, M3_C1FTrk_sys_libs=0x7ff3dd60, > M3_A2QN6Z_info_kind=17 '\021', M3_An02H2_mach=0x7ff7e63c) > at ..\src\Builder.m3:239 > #28 0x0040250a in Builder__BuildLib (M3_Bd56fi_lib=0x7ff67014, > M3_EEuw3X_units=0x7ff7e73c, M3_C1FTrk_sys_libs=0x7ff3dd60, > M3_AicXUJ_shared=1 '\001', M3_An02H2_m=0x7ff7e63c) at ..\src > \Builder.m3:36 > #29 0x0041afd0 in M3Build__DoLibrary (M3_An02H2_m=0x7ff7e63c, > M3_AcxOUs_n_args=1) at ..\src\M3Build.m3:1365 > #30 0x00477f27 in QMachine__DoCall (M3_An02H2_t=0x7ff7e63c, > M3_AcxOUs_n_args=1, M3_AicXUJ_isFunc=0 '\0', M3_AicXUJ_outer=0 > '\0') > at ..\src\QMachine.m3:529 > #31 0x00476f1d in QMachine__Eval (M3_An02H2_t=0x7ff7e63c) > at ..\src\QMachine.m3:405 > #32 0x0047f1f3 in QMachine (M3_An02H2_t=0x7ff7e63c, > M3_Bd56fi_path=0x7ff939b0, M3_AicXUJ_from_code=0 '\0') > at ..\src\QMachine.m3:1660 > #33 0x0047f038 in QMachine (M3_An02H2_t=0x7ff7e63c, > M3_Bd56fi_path=0x7ff939b0) > at ..\src\QMachine.m3:1631 > #34 0x00417959 in M3Build (M3_ABp1Zk_t=0x7ff7e63c, > M3_Bd56fi_file=0x7ff939b0) > at ..\src\M3Build.m3:579 > #35 0x0041935f in M3Build__DoIncludeDir (M3_An02H2_m=0x7ff7e63c, > M3_AcxOUs_n_args=1) at ..\src\M3Build.m3:954 > #36 0x00477f27 in QMachine__DoCall (M3_An02H2_t=0x7ff7e63c, > M3_AcxOUs_n_args=1, M3_AicXUJ_isFunc=0 '\0', M3_AicXUJ_outer=0 > '\0') > at ..\src\QMachine.m3:529 > #37 0x00476f1d in QMachine__Eval (M3_An02H2_t=0x7ff7e63c) > at ..\src\QMachine.m3:405 > #38 0x004756ef in QMachine__Evaluate (M3_An02H2_t=0x7ff7e63c, > M3_CYwAos_s=0x7ff93568) at ..\src\QMachine.m3:154 > #39 0x00486ad2 in Quake__Run (M3_An02H2_m=0x7ff7e63c, > ---Type to continue, or q to quit---q > M3_Bd56fi_source_file=0x7Quit > ) at ..\src\Quake.m3:19 > #40 0x00415ee7 in M3Build__Run (M3_ABp1Zk_t=0x7ff7e63c, > M3_Bd56fi_makefile=0x7ff9243c) at ..\src\M3Build.m3:183 > #41 0x0042c706 in Main__DoIt () at ..\src\Main.m3:81 > #42 0x0042c9dc in Main_M3 (M3_AcxOUs_mode=1) at ..\src\Main.m3:195 > #43 0x005fa1ee in RTLinker__RunMainBody (M3_DjPxE3_m=0x63c020) > at ..\src\runtime\common\RTLinker.m3:399 > #44 0x005f9719 in RTLinker__AddUnitI (M3_DjPxE3_m=0x63c020) > at ..\src\runtime\common\RTLinker.m3:113 > #45 0x005f97a0 in RTLinker__AddUnit (M3_DjPxE5_b=0x42c9c7) > at ..\src\runtime\common\RTLinker.m3:122 > #46 0x00401085 in main (argc=1, argv=0x14218e8, envp=0x1420090) > at _m3main.mc:4 > > > > > > Helping your favorite cause is as easy as instant messaging. You IM, > we give. Learn more. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Mon Mar 17 15:21:00 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 17 Mar 2008 10:21:00 -0400 Subject: [M3devel] SVN migration, also promotion In-Reply-To: <20080317031040.GA30645@topoi.pooq.com> References: <1205592376.29245.11.camel@hias.m3w.org> <20080315190300.xc13iviopws0ocoo@mail.elegosoft.com> <1205623790.29245.19.camel@hias.m3w.org> <1205654693.29245.34.camel@hias.m3w.org> <20080316134425.bog0v1yz48o8ckss@mail.elegosoft.com> <20080317031040.GA30645@topoi.pooq.com> Message-ID: <5E62AA15-6F63-4032-AFCB-78028301E44D@cs.purdue.edu> Whatever system is used, it should be widely adoptable on many platforms, and have strong support from the broadest community. To my mind, that means only cvs or svn at the present time. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 16, 2008, at 11:10 PM, hendrik at topoi.pooq.com wrote: > On Sun, Mar 16, 2008 at 01:44:25PM +0100, Olaf Wagner wrote: >> >> I don't think it will be furthered by switching from CVS to SVN >> or any other tool change. > ... > ... >> >> If there really is the wish within the community to switch from >> CVS to SVN, then those interested should build a team, investigate >> the impacts and propose a migration plan. Elego and me for one >> would need new server setup, the actual repository migration (which >> should preserve as much information as possible), setup of repository >> replication to two machines, integration of SVN into the regression >> test framework, setup of SVN clients, update / rewrite of all the >> repository access information and CM guidelines, and a migration >> strategy for all the existing HTTP links in search engines etc. >> >> If I should make an estimation for this project in a commercial >> environment, I'd say it will take 3-6 months and cost a client >> between 40,000 and 80,000 Euros (that's currently 60,000 to >> 120,000$). >> I'd rather invest these efforts for other goals. If other CM3 users >> think it is worthwhile to do, they should just do it; after all, this >> is an open source project. > > If such a group does reorganise the Modula 3 sources into a new > revision > control system, though, I'd strongly advise them to look into a > distributed one. > > -- hendrik -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Mon Mar 17 15:21:05 2008 From: jayk123 at hotmail.com (Jay) Date: Mon, 17 Mar 2008 14:21:05 +0000 Subject: [M3devel] possible cygwin createprocess/fork cost measurements.. Message-ID: I believe cm3 is affected by this.I don't have numbers yet. I propose some fairly obvious/small/simple/safe changes in order to likely achieve a large speed up in NT386GNU. I am skeptical that existing functions can be changed in a compatible enough way. So I propose, roughly: Add this to Process.i3: PROCEDURE Spawn(cmd: Pathname.T; READONLY params: ARRAY OF TEXT) : T RAISES {OSError.E};(* A restricted form of Create that is much faster on Cygwin. *) The name is very iffy. It could be in fact not be in the public interface, but merely notice if wd = stdin = stdout = stderr = nil. It could probably be in be less limited than shown. Probably all of the parameters are settable, by altering the parent's globals, within a critical section. Environment certainly is settable. It is tempting to leave it limited like this though, such as to be implementable perhaps with system. (It turns out Cygwin system is slower than spawnve; surprising since system is the most limited of the exec/spawn variants -- I think related to it having an implied sh wrapper but the others do not.)The intent is simple and obvious -- some path to spawnve or spawnvpe. p has path search. On all but Cygwin, this limited Create/Spawn will just call the normal Create. (Even on Win32). On Cygwin it will call spawnvpe (or spawnve if people really want, but "p" seems "better"). Now, in Quake, all the existing exec variants wrap the command line in either sh or cmd or command.com. Changing that is probably very dangerous, even with an attempt to discern if the wrapper buys anything, on a command line by command line basis. For example, if all of the characters * ? | < > % $ & ; ( ) are absent from the command, the shell wrapper probably doesn't buy anything and could be removed from existing paths. However that's not true -- for example system("echo foo") depends on a shell wrapper to run the builtin "echo" (at least on Windows, there is no echo.exe). I think there's no choice but to add a new Quake function, spawn, or limited_exec, or fast_exec, or process_runfast, exec_noshell, or something. Again I'm not sure what to call it, but it'd simply call Process.Spawn, or Process.Create but with right the right parameters to get into the Cygwin fast path.For now I'm going with Process.Spawn and fast_exec. I hope to have numbers "soon" as to the perf change. Another good option here, that I tried in the past but failed, and is partly not difficult to solve, but also partly, is to implemet Quake exec using Win32 CreateProcess instead of Cygwin spawn/exec. There are at least two sets of problems here. One is that the existing code returns a File.T, and for that there is the Posix and Win32 types, Cygwin uses Posix. You'd have to warp the code somehow here. I gave up on that point without much trying. It's not that much code though. Cygwin is using CreateProcess of course. The other problem is on the input, the interpretation of the command line. Again this is the value that presently Cygwin provides (albeit sometimes with great cost). Of course another angle is to work on Cygwin to make vfork efficient. It is presently implemented by calling fork. There is #ifdef'ed code for another path but it appears not enabled. I know polluting the system just for the sake of Cygwin isn't great, however: - I expect the win is quite large - "spawn*" is a pretty old thing, nothing new/controversial here, long known as an often viable replacement for fork+exec at least on Windows. It's in msvc*.dll for example. There may even be wins to be had on other Posix systems by avoiding the sh wrapper? "batching" where cm3cg is run once per directory seems like a very good idea and worth trying; the problem is, that still leaves the assembler. Perhaps the assembler could be linked in statically to cm3cg? Probably, but not particularly easily and probably unpopular upstream... Unless maybe some nice gcc perf gains would be demonstrated? - Jay From: jayk123 at hotmail.comTo: m3devel at elegosoft.comSubject: possible cygwin createprocess/fork cost measurements..Date: Mon, 17 Mar 2008 10:14:50 +0000 I ran some mostly scientific measures of Cygwin. On one machine, no reboots, one OS, one set of files. x86, single proc, Windows 2000 (I'll go back to XP soon). It shows that..well, at least that wrapping Cygwin processes with sh is VERY expensive. Like, the data isn't yet complete, but this could cut building Cygwin libm3 from around 100 seconds to around 20 seconds. Not counting the Modula-3 front end time. Just cm3cg+as. cd libm3\NT386GNU having already built successfully, all the *.ic *.mc files are present cm3cg not wrapped with sh (F1) Repeated runs. 28 seconds (other stuff running on machine) 16 seconds 13 seconds (13.?) 13.8 seconds 14.01 seconds 13.3 seconds now add the -o flag 13.64 seconds 14.07 seconds now without echoing 13.22 seconds 13.18 seconds cm3cg wrapped with sh (F2) 51 seconds 51.35 seconds 51.19 seconds 50.88 seconds now add the -o flag 51.76 seconds now without echoing 51.05 seconds These runs did NOT have -o flags, but subsequent runs with -o were about the same. I added -o so I could run the as variations. now the same with .s note that due to the way the above worked, I just have *.s files, and not the usual *.is and *.ms as not wrapped with sh (F3) 5.6 seconds 5.28 seconds now remove echo 5.08 seconds 5.08 seconds 5.04 seconds forgot -o flag, oh well, enough data as wrapped with sh (F4) 43 seconds 43.56 seconds forgot -o flag, oh well, enough data What is not yet confirmed is: 1) Does cm3 wrap everything with sh? 2) Does calling m3cg/as from cm3 have these costs? Very clear: Wrapping stuff with sh on Cygwin is expensive! Actions: Confirm this cost is being paid by cm3. Either: 1) implement some "batch modes" in cm3 and/or cm3cg 2) or maybe, um, just make sure that cm3 does not wrap with sh, and if cm3 itself causes this slowdown, because of how Cygwin works, try interposing a small Win32 helper app. I think Cygwin handles runnig Win32 apps and being run from Win32 apps differently than Cygwin running Cygwin -- i.e. not slowly. I'll see. Could be that creating twice the number of processes, in order to avoid Cygwin running Cygwin, could be faster. Not yet known. Maybe use system() instead vfork() + exec()? Odd, though, vfork instead of fork is supposed to help. Here is the test code, you edit it to run one case or another: @if not "%1" == "" goto :%1 at rem \cygwin\bin\time cmd /c %~f0 F1 at rem \cygwin\bin\time cmd /c %~f0 F2 at rem \cygwin\bin\time cmd /c %~f0 F3@\cygwin\bin\time cmd /c %~f0 F4 at goto :eof :F1 at echo off at del *s *ofor %%a in (*.ic *.mc) do cm3cg -quiet %%a -o %%a.s"goto :eof :F2 at echo off at del *s *ofor %%a in (*.ic *.mc) do sh -c "cm3cg -quiet %%a -o %%a.s"goto :eof :F3 at del *o at echo offfor %%a in (*.s) do as %%agoto :eof :F4 at del *ofor %%a in (*.s) do sh -c "as %%a"goto :eof - Jay Helping your favorite cause is as easy as instant messaging. You IM, we give. Learn more. _________________________________________________________________ Connect and share in new ways with Windows Live. http://www.windowslive.com/share.html?ocid=TXT_TAGHM_Wave2_sharelife_012008 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Mon Mar 17 15:26:31 2008 From: jayk123 at hotmail.com (Jay) Date: Mon, 17 Mar 2008 14:26:31 +0000 Subject: [M3devel] dtoa.h? In-Reply-To: References: Message-ID: It was probably compiled with -Os. With gcc, that I don't think I built myself, the "standard" Cygwin one (3.x, not 4.x, not my doing/choice, 3.x is what they have). C can be perfectly ok, and C can be difficult to understand. The same is probably true of Modula-3. I have to punt on this for now and just not optimize. Too much else to do. Hopefully some day someone else will try NT386GNU. Once I get the spawn/perf stuff in I can try to make a mondo "std" distribution, even with all the X stuff (did I mention that Juno appears to work? :) ) and declare it done! (modulo dtoa...leave it for someone else.. maybe still fix __stdcall but it kind of drops away as mattering, NT386 works for Win32 GUI, NT386GNU works for X Windows GUI, NT386MINGNU nobody cares about) - Jay From: hosking at cs.purdue.eduTo: jay.krell at cornell.eduDate: Mon, 17 Mar 2008 10:19:04 -0400CC: m3devel at elegosoft.comSubject: Re: [M3devel] dtoa.h? How was dtoa compiled? That code is a fragile part of the run-time system that uses C that breaks with some C compilers. (See why I hate non-portable C!). Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 17, 2008, at 6:11 AM, Jay wrote: any advise to for debuging..I think I'll just turn off optimization..that fixed it...no patience to debug codegen bugs or anything in dtoa..confusing stuff...This wasn't occuring..but now I can repro it consistently compiling some of m3core/extended*, m3core/real*, m3core/longreal*. The rest of m3core compiles ok.. Program received signal SIGSEGV, Segmentation fault.mult (a=0x1426860, b=0x1426880) at ..//src//Csupport//Common/dtoa.h:535535 *x = 0;(gdb) bt#0 mult (a=0x1426860, b=0x1426880) at ..//src//Csupport//Common/dtoa.h:535#1 0x00626721 in pow5mult (b=0x1426880, k=1185) at ..//src//Csupport//Common/dtoa.h:620#2 0x00628011 in m3_dtoa (d=4.9406564584124654e-324, mode=2, ndigits=14, decpt=0x22bc74, sign=0x22bc60, rve=0x22bc68) at ..//src//Csupport//Common/dtoa.h:2224#3 0x0061534d in Convert__InternalFromLongFloat (M3_CKMnXU_buf=0x22be58, M3_CtKayy_v=4.9406564584124654e-324, M3_AcxOUs_p=13, M3_BTpO8m_style=2 '\002', M3_Bkn9rd_exponentChar=88 'X') at ..\src\convert\Convert.m3:367#4 0x00614c1f in Convert__FromExtended (M3_CKMnXU_buf=0x22be58, M3_C4YHmT_v=4.9406564584124654e-324, M3_AcxOUs_p=13, M3_BTpO8m_style=2 '\002') at ..\src\convert\Convert.m3:282#5 0x005bb6ca in M3Buf__PutFloat (M3_DmgyYA_t=0x7fe9d290, M3_DYj8sa_f=0x7fed4b5c) at ..\src\M3Buf.m3:156#6 0x005760c9 in ReelExpr__GenFPLiteral (M3_DopDyP_p=0x7fed4b4c, M3_DmgyYA_buf=0x7fe9d290) at ..\src\exprs\ReelExpr.m3:278#7 0x0054bf68 in Expr__GenFPLiteral (M3_ES44mX_t=0x7fed4b4c, M3_DmgyYA_mbuf=0x7fe9d290) at ..\src\exprs\Expr.m3:224#8 0x005301b2 in Constant__AddFPTag (M3_AjdbKy_t=0x7fed4abc, M3_BlO1Kl_x=0x22bf68) at ..\src\values\Constant.m3:225#9 0x004cb94c in Value__AddFPTag (M3_EjfEr4_t=0x7fed4abc, M3_BlO1Kl_x=0x22bf68) at ..\src\values\Value.m3:143#10 0x004cb6e1 in Value__ToFP (M3_D9M1dT__result=0x22bff4, M3_EjfEr4_t=0x7fed4abc) at ..\src\values\Value.m3:120#11 0x004cbec3 in Value__Declare (M3_EjfEr4_t=0x7fed4abc) at ..\src\values\Value.m3:202#12 0x004bb7dd in Scope__Enter (M3_DurAC9_t=0x7fed48e0) at ..\src\misc\Scope.m3:239#13 0x004b2d70 in Module__CompileInterface (M3_DZ1mTg_t=0x7fecf77c) at ..\src\values\Module.m3:870#14 0x004b2985 in Module__Compile (M3_DZ1mTg_t=0x7fecf77c) at ..\src\values\Module.m3:836#15 0x0049fd4d in M3Front__DoCompile () at ..\src\misc\M3Front.m3:196#16 0x0049f37a in M3Front__Compile (M3_AT7GN4_input=0x22c1ec, M3_AUoUsm_env=0x7fec084c, M3_BySCJy_options=0x7ff6415c) at ..\src\misc\M3Front.m3:64#17 0x0040aad8 in Builder__RunM3 (M3_Bd9FNt_s=0x7ff3de1c, M3_AXLf8B_u=0x7fecc1dc, M3_Bd56fi_object=0x7ff640e8) at ..\src\Builder.m3:1454#18 0x0040927b in Builder__PushOneM3 (M3_Bd9FNt_s=0x7ff3de1c, M3_AXLf8B_u=0x7fecc1dc) at ..\src\Builder.m3:1125#19 0x00408e94 in Builder__CompileM3 (M3_Bd9FNt_s=0x7ff3de1c, M3_AXLf8B_u=0x7fecc1dc) at ..\src\Builder.m3:1076#20 0x00407b15 in Builder__CompileOne (M3_Bd9FNt_s=0x7ff3de1c,---Type to continue, or q to quit---qM3_AXLf8B_u=0x7fQuit) at ..\src\Builder.m3:911#21 0x00409f46 in Builder__CheckImp (M3_Bd9FNt_s=0x7ff3de1c, M3_A7mgsK_u=0x7ff79d54, M3_EIVwTa_z=0x7ff79d68, M3_A2QN6Z_kind=1 '\001') at ..\src\Builder.m3:1292#22 0x00409e45 in Builder__CheckImports (M3_Bd9FNt_s=0x7ff3de1c, M3_A7mgsK_u=0x7ff79d54) at ..\src\Builder.m3:1279#23 0x00409acf in Builder__M3isStale (M3_Bd9FNt_s=0x7ff3de1c, M3_AXLf8B_u=0x7fe5c270) at ..\src\Builder.m3:1229#24 0x00408e4b in Builder__CompileM3 (M3_Bd9FNt_s=0x7ff3de1c, M3_AXLf8B_u=0x7fe5c270) at ..\src\Builder.m3:1071#25 0x00407b15 in Builder__CompileOne (M3_Bd9FNt_s=0x7ff3de1c, M3_AXLf8B_u=0x7fe5c270) at ..\src\Builder.m3:911#26 0x004077e7 in Builder__CompileEverything (M3_Bd9FNt_s=0x7ff3de1c, M3_Cw4bpV_schedule=0x7ff247f0) at ..\src\Builder.m3:871#27 0x0040391d in Builder__CompileUnits (M3_Bd56fi_main=0x7ff67014, M3_EEuw3X_units=0x7ff7e73c, M3_C1FTrk_sys_libs=0x7ff3dd60, M3_A2QN6Z_info_kind=17 '\021', M3_An02H2_mach=0x7ff7e63c) at ..\src\Builder.m3:239#28 0x0040250a in Builder__BuildLib (M3_Bd56fi_lib=0x7ff67014, M3_EEuw3X_units=0x7ff7e73c, M3_C1FTrk_sys_libs=0x7ff3dd60, M3_AicXUJ_shared=1 '\001', M3_An02H2_m=0x7ff7e63c) at ..\src\Builder.m3:36#29 0x0041afd0 in M3Build__DoLibrary (M3_An02H2_m=0x7ff7e63c, M3_AcxOUs_n_args=1) at ..\src\M3Build.m3:1365#30 0x00477f27 in QMachine__DoCall (M3_An02H2_t=0x7ff7e63c, M3_AcxOUs_n_args=1, M3_AicXUJ_isFunc=0 '\0', M3_AicXUJ_outer=0 '\0') at ..\src\QMachine.m3:529#31 0x00476f1d in QMachine__Eval (M3_An02H2_t=0x7ff7e63c) at ..\src\QMachine.m3:405#32 0x0047f1f3 in QMachine (M3_An02H2_t=0x7ff7e63c, M3_Bd56fi_path=0x7ff939b0, M3_AicXUJ_from_code=0 '\0') at ..\src\QMachine.m3:1660#33 0x0047f038 in QMachine (M3_An02H2_t=0x7ff7e63c, M3_Bd56fi_path=0x7ff939b0) at ..\src\QMachine.m3:1631#34 0x00417959 in M3Build (M3_ABp1Zk_t=0x7ff7e63c, M3_Bd56fi_file=0x7ff939b0) at ..\src\M3Build.m3:579#35 0x0041935f in M3Build__DoIncludeDir (M3_An02H2_m=0x7ff7e63c, M3_AcxOUs_n_args=1) at ..\src\M3Build.m3:954#36 0x00477f27 in QMachine__DoCall (M3_An02H2_t=0x7ff7e63c, M3_AcxOUs_n_args=1, M3_AicXUJ_isFunc=0 '\0', M3_AicXUJ_outer=0 '\0') at ..\src\QMachine.m3:529#37 0x00476f1d in QMachine__Eval (M3_An02H2_t=0x7ff7e63c) at ..\src\QMachine.m3:405#38 0x004756ef in QMachine__Evaluate (M3_An02H2_t=0x7ff7e63c, M3_CYwAos_s=0x7ff93568) at ..\src\QMachine.m3:154#39 0x00486ad2 in Quake__Run (M3_An02H2_m=0x7ff7e63c,---Type to continue, or q to quit---qM3_Bd56fi_source_file=0x7Quit) at ..\src\Quake.m3:19#40 0x00415ee7 in M3Build__Run (M3_ABp1Zk_t=0x7ff7e63c, M3_Bd56fi_makefile=0x7ff9243c) at ..\src\M3Build.m3:183#41 0x0042c706 in Main__DoIt () at ..\src\Main.m3:81#42 0x0042c9dc in Main_M3 (M3_AcxOUs_mode=1) at ..\src\Main.m3:195#43 0x005fa1ee in RTLinker__RunMainBody (M3_DjPxE3_m=0x63c020) at ..\src\runtime\common\RTLinker.m3:399#44 0x005f9719 in RTLinker__AddUnitI (M3_DjPxE3_m=0x63c020) at ..\src\runtime\common\RTLinker.m3:113#45 0x005f97a0 in RTLinker__AddUnit (M3_DjPxE5_b=0x42c9c7) at ..\src\runtime\common\RTLinker.m3:122#46 0x00401085 in main (argc=1, argv=0x14218e8, envp=0x1420090) at _m3main.mc:4 Helping your favorite cause is as easy as instant messaging. You IM, we give. Learn more. _________________________________________________________________ Connect and share in new ways with Windows Live. http://www.windowslive.com/share.html?ocid=TXT_TAGHM_Wave2_sharelife_012008 -------------- next part -------------- An HTML attachment was scrubbed... URL: From hendrik at topoi.pooq.com Mon Mar 17 15:40:36 2008 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Mon, 17 Mar 2008 10:40:36 -0400 Subject: [M3devel] SVN migration, also promotion In-Reply-To: <5E62AA15-6F63-4032-AFCB-78028301E44D@cs.purdue.edu> References: <1205592376.29245.11.camel@hias.m3w.org> <20080315190300.xc13iviopws0ocoo@mail.elegosoft.com> <1205623790.29245.19.camel@hias.m3w.org> <1205654693.29245.34.camel@hias.m3w.org> <20080316134425.bog0v1yz48o8ckss@mail.elegosoft.com> <20080317031040.GA30645@topoi.pooq.com> <5E62AA15-6F63-4032-AFCB-78028301E44D@cs.purdue.edu> Message-ID: <20080317144036.GA32117@topoi.pooq.com> On Mon, Mar 17, 2008 at 10:21:00AM -0400, Tony Hosking wrote: > Whatever system is used, it should be widely adoptable on many > platforms, and have strong support from the broadest community. To my > mind, that means only cvs or svn at the present time. Probably true. In-house, I use monotone. It's distributed. But what I like about it is its utter paranoia about losing data for any reason, even hardware failure. It does a lot of data-base consistency checks, and refuses to sync with other incarnations of itself if anything is wrong (thereby not spreading corruption). I seem to remember rumours of an attempt to make svn into a distributed system, but I don't recall where. -- hendrik From jayk123 at hotmail.com Mon Mar 17 15:46:03 2008 From: jayk123 at hotmail.com (Jay) Date: Mon, 17 Mar 2008 14:46:03 +0000 Subject: [M3devel] SVN migration, also promotion In-Reply-To: <5E62AA15-6F63-4032-AFCB-78028301E44D@cs.purdue.edu> References: <1205592376.29245.11.camel@hias.m3w.org> <20080315190300.xc13iviopws0ocoo@mail.elegosoft.com> <1205623790.29245.19.camel@hias.m3w.org> <1205654693.29245.34.camel@hias.m3w.org> <20080316134425.bog0v1yz48o8ckss@mail.elegosoft.com> <20080317031040.GA30645@topoi.pooq.com> <5E62AA15-6F63-4032-AFCB-78028301E44D@cs.purdue.edu> Message-ID: I should just stay out of this.... > widely adoptable on many platforms "As someone who uses Win32.." There appear to be a few other contenders. Monotone mostly. Maybe git. SVN certainly seems to be the all around popularity leader. Perforce is excellent and I think free (beer) in this context. Gaining in popularity, but not open source. I can't claim to know half of what Olaf knows, but I did do a fair amount of research, with unusual goals. I wanted source control just for myself. I don't need branching. I want changesets. cvs fails. I don't want to host anything myself. None of my machines are always on. They are all laptops. I want to access the content from friend's place. I was willing to spend money (and am spending some). A Windows-only solution would have been ok, though perhaps not preferred. I'd like to be able to see diffs in my current workspace without going to the network. Ability to see history without the network is nice but not crucial. The only hosting I could find was for CVS and SVN. This was critical. "Code Co-Op" is a Windows-only system that is cheap, "distributed", has an initially-crazy seeming transport that actually is a good idea and works (email, setting up a GMail account for each node), does a good job in the GUI of walking you through the workflow. I just had a few complaints about how it worked, the cost ($150 per node I think), and Windows-only. It has the same problem as SVN as not knowing what changes have been picked up where, and you create that temporary branch situation just by editing a file on two machines. I ended up with SVN hosting for under $10/month. GUI and command line. Available everywhere. Change sets. Not distributed, but ok. >From what I read, SVN does not do branching correctly. In multiple ways. First, they don't remember what changes have been merged to what branches. It is up to people to know. That is horribly broken. Last I checked it was going to be fixed soon, at least in the ui. Second, it seems they have just mainline and branches off it. They don't seem to really get branching like Perforce. In Perforce, the branching structure is totally hierarchical, and you don't HAVE to merge anything to anything, and a KEY feature is that it keeps track of what has been integrated where. But it isn't distributed, isn't open source, isn't super cheap for commercial use (like $800 per seat). It is nice though. OpenWatcom uses it, for example. I could not find any hosting service. I might have spent the money if I could find one. I'm also pretty sure the CVS web ui could be much better with little work. I want to see the last n changes. The last n day's changes. And then quickly view the diffs of those changes. Currently I have to navigate all around and click individual files. Branching can be organizationally expensive. You have to know what is branched. What changes have been picked up. Which branches are buildable and pass tests (tinderbox per branch). Too much branching can be bad. > and have strong support from the broadest community What will happen EVENTUALLY is CVS support will die off in favor of SVN or other. I don't see community support for sccs and rcs for example. I'm using http://www.svnrepository.com/ It would be nice if a source control system only dropped files/directories at the root of a tree, instead of in every single directory. Like Perforce. Unlike CVS and SVN. - Jay From: hosking at cs.purdue.eduTo: hendrik at topoi.pooq.comDate: Mon, 17 Mar 2008 10:21:00 -0400CC: m3devel at elegosoft.comSubject: Re: [M3devel] SVN migration, also promotion Whatever system is used, it should be widely adoptable on many platforms, and have strong support from the broadest community. To my mind, that means only cvs or svn at the present time. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 16, 2008, at 11:10 PM, hendrik at topoi.pooq.com wrote: On Sun, Mar 16, 2008 at 01:44:25PM +0100, Olaf Wagner wrote: I don't think it will be furthered by switching from CVS to SVN or any other tool change....... If there really is the wish within the community to switch from CVS to SVN, then those interested should build a team, investigate the impacts and propose a migration plan. Elego and me for one would need new server setup, the actual repository migration (which should preserve as much information as possible), setup of repository replication to two machines, integration of SVN into the regression test framework, setup of SVN clients, update / rewrite of all the repository access information and CM guidelines, and a migration strategy for all the existing HTTP links in search engines etc. If I should make an estimation for this project in a commercial environment, I'd say it will take 3-6 months and cost a client between 40,000 and 80,000 Euros (that's currently 60,000 to 120,000$). I'd rather invest these efforts for other goals. If other CM3 users think it is worthwhile to do, they should just do it; after all, this is an open source project.If such a group does reorganise the Modula 3 sources into a new revision control system, though, I'd strongly advise them to look into a distributed one.-- hendrik _________________________________________________________________ Climb to the top of the charts!?Play the word scramble challenge with star power. http://club.live.com/star_shuffle.aspx?icid=starshuffle_wlmailtextlink_jan -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Mon Mar 17 15:46:04 2008 From: jayk123 at hotmail.com (Jay) Date: Mon, 17 Mar 2008 14:46:04 +0000 Subject: [M3devel] SVN migration, also promotion In-Reply-To: <5E62AA15-6F63-4032-AFCB-78028301E44D@cs.purdue.edu> References: <1205592376.29245.11.camel@hias.m3w.org> <20080315190300.xc13iviopws0ocoo@mail.elegosoft.com> <1205623790.29245.19.camel@hias.m3w.org> <1205654693.29245.34.camel@hias.m3w.org> <20080316134425.bog0v1yz48o8ckss@mail.elegosoft.com> <20080317031040.GA30645@topoi.pooq.com> <5E62AA15-6F63-4032-AFCB-78028301E44D@cs.purdue.edu> Message-ID: I should just stay out of this.... > widely adoptable on many platforms "As someone who uses Win32.." There appear to be a few other contenders. Monotone mostly. Maybe git. SVN certainly seems to be the all around popularity leader. Perforce is excellent and I think free (beer) in this context. Gaining in popularity, but not open source. I can't claim to know half of what Olaf knows, but I did do a fair amount of research, with unusual goals. I wanted source control just for myself. I don't need branching. I want changesets. cvs fails. I don't want to host anything myself. None of my machines are always on. They are all laptops. I want to access the content from friend's place. I was willing to spend money (and am spending some). A Windows-only solution would have been ok, though perhaps not preferred. I'd like to be able to see diffs in my current workspace without going to the network. Ability to see history without the network is nice but not crucial. The only hosting I could find was for CVS and SVN. This was critical. "Code Co-Op" is a Windows-only system that is cheap, "distributed", has an initially-crazy seeming transport that actually is a good idea and works (email, setting up a GMail account for each node), does a good job in the GUI of walking you through the workflow. I just had a few complaints about how it worked, the cost ($150 per node I think), and Windows-only. It has the same problem as SVN as not knowing what changes have been picked up where, and you create that temporary branch situation just by editing a file on two machines. I ended up with SVN hosting for under $10/month. GUI and command line. Available everywhere. Change sets. Not distributed, but ok. >From what I read, SVN does not do branching correctly. In multiple ways. First, they don't remember what changes have been merged to what branches. It is up to people to know. That is horribly broken. Last I checked it was going to be fixed soon, at least in the ui. Second, it seems they have just mainline and branches off it. They don't seem to really get branching like Perforce. In Perforce, the branching structure is totally hierarchical, and you don't HAVE to merge anything to anything, and a KEY feature is that it keeps track of what has been integrated where. But it isn't distributed, isn't open source, isn't super cheap for commercial use (like $800 per seat). It is nice though. OpenWatcom uses it, for example. I could not find any hosting service. I might have spent the money if I could find one. I'm also pretty sure the CVS web ui could be much better with little work. I want to see the last n changes. The last n day's changes. And then quickly view the diffs of those changes. Currently I have to navigate all around and click individual files. Branching can be organizationally expensive. You have to know what is branched. What changes have been picked up. Which branches are buildable and pass tests (tinderbox per branch). Too much branching can be bad. > and have strong support from the broadest community What will happen EVENTUALLY is CVS support will die off in favor of SVN or other. I don't see community support for sccs and rcs for example. I'm using http://www.svnrepository.com/ It would be nice if a source control system only dropped files/directories at the root of a tree, instead of in every single directory. Like Perforce. Unlike CVS and SVN. - Jay From: hosking at cs.purdue.eduTo: hendrik at topoi.pooq.comDate: Mon, 17 Mar 2008 10:21:00 -0400CC: m3devel at elegosoft.comSubject: Re: [M3devel] SVN migration, also promotion Whatever system is used, it should be widely adoptable on many platforms, and have strong support from the broadest community. To my mind, that means only cvs or svn at the present time. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 16, 2008, at 11:10 PM, hendrik at topoi.pooq.com wrote: On Sun, Mar 16, 2008 at 01:44:25PM +0100, Olaf Wagner wrote: I don't think it will be furthered by switching from CVS to SVN or any other tool change....... If there really is the wish within the community to switch from CVS to SVN, then those interested should build a team, investigate the impacts and propose a migration plan. Elego and me for one would need new server setup, the actual repository migration (which should preserve as much information as possible), setup of repository replication to two machines, integration of SVN into the regression test framework, setup of SVN clients, update / rewrite of all the repository access information and CM guidelines, and a migration strategy for all the existing HTTP links in search engines etc. If I should make an estimation for this project in a commercial environment, I'd say it will take 3-6 months and cost a client between 40,000 and 80,000 Euros (that's currently 60,000 to 120,000$). I'd rather invest these efforts for other goals. If other CM3 users think it is worthwhile to do, they should just do it; after all, this is an open source project.If such a group does reorganise the Modula 3 sources into a new revision control system, though, I'd strongly advise them to look into a distributed one.-- hendrik _________________________________________________________________ Climb to the top of the charts!?Play the word scramble challenge with star power. http://club.live.com/star_shuffle.aspx?icid=starshuffle_wlmailtextlink_jan -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Mon Mar 17 15:49:37 2008 From: jayk123 at hotmail.com (Jay) Date: Mon, 17 Mar 2008 14:49:37 +0000 Subject: [M3devel] SVN migration, also promotion In-Reply-To: References: <1205592376.29245.11.camel@hias.m3w.org> <20080315190300.xc13iviopws0ocoo@mail.elegosoft.com> <1205623790.29245.19.camel@hias.m3w.org> <1205654693.29245.34.camel@hias.m3w.org> <20080316134425.bog0v1yz48o8ckss@mail.elegosoft.com> <20080317031040.GA30645@topoi.pooq.com> <5E62AA15-6F63-4032-AFCB-78028301E44D@cs.purdue.edu> Message-ID: > The only hosting I could find was for CVS and SVN. This was critical. > I'm using http://www.svnrepository.com/ They have GIT hosting now too. - Jay From: jayk123 at hotmail.comTo: hosking at cs.purdue.edu; hendrik at topoi.pooq.comDate: Mon, 17 Mar 2008 14:46:04 +0000CC: m3devel at elegosoft.comSubject: Re: [M3devel] SVN migration, also promotion _________________________________________________________________ Shed those extra pounds with MSN and The Biggest Loser! http://biggestloser.msn.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Mon Mar 17 19:01:18 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Mon, 17 Mar 2008 19:01:18 +0100 Subject: [M3devel] SVN migration, also promotion In-Reply-To: References: <1205592376.29245.11.camel@hias.m3w.org> <20080315190300.xc13iviopws0ocoo@mail.elegosoft.com> <1205623790.29245.19.camel@hias.m3w.org> <1205654693.29245.34.camel@hias.m3w.org> <20080316134425.bog0v1yz48o8ckss@mail.elegosoft.com> <20080317031040.GA30645@topoi.pooq.com> <5E62AA15-6F63-4032-AFCB-78028301E44D@cs.purdue.edu> Message-ID: <20080317190118.fls77ickw048848o@mail.elegosoft.com> Quoting Jay : > I wanted source control just for myself. > I don't need branching. > I want changesets. cvs fails. DCVS is a compatible extension of CVS (done by Elego) for a network of distributed CVS servers, which also features snapshots and changesets (which should have been ported back to CVS, but nobody has done that yet). It has never come into wide use, as SVN's rise has been too thriving and their marketing too good :-) We're now doing much with SVN ourselves, but are still using DCVS for some internal purposes. I still do not suggest to change anything for CM3, just wanted to mention that it's really not that hard to extend CVS with changesets etc. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From wagner at elegosoft.com Tue Mar 18 00:06:15 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 18 Mar 2008 00:06:15 +0100 Subject: [M3devel] Trac and other news Message-ID: <20080318000615.9s58wu6u34cosk0c@mail.elegosoft.com> I've updated the news items on the web page. The old GNATS bug tracker has been replaced by a trac installation for CM3. Everybody who wants WiKi write access or administrative access should send a mail to m3-support at elego.de and save his favourite password in his ~/.tracpass file with private permissions (0600) on birch.elegosoft.com. If anybody wants to take care for the administrative side of CM3 trac, it will be appreciated. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jayk123 at hotmail.com Tue Mar 18 10:06:24 2008 From: jayk123 at hotmail.com (Jay) Date: Tue, 18 Mar 2008 09:06:24 +0000 Subject: [M3devel] possible cygwin createprocess/fork cost measurements.. Message-ID: There are a surprising number of options here. What I describe below is ok. Other options include: The heuristic I define below is not bad. If a command has none of $%^\*?()&|=;<>{}[]'"`~ then no sh wrapper is needed. Right? Heck -- define the set as anything but [a-z0-9-/ ]? What about shell builtins? Are they useful?Code that requires a sh wrapper is veering into non-portability, butsome of the sh semantics are the same on NT and Posix.Specifically <>|&. If code absolutely needed a sh wrapper, and wasn't portable anyway, itcould be provided manually: exec("sh -c \"echo foo\"") As well, a new prefix character could be used for "direct" or "fast" exec. As well, since m3cg and as are the most interesting by far, they could bydefault be implemented in Modula-3, without a sh wrapper, and only ifa Quake function is provided, use it. As well, "half" the gain could be had by adding a switch to cm3cg to have itrun as itself. Since this is only half the gain, not worthwhile. As well, all of this would be gated on some global like Process.IsSpawnFaster()and/or Process.IsShWrapperSlow(), so that the default behavior would be unchanged. As well, one thing my measurements (or reporting) failed to separately point outis if it is the sh vs. no-sh or vfork/exec vs. spawn that is slow.I'll have to test those separately.Likely that they each add slowness, but separate measure would be good. ok, I ran a bunch more tests.A small "combinatorial explosion". The sh wrapper is the larger hit, but spawn is still measurably faster than fork/exec.ash and zsh are also faster than sh. But no matter, I'll use none.Ubuntu uses ash for /bin/sh.There are some surprising cases where "Cyg" is faster than "MS", but not where it matters here.cygwin system and "crt0.o" seem faster, at least in this context.I'm suspicious of those numbers though.Could be something is slowing down the MS paths here. The test code is attached. Relevant excerpts from the data are: (NOTE this is a different machine than the other numbers, and on XP.) 100 iterations: current behavior:vfork/exec(sh -c ./cygnothing) 7.489000 change to spawn but keep sh wrapper:spawn(nowait, sh -c ./cygnothing)/waitpid 5.323000 keep fork/exec but lose sh wrapper:vfork/exec(.\cygnothing) 4.133000 lose sh wrapper and fork/execspawn(nowait, .\cygnothing)/waitpid 1.946000 some of the fastest cases: spawn(wait, .\cygnothing) 2.009000 msvcrt_system(.\cygnothing) 0.313000 spawn(nowait, .\ms1nothing)/waitpid 0.188000 spawn(wait, .\ms1nothing) 0.062000 msvcrt_system(.\ms1nothing) 0.280000 msvcrt_system(.\ms2nothing) 0.156000 spawn(nowait, .\ms3nothing)/waitpid 0.094000 spawn(wait, .\ms3nothing) 0.203000 msvcrt_system(.\ms3nothing) 0.265000 I hadn't previously seen such much faster numbersthan I was going to easily achieve, so let's try with 500 iterations:(if you don't like the numbers, run again... :) ) current behavior: vfork/exec(sh -c ./cygnothing) 37.415000 replace fork/exec with spawn spawn(nowait, sh -c ./cygnothing)/waitpid 26.851000 spawn(wait, sh -c ./cygnothing) 27.017000 (why is this slower?) cygwin1 startup code in child process measurably slow! vfork/exec(sh -c ./ms1nothing) 28.208000 spawn(nowait, sh -c ./ms1nothing)/waitpid 17.549000 spawn(wait, sh -c ./ms1nothing) 17.506000 remove sh wrapper, cut cost by /roughly/ 50% vfork/exec(.\cygnothing) 20.321000 remove sh wrapper and fork/exe, another /roughly/ 50% spawn(nowait, .\cygnothing)/waitpid 9.933000 and again there are some much faster options, all around 1 second: msvcrt_system(.\cygnothing) 1.125000 spawn(nowait, .\ms1nothing)/waitpid 0.766000 spawn(wait, .\ms1nothing) 0.733000 msvcrt_system(.\ms1nothing) 1.157000 spawn(nowait, .\ms2nothing)/waitpid 0.875000 spawn(wait, .\ms2nothing) 0.938000 msvcrt_system(.\ms2nothing) 1.142000 spawn(nowait, .\ms3nothing)/waitpid 1.204000 spawn(wait, .\ms3nothing) 0.265000 The best, but doesn't make sense. msvcrt_system(.\ms3nothing) 1.187000 I think ultimately the thing to do is try using the Win32 code, and then http://cygwin.com/cygwin-api/cygwin-functions.html#func-cygwin-attach-handle-to-fd to get back child handles for stdin/out/err and http://cygwin.com/cygwin-api/func-cygwin-winpid-to-pid.html to get a Cygwin pid, if that works having run CreateProcess. That is, with the above numbers, my plan so far would get from 37 seconds to 10 seconds, but getting down to 1 second or less may be viable. I should run these tests on MacOSX/PowerPC and Linux/PowerPPC too.. Since that .265 number is wierd, let's run again: current behavior: vfork/exec(sh -c ./cygnothing) 37.669000 lose fork/exec, keep sh wrapper spawn(nowait, sh -c ./cygnothing)/waitpid 26.823000 similarly, lose cygwin startup code (hm!) vfork/exec(sh -c ./ms1nothing) 27.941000 lose fork/exec again (again notice cygwin1.dll dependency hurts!) spawn(nowait, sh -c ./ms1nothing)/waitpid 17.595000 lose sh wrapper vfork/exec(.\cygnothing) 20.612000 lose sh wrapper and fork/exec spawn(nowait, .\cygnothing)/waitpid 9.965000 and again faster cases of 1 second or less: msvcrt_system(.\cygnothing) 1.172000 spawn(nowait, .\ms1nothing)/waitpid 0.531000 spawn(wait, .\ms1nothing) 0.594000 msvcrt_system(.\ms1nothing) 1.297000 spawn(nowait, .\ms2nothing)/waitpid 0.999000 spawn(wait, .\ms2nothing) 0.860000 msvcrt_system(.\ms2nothing) 1.047000 spawn(nowait, .\ms3nothing)/waitpid 0.562000 spawn(wait, .\ms3nothing) 0.688000 msvcrt_system(.\ms3nothing) 1.203000 Lastly, let's try CreateProcess directly.Ok, CreateProcess gets under .4 seconds consistently running cygnothing, but cygwin_winpid_to_pid does not work. I'll see what I can do. It'd be nice to get to the 1 or .5 second case, and not just the 10 second case.Though 10 is still preferable to 37. Btw, notice, no more annoying ads in my email! :) - Jay From: jayk123 at hotmail.comTo: m3devel at elegosoft.comSubject: RE: possible cygwin createprocess/fork cost measurements..Date: Mon, 17 Mar 2008 14:21:05 +0000 I believe cm3 is affected by this.I don't have numbers yet. I propose some fairly obvious/small/simple/safe changes in order to likely achieve a large speed up in NT386GNU.I am skeptical that existing functions can be changed in a compatible enough way. So I propose, roughly: Add this to Process.i3: PROCEDURE Spawn(cmd: Pathname.T; READONLY params: ARRAY OF TEXT) : T RAISES {OSError.E};(* A restricted form of Create that is much faster on Cygwin. *) The name is very iffy.It could be in fact not be in the public interface, but merely notice if wd = stdin = stdout = stderr = nil.It could probably be in be less limited than shown.Probably all of the parameters are settable, by altering the parent's globals, within a critical section.Environment certainly is settable.It is tempting to leave it limited like this though, such as to be implementable perhaps with system.(It turns out Cygwin system is slower than spawnve; surprising since system is the most limited of the exec/spawn variants -- I think related to it having an implied sh wrapper but the others do not.)The intent is simple and obvious -- some path to spawnve or spawnvpe.p has path search. On all but Cygwin, this limited Create/Spawn will just call the normal Create. (Even on Win32).On Cygwin it will call spawnvpe (or spawnve if people really want, but "p" seems "better"). Now, in Quake, all the existing exec variants wrap the command line in either sh or cmd or command.com.Changing that is probably very dangerous, even with an attempt to discern if the wrapper buys anything, on a command line by command line basis.For example, if all of the characters * ? | < > % $ & ; ( ) are absent from the command, the shell wrapper probably doesn't buy anything and could be removed from existing paths. However that's not true -- for example system("echo foo") depends on a shell wrapper to run the builtin "echo" (at least on Windows, there is no echo.exe). I think there's no choice but to add a new Quake function, spawn, or limited_exec, or fast_exec, or process_runfast, exec_noshell, or something.Again I'm not sure what to call it, but it'd simply call Process.Spawn, or Process.Create but with right the right parameters to get into the Cygwin fast path.For now I'm going with Process.Spawn and fast_exec.I hope to have numbers "soon" as to the perf change. Another good option here, that I tried in the past but failed, and is partly not difficult to solve, but also partly, is to implemet Quake exec using Win32 CreateProcess instead of Cygwin spawn/exec. There are at least two sets of problems here. One is that the existing code returns a File.T, and for that there is the Posix and Win32 types, Cygwin uses Posix. You'd have to warp the code somehow here. I gave up on that point without much trying. It's not that much code though. Cygwin is using CreateProcess of course.The other problem is on the input, the interpretation of the command line. Again this is the value that presently Cygwin provides (albeit sometimes with great cost). Of course another angle is to work on Cygwin to make vfork efficient. It is presently implemented by calling fork.There is #ifdef'ed code for another path but it appears not enabled. I know polluting the system just for the sake of Cygwin isn't great, however: - I expect the win is quite large - "spawn*" is a pretty old thing, nothing new/controversial here, long known as an often viable replacement for fork+exec at least on Windows. It's in msvc*.dll for example. There may even be wins to be had on other Posix systems by avoiding the sh wrapper? "batching" where cm3cg is run once per directory seems like a very good idea and worth trying; the problem is, that still leaves the assembler.Perhaps the assembler could be linked in statically to cm3cg? Probably, but not particularly easily and probably unpopular upstream...Unless maybe some nice gcc perf gains would be demonstrated? - Jay From: jayk123 at hotmail.comTo: m3devel at elegosoft.comSubject: possible cygwin createprocess/fork cost measurements..Date: Mon, 17 Mar 2008 10:14:50 +0000 I ran some mostly scientific measures of Cygwin. On one machine, no reboots, one OS, one set of files. x86, single proc, Windows 2000 (I'll go back to XP soon). It shows that..well, at least that wrapping Cygwin processes with sh is VERY expensive. Like, the data isn't yet complete, but this could cut building Cygwin libm3 from around 100 seconds to around 20 seconds. Not counting the Modula-3 front end time. Just cm3cg+as. cd libm3\NT386GNU having already built successfully, all the *.ic *.mc files are present cm3cg not wrapped with sh (F1) Repeated runs. 28 seconds (other stuff running on machine) 16 seconds 13 seconds (13.?) 13.8 seconds 14.01 seconds 13.3 seconds now add the -o flag 13.64 seconds 14.07 seconds now without echoing 13.22 seconds 13.18 seconds cm3cg wrapped with sh (F2) 51 seconds 51.35 seconds 51.19 seconds 50.88 seconds now add the -o flag 51.76 seconds now without echoing 51.05 seconds These runs did NOT have -o flags, but subsequent runs with -o were about the same. I added -o so I could run the as variations. now the same with .s note that due to the way the above worked, I just have *.s files, and not the usual *.is and *.ms as not wrapped with sh (F3) 5.6 seconds 5.28 seconds now remove echo 5.08 seconds 5.08 seconds 5.04 seconds forgot -o flag, oh well, enough data as wrapped with sh (F4) 43 seconds 43.56 seconds forgot -o flag, oh well, enough data What is not yet confirmed is: 1) Does cm3 wrap everything with sh? 2) Does calling m3cg/as from cm3 have these costs? Very clear: Wrapping stuff with sh on Cygwin is expensive! Actions: Confirm this cost is being paid by cm3. Either: 1) implement some "batch modes" in cm3 and/or cm3cg 2) or maybe, um, just make sure that cm3 does not wrap with sh, and if cm3 itself causes this slowdown, because of how Cygwin works, try interposing a small Win32 helper app. I think Cygwin handles runnig Win32 apps and being run from Win32 apps differently than Cygwin running Cygwin -- i.e. not slowly. I'll see. Could be that creating twice the number of processes, in order to avoid Cygwin running Cygwin, could be faster. Not yet known. Maybe use system() instead vfork() + exec()? Odd, though, vfork instead of fork is supposed to help. Here is the test code, you edit it to run one case or another: @if not "%1" == "" goto :%1 at rem \cygwin\bin\time cmd /c %~f0 F1 at rem \cygwin\bin\time cmd /c %~f0 F2 at rem \cygwin\bin\time cmd /c %~f0 F3@\cygwin\bin\time cmd /c %~f0 F4 at goto :eof :F1 at echo off at del *s *ofor %%a in (*.ic *.mc) do cm3cg -quiet %%a -o %%a.s"goto :eof :F2 at echo off at del *s *ofor %%a in (*.ic *.mc) do sh -c "cm3cg -quiet %%a -o %%a.s"goto :eof :F3 at del *o at echo offfor %%a in (*.s) do as %%agoto :eof :F4 at del *ofor %%a in (*.s) do sh -c "as %%a"goto :eof - Jay Helping your favorite cause is as easy as instant messaging. You IM, we give. Learn more. Connect and share in new ways with Windows Live. Get it now! -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: fork.c URL: From jayk123 at hotmail.com Tue Mar 18 11:22:24 2008 From: jayk123 at hotmail.com (Jay) Date: Tue, 18 Mar 2008 10:22:24 +0000 Subject: [M3devel] possible cygwin createprocess/fork cost measurements.. In-Reply-To: References: Message-ID: A lot of those numbers didn't make sense, if you watch it run, though the relative values are still I think right. I "rewrote" the time measurement and added redundant time reporting and the numbers make more sense now.I also added a new faster method and renamed stuff for clarity. cygnothing -- just main returning 0 cygnocyg -- same thing but with -mno-cygwin msvcdll -- same thing but with visual c++ using msvcr80.dll msvcstatic -- same thing but using libcmt.lib msvcmin -- Visual C++ and no runtime, just TerminateProcess (ExitProcess is "nicer") spawn(nowait) + waitpid is about the same as spawn(wait), and is close to system() in inflexibility, but system() has an implicit shell wrapper and so is slow. It seems that spawn is faster than CreateProcess at running .exes that use cygwin1.dll. That is a little odd. It must be something to do with the child startup code talking to its parent or something and doing so more efficiently when the immediate parent uses Cygwin. I might dig into that. Like, a dynamic choice between CreateProcess and spawn could be more efficient, but so far I'm not sure even using CreateProcess here is "easy enough". Is there anywhere in the cm3 tree appropriate to checkin such experimental code? I can put it in my private svn... Tue Mar 18 03:04:35 2008 vfork 1.163909Tue Mar 18 03:04:52 2008 vfork/exec(sh -c ./cygnothing) 16.815984 current behavior Tue Mar 18 03:05:06 2008 spawn(nowait, sh -c ./cygnothing)/waitpid 13.296170 eliminate fork/execTue Mar 18 03:05:19 2008 spawn(wait, sh -c ./cygnothing) 13.814871Tue Mar 18 03:05:37 2008 vfork/exec(sh -c ./cygnocyg) 17.898738 Tue Mar 18 03:05:51 2008 spawn(nowait, sh -c ./cygnocyg)/waitpid 13.545283Tue Mar 18 03:06:05 2008 spawn(wait, sh -c ./cygnocyg) 13.639426Tue Mar 18 03:06:24 2008 vfork/exec(sh -c ./msvcdll) 19.699897Tue Mar 18 03:06:40 2008 spawn(nowait, sh -c ./msvcdll)/waitpid 16.101528Tue Mar 18 03:06:57 2008 spawn(wait, sh -c ./msvcdll) 16.219237Tue Mar 18 03:07:04 2008 vfork/exec(.\cygnothing) 7.816323 eliminate sh wrapperTue Mar 18 03:07:08 2008 spawn(nowait, .\cygnothing)/waitpid 3.783029 eliminate sh and fork/exec kind of the initial goal Tue Mar 18 03:07:12 2008 spawn(wait, .\cygnothing) 3.794715Tue Mar 18 03:07:28 2008 cygwin_system(./cygnothing) 15.840816Tue Mar 18 03:07:48 2008 msvcrt_system(.\cygnothing) 20.502232Tue Mar 18 03:07:54 2008 CreateProcess(.\cygnothing) 5.853595 surprisingly slow Tue Mar 18 03:08:00 2008 vfork/exec(.\cygnocyg) 6.335709Tue Mar 18 03:08:02 2008 spawn(nowait, .\cygnocyg)/waitpid 1.959032Tue Mar 18 03:08:04 2008 spawn(wait, .\cygnocyg) 1.976474Tue Mar 18 03:08:18 2008 cygwin_system(./cygnocyg) 13.295613Tue Mar 18 03:08:34 2008 msvcrt_system(.\cygnocyg) 15.849668Tue Mar 18 03:08:35 2008 CreateProcess(.\cygnocyg) 1.541903 one of the best casesTue Mar 18 03:08:43 2008 vfork/exec(.\msvcdll) 8.139020Tue Mar 18 03:08:48 2008 spawn(nowait, .\msvcdll)/waitpid 4.357404Tue Mar 18 03:08:52 2008 spawn(wait, .\msvcdll) 4.378520Tue Mar 18 03:09:08 2008 cygwin_system(./msvcdll) 15.726932Tue Mar 18 03:09:26 2008 msvcrt_system(.\msvcdll) 18.211870Tue Mar 18 03:09:30 2008 CreateProcess(.\msvcdll) 3.901068 Tue Mar 18 03:09:36 2008 vfork/exec(.\msvcstatic) 5.783474Tue Mar 18 03:09:37 2008 spawn(nowait, .\msvcstatic)/waitpid 1.880097Tue Mar 18 03:09:39 2008 spawn(wait, .\msvcstatic) 1.915825Tue Mar 18 03:09:53 2008 cygwin_system(./msvcstatic) 13.178187Tue Mar 18 03:10:08 2008 msvcrt_system(.\msvcstatic) 15.827244Tue Mar 18 03:10:10 2008 CreateProcess(.\msvcstatic) 1.460144 one of the bestTue Mar 18 03:10:15 2008 vfork/exec(.\msvcmin) 5.491899Tue Mar 18 03:10:17 2008 spawn(nowait, .\msvcmin)/waitpid 1.674227Tue Mar 18 03:10:19 2008 spawn(wait, .\msvcmin) 1.698951Tue Mar 18 03:10:32 2008 cygwin_system(./msvcmin) 12.980475Tue Mar 18 03:10:47 2008 msvcrt_system(.\msvcmin) 15.624337Tue Mar 18 03:10:49 2008 CreateProcess(.\msvcmin) 1.276997 best case as expected - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: cygexec.c URL: From hendrik at topoi.pooq.com Wed Mar 19 14:38:52 2008 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Wed, 19 Mar 2008 09:38:52 -0400 Subject: [M3devel] random-acces I/O In-Reply-To: <1204416764.10030.12.camel@hias.m3w.org> References: <200803012249.m21MnOnF018386@camembert.async.caltech.edu> <1204416764.10030.12.camel@hias.m3w.org> Message-ID: <20080319133852.GA7741@topoi.pooq.com> On Sun, Mar 02, 2008 at 01:12:44AM +0100, Dragi?a Duri? wrote: > rdwr is package by Blair MacIntyre and it's not related to anything > File.* except indirectly, ie if you make File Wr part of TeeWr. The RdWr I found is at http://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-comm/rdwr/ It contains several .i3's amd .m3's, and seems to be a program that sets up a pipe between an Rd and a Rw. Not really what I need, not being random access. Is there some other RdWr I'm not finding? -- hendrik From dabenavidesd at yahoo.es Wed Mar 19 22:35:06 2008 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Wed, 19 Mar 2008 22:35:06 +0100 (CET) Subject: [M3devel] do-cm3-std.sh buildship -> package sysutils not found Message-ID: <62256.30048.qm@web27107.mail.ukl.yahoo.com> Hi all: I'm getting that error, when launching the script with the latests sources. Can you see what is missing, I think the sources tree is complete. Thanks adminfbd at fbd-desktop:/home/daniel/code/cm3/scripts$ sudo ./do-cm3-std.sh buildship /home/daniel/code/cm3/scripts/pkgmap.sh -c "/usr/local/cm3/bin/cm3 -build -DROOT='/home/daniel/code/cm3' -DCM3_VERSION_TEXT='d5.7.0' -DCM3_VERSION_NUMBER='050700' -DCM3_LAST_CHANGED='2008-03-16' && /usr/local/cm3/bin/cm3 -ship -DROOT='/home/daniel/code/cm3' -DCM3_VERSION_TEXT='d5.7.0' -DCM3_VERSION_NUMBER='050700' -DCM3_LAST_CHANGED='2008-03-16' " m3bundle === package /home/daniel/code/cm3/m3-tools/m3bundle === +++ /usr/local/cm3/bin/cm3 -build -DROOT='/home/daniel/code/cm3' -DCM3_VERSION_TEXT='d5.7.0' -DCM3_VERSION_NUMBER='050700' -DCM3_LAST_CHANGED='2008-03-16' && /usr/local/cm3/bin/cm3 -ship -DROOT='/home/daniel/code/cm3' -DCM3_VERSION_TEXT='d5.7.0' -DCM3_VERSION_NUMBER='050700' -DCM3_LAST_CHANGED='2008-03-16' +++ --- building in LINUXLIBC6 --- ignoring ../src/m3overrides --- shipping from LINUXLIBC6 --- . => /usr/local/cm3/pkg/m3bundle/LINUXLIBC6 .M3EXPORTS .M3WEB ../src => /usr/local/cm3/pkg/m3bundle/src m3bundle.m3 . => /usr/local/cm3/bin m3bundle . => /usr/local/cm3/man/man1 m3bundle.1 ==> /home/daniel/code/cm3/m3-tools/m3bundle done /home/daniel/code/cm3/scripts/pkgmap.sh -c "/usr/local/cm3/bin/cm3 -build -DROOT='/home/daniel/code/cm3' -DCM3_VERSION_TEXT='d5.7.0' -DCM3_VERSION_NUMBER='050700' -DCM3_LAST_CHANGED='2008-03-16' && /usr/local/cm3/bin/cm3 -ship -DROOT='/home/daniel/code/cm3' -DCM3_VERSION_TEXT='d5.7.0' -DCM3_VERSION_NUMBER='050700' -DCM3_LAST_CHANGED='2008-03-16' " import-libs m3core libm3 patternmatching sysutils unittest m3middle m3quake m3scanner m3tools m3cgcat m3cggen m3gdb m3bundle mklib fix_nl libdump arithmetic unittest-numeric bitvector digraph parseparams realgeometry set slisp sortedtableextras table-list tempfiles tcl tcp udp libsio libbuf debug listfuncs embutils m3tk-misc http binIO commandrw tapi serial m3tk mtex m3totex m3tohtml m3scan m3markup m3browser cmpdir cmpfp dirfp uniq netobj netobjd stubgen events rdwr sharedobj sharedobjgen odbc postgres95 db smalldb stablegen stable X11R4 ui PEX vbtkit cmvbt jvideo videovbt web formsvbtpixmaps formsvbt formsview formsedit codeview mg mgkit opengl anim3D zeus m3zume synloc synex metasyn obliqrt obliqparse obliqprint obliq obliqlibemb obliqlibm3 obliqlibui obliqlibanim obliqsrvstd obliqsrvui obliqbinmin obliqbinstd obliqbinui obliqbinanim visualobliq vocgi voquery vorun webvbt recordheap rehearsecode replayheap showheap shownew showthread pkl-fonts juno-machine juno-compiler juno-app cube calculator fisheye mentor package sysutils not found --------------------------------- Enviado desde Correo Yahoo! Disfruta de una bandeja de entrada m?s inteligente.. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Thu Mar 20 00:23:45 2008 From: jayk123 at hotmail.com (Jay) Date: Wed, 19 Mar 2008 23:23:45 +0000 Subject: [M3devel] do-cm3-std.sh buildship -> package sysutils not found In-Reply-To: <62256.30048.qm@web27107.mail.ukl.yahoo.com> References: <62256.30048.qm@web27107.mail.ukl.yahoo.com> Message-ID: 1) be sure us -dAP in the cvs upd command. cd /home/daniel/code/cm3 cvs -z3 -dAP Otherwise you miss out on new stuff. Or at least not delete deleted stuff. ? cvs --help to double check this. 2) delete pkgsdb rm /home/daniel/code/cm3/scripts/PKGSDB I wonder if maybe the file should have a version suffix, and when a rebuild is needed, checkin an update to bump up the version suffix, and before any rebuild, rm PKGSDB*? That should fix this problem reasonably well, simply seamless, right? Just that occasionally some folks would have to remember to update the version. The existing version numbers could do. Deleting PKGSDB when there is any doubt is always safe. It will be recreated if it doesn't exist. Recreation is slow though and worth optimizing almost all of the time. - Jay Date: Wed, 19 Mar 2008 22:35:06 +0100From: dabenavidesd at yahoo.esTo: m3devel at elegosoft.comSubject: [M3devel] do-cm3-std.sh buildship -> package sysutils not foundHi all:I'm getting that error, when launching the script with the latests sources.Can you see what is missing, I think the sources tree is complete.Thanksadminfbd at fbd-desktop:/home/daniel/code/cm3/scripts$ sudo ./do-cm3-std.sh buildship/home/daniel/code/cm3/scripts/pkgmap.sh -c "/usr/local/cm3/bin/cm3 -build -DROOT='/home/daniel/code/cm3' -DCM3_VERSION_TEXT='d5.7.0' -DCM3_VERSION_NUMBER='050700' -DCM3_LAST_CHANGED='2008-03-16' && /usr/local/cm3/bin/cm3 -ship -DROOT='/home/daniel/code/cm3' -DCM3_VERSION_TEXT='d5.7.0' -DCM3_VERSION_NUMBER='050700' -DCM3_LAST_CHANGED='2008-03-16' " m3bundle=== package /home/daniel/code/cm3/m3-tools/m3bundle === +++ /usr/local/cm3/bin/cm3 -build -DROOT='/home/daniel/code/cm3' -DCM3_VERSION_TEXT='d5.7.0' -DCM3_VERSION_NUMBER='050700' -DCM3_LAST_CHANGED='2008-03-16' && /usr/local/cm3/bin/cm3 -ship -DROOT='/home/daniel/code/cm3' -DCM3_VERSION_TEXT='d5.7.0' -DCM3_VERSION_NUMBER='050700' -DCM3_LAST_CHANGED='2008-03-16' +++--- building in LINUXLIBC6 ---ignoring ../src/m3overrides--- shipping from LINUXLIBC6 ---. => /usr/local/cm3/pkg/m3bundle/LINUXLIBC6 .M3EXPORTS .M3WEB../src => /usr/local/cm3/pkg/m3bundle/src m3bundle.m3. => /usr/local/cm3/bin m3bundle. => /usr/local/cm3/man/man1 m3bundle.1 ==> /home/daniel/code/cm3/m3-tools/m3bundle done/home/daniel/code/cm3/scripts/pkgmap.sh -c "/usr/local/cm3/bin/cm3 -build -DROOT='/home/daniel/code/cm3' -DCM3_VERSION_TEXT='d5.7.0' -DCM3_VERSION_NUMBER='050700' -DCM3_LAST_CHANGED='2008-03-16' && /usr/local/cm3/bin/cm3 -ship -DROOT='/home/daniel/code/cm3' -DCM3_VERSION_TEXT='d5.7.0' -DCM3_VERSION_NUMBER='050700' -DCM3_LAST_CHANGED='2008-03-16' " import-libs m3core libm3 patternmatching sysutils unittest m3middle m3quake m3scanner m3tools m3cgcat m3cggen m3gdb m3bundle mklib fix_nl libdump arithmetic unittest-numeric bitvector digraph parseparams realgeometry set slisp sortedtableextras table-list tempfiles tcl tcp udp libsio libbuf debug listfuncs embutils m3tk-misc http binIO commandrw tapi serial m3tk mtex m3totex m3tohtml m3scan m3markup m3browser cmpdir cmpfp dirfp uniq netobj netobjd stubgen events rdwr sharedobj sharedobjgen odbc postgres95 db smalldb stablegen stable X11R4 ui PEX vbtkit cmvbt jvideo videovbt web formsvbtpixmaps formsvbt formsview formsedit codeview mg mgkit opengl anim3D zeus m3zume synloc synex metasyn obliqrt obliqparse obliqprint obliq obliqlibemb obliqlibm3 obliqlibui obliqlibanim obliqsrvstd obliqsrvui obliqbinmin obliqbinstd obliqbinui obliqbinanim visualobliq vocgi voquery vorun webvbt recordheap rehearsecode replayheap showheap shownew showthread pkl-fonts juno-machine juno-compiler juno-app cube calculator fisheye mentorpackage sysutils not found Enviado desde Correo Yahoo!Disfruta de una bandeja de entrada m?s inteligente.. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Thu Mar 20 00:27:47 2008 From: jayk123 at hotmail.com (Jay) Date: Wed, 19 Mar 2008 23:27:47 +0000 Subject: [M3devel] do-cm3-std.sh buildship -> package sysutils not found In-Reply-To: References: <62256.30048.qm@web27107.mail.ukl.yahoo.com> Message-ID: typos tpyos.. 1) Be sure to use -dAP in the cvs upd command. cd /home/daniel/code/cm3 cvs -z3 upd -dAP2) I'll likely do this soon unless there is a counterpoint or someone beats me to it. It's about three lines of sh/cmd/python (I can get all three, though I'm tempted to delete cmds instead; also, all three can now share one PKGSDB file, they all use relative paths from $CVSROOT and all always use forward slashes, so maybe I'll fix that too). - Jay From: jayk123 at hotmail.comTo: dabenavidesd at yahoo.es; m3devel at elegosoft.comDate: Wed, 19 Mar 2008 23:23:45 +0000Subject: Re: [M3devel] do-cm3-std.sh buildship -> package sysutils not found 1) be sure us -dAP in the cvs upd command. cd /home/daniel/code/cm3 cvs -z3 -dAP Otherwise you miss out on new stuff. Or at least not delete deleted stuff. ? cvs --help to double check this. 2) delete pkgsdb rm /home/daniel/code/cm3/scripts/PKGSDB I wonder if maybe the file should have a version suffix, and when a rebuild is needed, checkin an update to bump up the version suffix, and before any rebuild, rm PKGSDB*? That should fix this problem reasonably well, simply seamless, right? Just that occasionally some folks would have to remember to update the version. The existing version numbers could do. Deleting PKGSDB when there is any doubt is always safe. It will be recreated if it doesn't exist.Recreation is slow though and worth optimizing almost all of the time. - Jay Date: Wed, 19 Mar 2008 22:35:06 +0100From: dabenavidesd at yahoo.esTo: m3devel at elegosoft.comSubject: [M3devel] do-cm3-std.sh buildship -> package sysutils not foundHi all:I'm getting that error, when launching the script with the latests sources.Can you see what is missing, I think the sources tree is complete.Thanksadminfbd at fbd-desktop:/home/daniel/code/cm3/scripts$ sudo ./do-cm3-std.sh buildship/home/daniel/code/cm3/scripts/pkgmap.sh -c "/usr/local/cm3/bin/cm3 -build -DROOT='/home/daniel/code/cm3' -DCM3_VERSION_TEXT='d5.7.0' -DCM3_VERSION_NUMBER='050700' -DCM3_LAST_CHANGED='2008-03-16' && /usr/local/cm3/bin/cm3 -ship -DROOT='/home/daniel/code/cm3' -DCM3_VERSION_TEXT='d5.7.0' -DCM3_VERSION_NUMBER='050700' -DCM3_LAST_CHANGED='2008-03-16' " m3bundle=== package /home/daniel/code/cm3/m3-tools/m3bundle === +++ /usr/local/cm3/bin/cm3 -build -DROOT='/home/daniel/code/cm3' -DCM3_VERSION_TEXT='d5.7.0' -DCM3_VERSION_NUMBER='050700' -DCM3_LAST_CHANGED='2008-03-16' && /usr/local/cm3/bin/cm3 -ship -DROOT='/home/daniel/code/cm3' -DCM3_VERSION_TEXT='d5.7.0' -DCM3_VERSION_NUMBER='050700' -DCM3_LAST_CHANGED='2008-03-16' +++--- building in LINUXLIBC6 ---ignoring ../src/m3overrides--- shipping from LINUXLIBC6 ---. => /usr/local/cm3/pkg/m3bundle/LINUXLIBC6 .M3EXPORTS .M3WEB../src => /usr/local/cm3/pkg/m3bundle/src m3bundle.m3. => /usr/local/cm3/bin m3bundle. => /usr/local/cm3/man/man1 m3bundle.1 ==> /home/daniel/code/cm3/m3-tools/m3bundle done/home/daniel/code/cm3/scripts/pkgmap.sh -c "/usr/local/cm3/bin/cm3 -build -DROOT='/home/daniel/code/cm3' -DCM3_VERSION_TEXT='d5.7.0' -DCM3_VERSION_NUMBER='050700' -DCM3_LAST_CHANGED='2008-03-16' && /usr/local/cm3/bin/cm3 -ship -DROOT='/home/daniel/code/cm3' -DCM3_VERSION_TEXT='d5.7.0' -DCM3_VERSION_NUMBER='050700' -DCM3_LAST_CHANGED='2008-03-16' " import-libs m3core libm3 patternmatching sysutils unittest m3middle m3quake m3scanner m3tools m3cgcat m3cggen m3gdb m3bundle mklib fix_nl libdump arithmetic unittest-numeric bitvector digraph parseparams realgeometry set slisp sortedtableextras table-list tempfiles tcl tcp udp libsio libbuf debug listfuncs embutils m3tk-misc http binIO commandrw tapi serial m3tk mtex m3totex m3tohtml m3scan m3markup m3browser cmpdir cmpfp dirfp uniq netobj netobjd stubgen events rdwr sharedobj sharedobjgen odbc postgres95 db smalldb stablegen stable X11R4 ui PEX vbtkit cmvbt jvideo videovbt web formsvbtpixmaps formsvbt formsview formsedit codeview mg mgkit opengl anim3D zeus m3zume synloc synex metasyn obliqrt obliqparse obliqprint obliq obliqlibemb obliqlibm3 obliqlibui obliqlibanim obliqsrvstd obliqsrvui obliqbinmin obliqbinstd obliqbinui obliqbinanim visualobliq vocgi voquery vorun webvbt recordheap rehearsecode replayheap showheap shownew showthread pkl-fonts juno-machine juno-compiler juno-app cube calculator fisheye mentorpackage sysutils not found Enviado desde Correo Yahoo!Disfruta de una bandeja de entrada m?s inteligente.. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Thu Mar 20 07:50:39 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Thu, 20 Mar 2008 07:50:39 +0100 Subject: [M3devel] do-cm3-std.sh buildship -> package sysutils not found In-Reply-To: References: <62256.30048.qm@web27107.mail.ukl.yahoo.com> Message-ID: <20080320075039.e5qm88guocg8okss@mail.elegosoft.com> Quoting Jay : > counterpoint or someone beats me to it. It's about three lines of > sh/cmd/python (I can get all three, though I'm tempted to delete > cmds instead; also, all three can now share one PKGSDB file, they > all use relative paths from $CVSROOT and all always use forward > slashes, so maybe I'll fix that too). The file used by the shell scripts is cm3/scripts/PKGS, unless you have set the PKGSDB environment variable. It can be removed anytime, but needs some time to rebuild. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From wagner at elegosoft.com Thu Mar 20 09:09:48 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Thu, 20 Mar 2008 09:09:48 +0100 Subject: [M3devel] possible cygwin createprocess/fork cost measurements.. In-Reply-To: References: Message-ID: <20080320090948.sbqj156m800s080o@mail.elegosoft.com> Quoting Jay : > spawn(nowait) + waitpid is about the same as spawn(wait), and is > close to system() in inflexibility, but system() has an implicit > shell wrapper and so is slow. To eliminate the shell wrapper you could use System.ExecuteList from package sysutils in M3 and q_exec( cmd ) --> int (since CM3 d5.6.0) in quake (see quake manual on WWW), or does any Process.Create on Windows invoke a shell on NT386GNU? I wouldn't think so (but haven't checked). > It seems that spawn is faster than CreateProcess at running .exes > that use cygwin1.dll. That is a little odd. It must be something to > do with the child startup code talking to its parent or something > and doing so more efficiently when the immediate parent uses Cygwin. > I might dig into that. Like, a dynamic choice between CreateProcess > and spawn could be more efficient, but so far I'm not sure even > using CreateProcess here is "easy enough". > > Is there anywhere in the cm3 tree appropriate to checkin such > experimental code? > I can put it in my private svn... Yes, just create a branch: cvs tag -b devel_process_perf_opt m3core ... cvs up -r devel_process_perf_opt m3core ... and you are then working on the branch, until you issue cvs up -A m3core ... Hope this helps, Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jayk123 at hotmail.com Thu Mar 20 10:25:11 2008 From: jayk123 at hotmail.com (Jay) Date: Thu, 20 Mar 2008 09:25:11 +0000 Subject: [M3devel] possible cygwin createprocess/fork cost measurements.. In-Reply-To: <20080320090948.sbqj156m800s080o@mail.elegosoft.com> References: <20080320090948.sbqj156m800s080o@mail.elegosoft.com> Message-ID: Olaf, No, not every process create, creates a shell. AND ESP. NOT a sh/bash ONE! :) Merely replacing sh with cmd nets some perf win. The remaining perf cost is apparently the vfork/exec vs. spawn. I'm working on it.. (actually, not at the moment, but whenever I am home and not at work. :) ) My numbers don't all make sense to me so far or at the moment -- oh, maybe one set of perf tests launches more copies of sh than a real build. Anyway, I'm also measuring real builds. I was just really trying to do the "cheaper" up front prototyping/measuring before more real implementing/measuring, though of course I still get caught in refining the throw away work. :) The branch advise helps, but as well, I mean "directory structure"-wise, experimental, possibly throw away work, possibly interesting to keep around for future measuring, but not in any default build. Perhaps that is what you meant -- perhaps a private branch is ok for "personal history" even if it ends up being deleted and never going to another branch? Kind of an abuse, but useful, and maybe affordable? You know, rather than me saving away backups every hour in case I want to go back to an older version of what will still actually be thrown away? Or I could just use my private SVN. Yep. I did that. I'm wondering about making it world readable or somehow sharing it out. It's not chock full of stuff, but, of course, I get to decide what I am allowed to checkin. :) The general dilemna I think I see is that people that "do this sort of stuff" initially develop their command lines in a fancy interactive shell such as bash or cmd. With such features as pipes, redirection, environment variables, or even, gasp, wildcard expansion. They then move their command lines into a somewhat programmatic environment and expect all those features to keep working. And thus, various systems that "run command lines" end up wrapping them in shells to retain those fancy features. It's a bit unfortunate. Though in "native" environments the cost isn't so high. Wrapping in cmd isn't nearly as expensive in native Windows as wrapping in sh on Cygwin, nor I expect is wrapping in sh anywhere else. Though you all should read the Ubuntu Wiki page about changing /bin/sh from bash to ash. I don't think they have numbers, but they did this strictly for perf. While you may need fancy piping, redirection, etc. from /bin/sh, you don't for example need fancy command line editing/history/completion. To a large extent, uses of exec in Quake can't/shouldn't depend on much, since there is some hypothetical portability to cmd and sh, however the Quake code tends to be in target-specific files, which I think ends up being host/native-build specific, as well cmd and sh have an actually useful compatible subset -- pipes, redirection, combining commands with && and ||, so people can rely on having a shell, either one. Anyway, I think I had a plan in mind at least enough for now. There shall be three implementations of Process. Posix, Cygwin, and Win32. Only Cygwin changes from current. The Cygwin one will compare workingdirectory and stdin/out/err to the globals, and if they are equal, use spawn instead of fork/exec. Process only adds sh if it gets back a noexec error. The other part, the sh wrapper that Quake always adds, well, for now I set QUAKE_SHELL or whatever in my environment to cmd. That gets me most of the way there and then measurable. Not sure how to give that benefit to everyone always, probably with the idea of implementing m3_backend and assemble in Modula-3, and then defering to the Quake code if it is exists, and then making sure the Modula-3 version is very viable at least or only for Cygwin so it'll tend not to be replaced. Really, these functions are just a few lines. Another idea is to add a special prefix character to exec like an explanation point (for "fast" or "direct"). I don't like this use of special characters as flags though. It's the wrong way to design a function interface. Everything should be named constants or named boolean parameters. Every special character taken is a character that can't be used as the first character of a file name, as well. The heuristic to check the command line for "special characters" and deem them worthy of a sh wrapper isn't terrible either. This could be limited to Cygwin as well, like there could be Process.IsShSlow() (yeah yeah, bad choice of names, it doesn't belong in a public place Process, it would be Quake) that is false on all other platforms and gates the heuristic. A batch mode in cm3cg where it runs over an entire directory is also a good idea, along with a switch to it to have it exec as directly. That would also work as well. It MIGHT speed up other systems too, but I'd have to measure it, and I doubt it'd be much. I suspect the process create cost is ONLY signficant on Cygwin. It really bugs me 1) that there aren't more .dlls/.libs for use in-proc instead of shelling out to command lines 2) the related licensing unclarity. cm3cg and as really should not cost a process, the compiler should output .objs directly. On Windows, Microsoft nmake has a "double colon batch feature", where instead of saying: .c.o: cl -c $< You can say: .c.o:: cl -c $< and what this does is instead of running cl -c foo.c cl -c bar.c it runs: cl -c foo.c bar.c It can REALLY speed up builds. Modula-3 already has this behavior for the front end, just not the backend. I have no numbers. - Jay > Date: Thu, 20 Mar 2008 09:09:48 +0100> From: wagner at elegosoft.com> To: m3devel at elegosoft.com> Subject: Re: [M3devel] possible cygwin createprocess/fork cost measurements..> > Quoting Jay :> > > spawn(nowait) + waitpid is about the same as spawn(wait), and is > > close to system() in inflexibility, but system() has an implicit > > shell wrapper and so is slow.> > To eliminate the shell wrapper you could use System.ExecuteList> from package sysutils in M3 and> > q_exec( cmd ) --> int (since CM3 d5.6.0)> > in quake (see quake manual on WWW), or does any Process.Create> on Windows invoke a shell on NT386GNU? I wouldn't think so> (but haven't checked).> > > It seems that spawn is faster than CreateProcess at running .exes > > that use cygwin1.dll. That is a little odd. It must be something to > > do with the child startup code talking to its parent or something > > and doing so more efficiently when the immediate parent uses Cygwin. > > I might dig into that. Like, a dynamic choice between CreateProcess > > and spawn could be more efficient, but so far I'm not sure even > > using CreateProcess here is "easy enough".> >> > Is there anywhere in the cm3 tree appropriate to checkin such > > experimental code?> > I can put it in my private svn...> > Yes, just create a branch:> > cvs tag -b devel_process_perf_opt m3core ...> cvs up -r devel_process_perf_opt m3core ...> > and you are then working on the branch, until you issue> > cvs up -A m3core ...> > Hope this helps,> > Olaf> -- > Olaf Wagner -- elego Software Solutions GmbH> Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany> phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95> http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin> Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Thu Mar 20 10:26:08 2008 From: jayk123 at hotmail.com (Jay) Date: Thu, 20 Mar 2008 09:26:08 +0000 Subject: [M3devel] possible cygwin createprocess/fork cost measurements.. In-Reply-To: <20080320090948.sbqj156m800s080o@mail.elegosoft.com> References: <20080320090948.sbqj156m800s080o@mail.elegosoft.com> Message-ID: oops, yeah, q_exec sounds great. I'll look into it, thanks. Along with sometimes using spawn instead of fork/exec. - Jay From: jayk123 at hotmail.comTo: wagner at elegosoft.com; m3devel at elegosoft.comSubject: RE: [M3devel] possible cygwin createprocess/fork cost measurements..Date: Thu, 20 Mar 2008 09:25:11 +0000 Olaf,No, not every process create, creates a shell. AND ESP. NOT a sh/bash ONE! :)Merely replacing sh with cmd nets some perf win.The remaining perf cost is apparently the vfork/exec vs. spawn.I'm working on it.. (actually, not at the moment, but whenever I am home and not at work. :) ) My numbers don't all make sense to me so far or at the moment -- oh, maybe one set of perf tests launches more copies of sh than a real build. Anyway, I'm also measuring real builds. I was just really trying to do the "cheaper" up front prototyping/measuring before more real implementing/measuring, though of course I still get caught in refining the throw away work. :) The branch advise helps, but as well, I mean "directory structure"-wise, experimental, possibly throw away work, possibly interesting to keep around for future measuring, but not in any default build. Perhaps that is what you meant -- perhaps a private branch is ok for "personal history" even if it ends up being deleted and never going to another branch? Kind of an abuse, but useful, and maybe affordable? You know, rather than me saving away backups every hour in case I want to go back to an older version of what will still actually be thrown away? Or I could just use my private SVN. Yep. I did that. I'm wondering about making it world readable or somehow sharing it out. It's not chock full of stuff, but, of course, I get to decide what I am allowed to checkin. :) The general dilemna I think I see is that people that "do this sort of stuff" initially develop their command lines in a fancy interactive shell such as bash or cmd. With such features as pipes, redirection, environment variables, or even, gasp, wildcard expansion.They then move their command lines into a somewhat programmatic environment and expect all those features to keep working.And thus, various systems that "run command lines" end up wrapping them in shells to retain those fancy features.It's a bit unfortunate. Though in "native" environments the cost isn't so high. Wrapping in cmd isn't nearly as expensive in native Windows as wrapping in sh on Cygwin, nor I expect is wrapping in sh anywhere else. Though you all should read the Ubuntu Wiki page about changing /bin/sh from bash to ash. I don't think they have numbers, but they did this strictly for perf. While you may need fancy piping, redirection, etc. from /bin/sh, you don't for example need fancy command line editing/history/completion. To a large extent, uses of exec in Quake can't/shouldn't depend on much, since there is some hypothetical portability to cmd and sh, however the Quake code tends to be in target-specific files, which I think ends up being host/native-build specific, as well cmd and sh have an actually useful compatible subset -- pipes, redirection, combining commands with && and ||, so people can rely on having a shell, either one. Anyway, I think I had a plan in mind at least enough for now.There shall be three implementations of Process. Posix, Cygwin, and Win32. Only Cygwin changes from current.The Cygwin one will compare workingdirectory and stdin/out/err to the globals, and if they are equal, use spawn instead of fork/exec. Process only adds sh if it gets back a noexec error. The other part, the sh wrapper that Quake always adds, well, for now I set QUAKE_SHELL or whatever in my environment to cmd. That gets me most of the way there and then measurable. Not sure how to give that benefit to everyone always, probably with the idea of implementing m3_backend and assemble in Modula-3, and then defering to the Quake code if it is exists, and then making sure the Modula-3 version is very viable at least or only for Cygwin so it'll tend not to be replaced.Really, these functions are just a few lines. Another idea is to add a special prefix character to exec like an explanation point (for "fast" or "direct").I don't like this use of special characters as flags though. It's the wrong way to design a function interface.Everything should be named constants or named boolean parameters.Every special character taken is a character that can't be used as the first character of a file name, as well. The heuristic to check the command line for "special characters" and deem them worthy of a sh wrapper isn't terrible either.This could be limited to Cygwin as well, like there could be Process.IsShSlow() (yeah yeah, bad choice of names, it doesn't belong in a public place Process, it would be Quake) that is false on all other platforms and gates the heuristic. A batch mode in cm3cg where it runs over an entire directory is also a good idea, along with a switch to it to have it exec as directly. That would also work as well. It MIGHT speed up other systems too, but I'd have to measure it, and I doubt it'd be much. I suspect the process create cost is ONLY signficant on Cygwin. It really bugs me 1) that there aren't more .dlls/.libs for use in-proc instead of shelling out to command lines 2) the related licensing unclarity. cm3cg and as really should not cost a process, the compiler should output .objs directly. On Windows, Microsoft nmake has a "double colon batch feature", where instead of saying: .c.o: cl -c $< You can say:.c.o:: cl -c $< and what this does is instead of running cl -c foo.c cl -c bar.c it runs: cl -c foo.c bar.c It can REALLY speed up builds.Modula-3 already has this behavior for the front end, just not the backend. I have no numbers. - Jay > Date: Thu, 20 Mar 2008 09:09:48 +0100> From: wagner at elegosoft.com> To: m3devel at elegosoft.com> Subject: Re: [M3devel] possible cygwin createprocess/fork cost measurements..> > Quoting Jay :> > > spawn(nowait) + waitpid is about the same as spawn(wait), and is > > close to system() in inflexibility, but system() has an implicit > > shell wrapper and so is slow.> > To eliminate the shell wrapper you could use System.ExecuteList> from package sysutils in M3 and> > q_exec( cmd ) --> int (since CM3 d5.6.0)> > in quake (see quake manual on WWW), or does any Process.Create> on Windows invoke a shell on NT386GNU? I wouldn't think so> (but haven't checked).> > > It seems that spawn is faster than CreateProcess at running .exes > > that use cygwin1.dll. That is a little odd. It must be something to > > do with the child startup code talking to its parent or something > > and doing so more efficiently when the immediate parent uses Cygwin. > > I might dig into that. Like, a dynamic choice between CreateProcess > > and spawn could be more efficient, but so far I'm not sure even > > using CreateProcess here is "easy enough".> >> > Is there anywhere in the cm3 tree appropriate to checkin such > > experimental code?> > I can put it in my private svn...> > Yes, just create a branch:> > cvs tag -b devel_process_perf_opt m3core ...> cvs up -r devel_process_perf_opt m3core ...> > and you are then working on the branch, until you issue> > cvs up -A m3core ...> > Hope this helps,> > Olaf> -- > Olaf Wagner -- elego Software Solutions GmbH> Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany> phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95> http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin> Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Thu Mar 20 10:29:24 2008 From: jayk123 at hotmail.com (Jay) Date: Thu, 20 Mar 2008 09:29:24 +0000 Subject: [M3devel] FW: possible cygwin createprocess/fork cost measurements.. In-Reply-To: <20080320090948.sbqj156m800s080o@mail.elegosoft.com> References: <20080320090948.sbqj156m800s080o@mail.elegosoft.com> Message-ID: I got truncated again darnit. - Jay From: jayk123 at hotmail.comTo: wagner at elegosoft.com; m3devel at elegosoft.comSubject: RE: [M3devel] possible cygwin createprocess/fork cost measurements..Date: Thu, 20 Mar 2008 09:26:08 +0000 oops, yeah, q_exec sounds great. I'll look into it, thanks.Along with sometimes using spawn instead of fork/exec. - Jay From: jayk123 at hotmail.comTo: wagner at elegosoft.com; m3devel at elegosoft.comSubject: RE: [M3devel] possible cygwin createprocess/fork cost measurements..Date: Thu, 20 Mar 2008 09:25:11 +0000 Olaf,No, not every process create, creates a shell. AND ESP. NOT a sh/bash ONE! :)Merely replacing sh with cmd nets some perf win.The remaining perf cost is apparently the vfork/exec vs. spawn.I'm working on it.. (actually, not at the moment, but whenever I am home and not at work. :) ) My numbers don't all make sense to me so far or at the moment -- oh, maybe one set of perf tests launches more copies of sh than a real build. Anyway, I'm also measuring real builds. I was just really trying to do the "cheaper" up front prototyping/measuring before more real implementing/measuring, though of course I still get caught in refining the throw away work. :) The branch advise helps, but as well, I mean "directory structure"-wise, experimental, possibly throw away work, possibly interesting to keep around for future measuring, but not in any default build. Perhaps that is what you meant -- perhaps a private branch is ok for "personal history" even if it ends up being deleted and never going to another branch? Kind of an abuse, but useful, and maybe affordable? You know, rather than me saving away backups every hour in case I want to go back to an older version of what will still actually be thrown away? Or I could just use my private SVN. Yep. I did that. I'm wondering about making it world readable or somehow sharing it out. It's not chock full of stuff, but, of course, I get to decide what I am allowed to checkin. :) The general dilemna I think I see is that people that "do this sort of stuff" initially develop their command lines in a fancy interactive shell such as bash or cmd. With such features as pipes, redirection, environment variables, or even, gasp, wildcard expansion.They then move their command lines into a somewhat programmatic environment and expect all those features to keep working.And thus, various systems that "run command lines" end up wrapping them in shells to retain those fancy features.It's a bit unfortunate. Though in "native" environments the cost isn't so high. Wrapping in cmd isn't nearly as expensive in native Windows as wrapping in sh on Cygwin, nor I expect is wrapping in sh anywhere else. Though you all should read the Ubuntu Wiki page about changing /bin/sh from bash to ash. I don't think they have numbers, but they did this strictly for perf. While you may need fancy piping, redirection, etc. from /bin/sh, you don't for example need fancy command line editing/history/completion. To a large extent, uses of exec in Quake can't/shouldn't depend on much, since there is some hypothetical portability to cmd and sh, however the Quake code tends to be in target-specific files, which I think ends up being host/native-build specific, as well cmd and sh have an actually useful compatible subset -- pipes, redirection, combining commands with && and ||, so people can rely on having a shell, either one. Anyway, I think I had a plan in mind at least enough for now.There shall be three implementations of Process. Posix, Cygwin, and Win32. Only Cygwin changes from current.The Cygwin one will compare workingdirectory and stdin/out/err to the globals, and if they are equal, use spawn instead of fork/exec. Process only adds sh if it gets back a noexec error. The other part, the sh wrapper that Quake always adds, well, for now I set QUAKE_SHELL or whatever in my environment to cmd. That gets me most of the way there and then measurable. Not sure how to give that benefit to everyone always, probably with the idea of implementing m3_backend and assemble in Modula-3, and then defering to the Quake code if it is exists, and then making sure the Modula-3 version is very viable at least or only for Cygwin so it'll tend not to be replaced.Really, these functions are just a few lines. Another idea is to add a special prefix character to exec like an explanation point (for "fast" or "direct").I don't like this use of special characters as flags though. It's the wrong way to design a function interface.Everything should be named constants or named boolean parameters.Every special character taken is a character that can't be used as the first character of a file name, as well. The heuristic to check the command line for "special characters" and deem them worthy of a sh wrapper isn't terrible either.This could be limited to Cygwin as well, like there could be Process.IsShSlow() (yeah yeah, bad choice of names, it doesn't belong in a public place Process, it would be Quake) that is false on all other platforms and gates the heuristic. A batch mode in cm3cg where it runs over an entire directory is also a good idea, along with a switch to it to have it exec as directly. That would also work as well. It MIGHT speed up other systems too, but I'd have to measure it, and I doubt it'd be much. I suspect the process create cost is ONLY signficant on Cygwin. It really bugs me 1) that there aren't more .dlls/.libs for use in-proc instead of shelling out to command lines 2) the related licensing unclarity. cm3cg and as really should not cost a process, the compiler should output .objs directly. On Windows, Microsoft nmake has a "double colon batch feature", where instead of saying: .c.o: cl -c $< You can say:.c.o:: cl -c $< and what this does is instead of running cl -c foo.c cl -c bar.c it runs: cl -c foo.c bar.c It can REALLY speed up builds.Modula-3 already has this behavior for the front end, just not the backend. I have no numbers. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hendrik at topoi.pooq.com Thu Mar 20 17:41:12 2008 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Thu, 20 Mar 2008 12:41:12 -0400 Subject: [M3devel] random-acces I/O In-Reply-To: <47E1425C.1E75.00D7.1@scires.com> References: <200803012249.m21MnOnF018386@camembert.async.caltech.edu> <1204416764.10030.12.camel@hias.m3w.org> <20080319133852.GA7741@topoi.pooq.com> <47E1425C.1E75.00D7.1@scires.com> Message-ID: <20080320164112.GA9306@topoi.pooq.com> On Wed, Mar 19, 2008 at 04:42:05PM -0400, Randy Coleburn wrote: > Hendrik: > > I did a quick check of what I have available in this area. > > I've attached the .i3 files for four modules. (These probably have > Windows-style line endings, so you may have to convert for Unix.) > > Let me know if any of these are what you are looking for. I can > probably arrange to get you the source for the modules and/or put them > in the elegosoft repository. > > If you can be more specific about exactly what you are looking for, I > can check further to see if I have something else. No, these are not what I'm looking for. From the original post for this thread: I need to do direct-access I/O from a Modula 3 program with a file format that is externally specified. It is necessary to both read and write in random fashion during a run of the program, prefereably without having to close or open the file each time. Now Linux will do this, so what I actually *need* is just the "C" interfaces to the standard C library. The hard part is to integrate it with the readers and writers -- probably hard only if I'm actually going to figure out where it fits in the inheritance hierarchy of in/out things. What surprises me is that the Modula 3 distribution doesn't seem to have this already. Just to get on with my project, I can probably make do with writing "C" interfaces to the raw Linux I'O system. By the way, smarteiffel seems to be just as deficient. It's as if language designers never encounter database implementers. -- hendrik > > Regards, > Randy > From dragisha at m3w.org Fri Mar 21 09:28:27 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Fri, 21 Mar 2008 09:28:27 +0100 Subject: [M3devel] m3gdb crashing Message-ID: <1206088107.16885.15.camel@hias.m3w.org> Program received signal SIGINT, Interrupt. [Switching to Thread -1208531264 (LWP 20589)] 0x00110402 in __kernel_vsyscall () (m3gdb) bt #0 0x00110402 in __kernel_vsyscall () #1 0x43341a3e in __lll_mutex_lock_wait () from /lib/libpthread.so.0 #2 0x4333d890 in _L_mutex_lock_85 () from /lib/libpthread.so.0 ../../gdb/gdb/dwarf2-frame.c:490: internal-error: Unknown CFI encountered. A problem internal to GDB has been detected, further debugging may prove unreliable. -- Dragi?a Duri? From wagner at elegosoft.com Fri Mar 21 12:19:53 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Fri, 21 Mar 2008 12:19:53 +0100 Subject: [M3devel] possible cygwin createprocess/fork cost measurements.. In-Reply-To: References: <20080320090948.sbqj156m800s080o@mail.elegosoft.com> Message-ID: <20080321121953.39e767bl9ccgw0co@mail.elegosoft.com> Quoting Jay : > Olaf, > The branch advise helps, but as well, I mean "directory > structure"-wise, experimental, possibly throw away work, possibly > interesting to keep around for future measuring, but not in any > default build. Perhaps that is what you meant -- perhaps a private > branch is ok for "personal history" even if it ends up being deleted > and never going to another branch? Kind of an abuse, but useful, > and maybe affordable? You know, rather than me saving away backups > every hour in case I want to go back to an older version of what > will still actually be thrown away? That's a very common use for branches and exactly what is recommened in the CM3 SCM guidelines, IIRC. Currently in CM3 branches are used for three purposes: 1. vendor branches for imports of third party software (like gcc) 2. feature branches for experimental new development (which may or may not be merged into any release or the main line) 3. release engineering branches for the (very infrequent) official releases (last used for 5.4.0, IIRC) As almost all work is currently done on the main line, I'd like to encourage the use of branches for more experimental work, so as not to disturb maintenance work. I even suggested to restrict commits to the main line and allow only branch commits for development as a counter measure to loosing control of the various work, but am not convinced myself that it is worth the effort, and others didn't seem to like it very much, too. I don't think anybody would object if you commit more experimental or radical changes to a branch. > Or I could just use my private SVN. Yep. I did that. I'm wondering > about making it world readable or somehow sharing it out. It's not > chock full of stuff, but, of course, I get to decide what I am > allowed to checkin. :) That's of course another option; but I'd rather like to keep the resources centralized. What really is needed is more integration and ease of use, and not more diversion. > It really bugs me 1) that there aren't more .dlls/.libs for use > in-proc instead of shelling out to command lines 2) the related > licensing unclarity. cm3cg and as really should not cost a process, > the compiler should output .objs directly. Have you had a look at System.ExecuteList and q_exec? It really is a pure M3-based implementation of the most needed shell feataures like i/o redirection, command lists, and pipes. If you only use these features, they will run independently of the underlying target platform. If will of course not help with the issue of the separated compiler backend. Integrated backends can be developped independently of gcc though. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From wagner at elegosoft.com Fri Mar 21 12:53:31 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Fri, 21 Mar 2008 12:53:31 +0100 Subject: [M3devel] random-acces I/O In-Reply-To: <20080320164112.GA9306@topoi.pooq.com> References: <200803012249.m21MnOnF018386@camembert.async.caltech.edu> <1204416764.10030.12.camel@hias.m3w.org> <20080319133852.GA7741@topoi.pooq.com> <47E1425C.1E75.00D7.1@scires.com> <20080320164112.GA9306@topoi.pooq.com> Message-ID: <20080321125331.3o0fpz8ug4g4wks4@mail.elegosoft.com> Quoting hendrik at topoi.pooq.com: > On Wed, Mar 19, 2008 at 04:42:05PM -0400, Randy Coleburn wrote: >> Hendrik: > Now Linux will do this, so what I actually *need* is just the "C" > interfaces to the standard C library. The hard part is to integrate > it with the readers and writers -- probably hard only if I'm > actually going to figure out where it fits in the inheritance > hierarchy of in/out things. What surprises me is that the Modula 3 > distribution doesn't seem to have this already. > > Just to get on with my project, I can probably make do with writing "C" > interfaces to the raw Linux I'O system. > > By the way, smarteiffel seems to be just as deficient. It's as if > language designers never encounter database implementers. It's probably only needed for database implementation, and those are added as third-party-products ;-) There was a research paper on the design of the IO classes; I think it may be ftp://gatekeeper.research.compaq.com/pub/DEC/SRC/research-reports/abstracts/src-rr-053.html from the M3 bibliography. I don't remember if random I/O was a topic there though. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From hosking at cs.purdue.edu Fri Mar 21 14:42:30 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 21 Mar 2008 09:42:30 -0400 Subject: [M3devel] m3gdb crashing In-Reply-To: <1206088107.16885.15.camel@hias.m3w.org> References: <1206088107.16885.15.camel@hias.m3w.org> Message-ID: <088E4995-DA80-446B-A091-84B8233B1F84@cs.purdue.edu> Does it work with regular gdb? On Mar 21, 2008, at 4:28 AM, Dragi?a Duri? wrote: > Program received signal SIGINT, Interrupt. > [Switching to Thread -1208531264 (LWP 20589)] > 0x00110402 in __kernel_vsyscall () > (m3gdb) bt > #0 0x00110402 in __kernel_vsyscall () > #1 0x43341a3e in __lll_mutex_lock_wait () from /lib/libpthread.so.0 > #2 0x4333d890 in _L_mutex_lock_85 () from /lib/libpthread.so.0 > ../../gdb/gdb/dwarf2-frame.c:490: internal-error: Unknown CFI > encountered. > A problem internal to GDB has been detected, > further debugging may prove unreliable. > > -- > Dragi?a Duri? From dragisha at m3w.org Fri Mar 21 16:51:11 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Fri, 21 Mar 2008 16:51:11 +0100 Subject: [M3devel] m3gdb crashing In-Reply-To: <088E4995-DA80-446B-A091-84B8233B1F84@cs.purdue.edu> References: <1206088107.16885.15.camel@hias.m3w.org> <088E4995-DA80-446B-A091-84B8233B1F84@cs.purdue.edu> Message-ID: <1206114671.16885.25.camel@hias.m3w.org> (gdb) bt #0 0x00110402 in __kernel_vsyscall () #1 0x43341a3e in __lll_mutex_lock_wait () from /lib/libpthread.so.0 #2 0x4333d890 in _L_mutex_lock_85 () from /lib/libpthread.so.0 #3 0x4333d3dd in pthread_mutex_lock () from /lib/libpthread.so.0 #4 0x002f5003 in Thread__Acquire (M3_AYIbX3_m=0xb7f31960) at ../src/thread/PTHREAD/ThreadPThread.m3:143 #5 0x08049c61 in Connection__Append (M3_DlOx9a_b=0xb7f31960, M3_CKMnXU_pkt=0xbfd57034, M3_AicXUJ_signal=1 '\001') at ../src/Connection.m3:138 #6 0x0804c044 in PDU__Send (M3_DzQGng_b=0xb7526b58, M3_ALzNrn_c=0xb7f31900) at ../src/PDU.m3:124 #7 0x0804d802 in SMPP_M3 (M3_AcxOUs_mode=1) at ../src/SMPP.m3:200 #8 0x002e1032 in RTLinker__RunMainBody (M3_DjPxE3_m=0x804fa80) at ../src/runtime/common/RTLinker.m3:399 #9 0x002df92c in RTLinker__AddUnitI (M3_DjPxE3_m=0x804fa80) at ../src/runtime/common/RTLinker.m3:113 #10 0x002df9c3 in RTLinker__AddUnit (M3_DjPxE5_b=0x804d3df) at ../src/runtime/common/RTLinker.m3:122 #11 0x080493d8 in main (argc=1, argv=0xbfd57314, envp=0xbfd5731c) at _m3main.mc:4 This is output from regular gdb, it works. If it's important, this is piece of code... PROCEDURE Send(b: T; c: Connection.T) = BEGIN SetInt(b, 0, 4, b.used); LOCK c.output DO INC(c.seq); SetInt(b, 12, 4, c.seq); c.output.append(SUBARRAY(b.data^, 0, b.used)); END; END Send; My bug was... append method also tries to lock c.output... It is a moment when I Ctrl-C-ed it. dd On Fri, 2008-03-21 at 09:42 -0400, Tony Hosking wrote: > Does it work with regular gdb? > > On Mar 21, 2008, at 4:28 AM, Dragi?a Duri? wrote: > > > Program received signal SIGINT, Interrupt. > > [Switching to Thread -1208531264 (LWP 20589)] > > 0x00110402 in __kernel_vsyscall () > > (m3gdb) bt > > #0 0x00110402 in __kernel_vsyscall () > > #1 0x43341a3e in __lll_mutex_lock_wait () from /lib/libpthread.so.0 > > #2 0x4333d890 in _L_mutex_lock_85 () from /lib/libpthread.so.0 > > ../../gdb/gdb/dwarf2-frame.c:490: internal-error: Unknown CFI > > encountered. > > A problem internal to GDB has been detected, > > further debugging may prove unreliable. > > > > -- > > Dragi?a Duri? > -- Dragi?a Duri? From dragisha at m3w.org Fri Mar 21 20:54:03 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Fri, 21 Mar 2008 20:54:03 +0100 Subject: [M3devel] random-acces I/O In-Reply-To: <20080321125331.3o0fpz8ug4g4wks4@mail.elegosoft.com> References: <200803012249.m21MnOnF018386@camembert.async.caltech.edu> <1204416764.10030.12.camel@hias.m3w.org> <20080319133852.GA7741@topoi.pooq.com> <47E1425C.1E75.00D7.1@scires.com> <20080320164112.GA9306@topoi.pooq.com> <20080321125331.3o0fpz8ug4g4wks4@mail.elegosoft.com> Message-ID: <1206129243.16885.31.camel@hias.m3w.org> On Fri, 2008-03-21 at 12:53 +0100, Olaf Wagner wrote: ... > > By the way, smarteiffel seems to be just as deficient. It's as if > > language designers never encounter database implementers. > > It's probably only needed for database implementation, and those > are added as third-party-products ;-) Can, not must :) One of our development lines is OODBMS - m3/db2, working pretty well, and older version runs few apps. Current version is cca 10,000 lines and it's feature list is nice one :). Our plan is to finish it and publish it, some of these days, months, ... :) > > There was a research paper on the design of the IO classes; > I think it may be These are good standard high-level IO classess. Original poster has to look at File.T, there is all he needs. > > > ftp://gatekeeper.research.compaq.com/pub/DEC/SRC/research-reports/abstracts/src-rr-053.html > > from the M3 bibliography. I don't remember if random I/O was a > topic there though. > > Olaf -- Dragi?a Duri? From jayk123 at hotmail.com Sat Mar 22 07:51:03 2008 From: jayk123 at hotmail.com (Jay) Date: Sat, 22 Mar 2008 06:51:03 +0000 Subject: [M3devel] possible cygwin createprocess/fork cost measurements.. In-Reply-To: <20080321121953.39e767bl9ccgw0co@mail.elegosoft.com> References: <20080320090948.sbqj156m800s080o@mail.elegosoft.com> <20080321121953.39e767bl9ccgw0co@mail.elegosoft.com> Message-ID: > branches Sorry, but I have some "issues" around learning source control systems -- I do it very slowly.> the resources centralized. What really is needed is more integration> and ease of use, and not more diversion. On the one hand, I don't think you are missing much in not having the contents of my SVN. On the other hand, I can/should look into making it world readable. On the third hand (?), actually, there is some stuff that I think is useful and might benefit others, but it is also partly user/machine-specific and can only serve as good examples and not directly usable by anyone else. Specifically at least the various .bat files I have for configuring various environments for various toolsets. As well, since these are important to me, for "work" (ok, play), beyond Modula-3, it makes sense to not be in "the Modula-3 source control system". > Have you had a look at System.ExecuteList and q_exec? Sorry, not yet. Yes it sounds like it will fix the sh wrap issue. But not the fork/exec vs. spawn, which I will deal with. > compiler backend. Integrated backends can be developped independently> of gcc though. Yes, I think more of these will be good. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Sat Mar 22 14:17:48 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sat, 22 Mar 2008 14:17:48 +0100 Subject: [M3devel] blue sky proposal :) Message-ID: <1206191868.16885.55.camel@hias.m3w.org> Now that VM mechanism is fully obsoleted by Mr. Hoskings GC... Why would not we make something like this... <* READONLY object *> - to move object to VM protected pages of traced heap, so we can have RuntimeError.E when some code tries to write into object <* WRITEABLE object *> - to return it back to normal traced heap pages Pragmas... Can be ignored by non-compatible compiler, and totally are not changing language semantics. They just offer great debugging tool and help us define/enforce some static properties of our structures. This, of course, is something for GC people to do... As I see current situation (as deep as I went into it) it will not affect performance at all, and it will (IMHO) help straighten GC code even more :). dd -- Dragi?a Duri? From dabenavidesd at yahoo.es Thu Mar 20 03:03:26 2008 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Thu, 20 Mar 2008 03:03:26 +0100 (CET) Subject: [M3devel] do-cm3-std.sh buildship -> package sysutils not found In-Reply-To: Message-ID: <205051.28701.qm@web27110.mail.ukl.yahoo.com> Hi: Yes it's working when I deleted scripts/PKGS file but, then I also have to run do-cm3-std.sh realclean because I got a error by duplicate link info. I update in this machine the tree with cvsup, when using cvsup I just get into trouble when I make a change in any file of the sources tree but it doesn't update "again" to the official current version of the file from cvsup server. Thanks for your help. Jay escribi?: .hmmessage P { margin:0px; padding:0px } body.hmmessage { FONT-SIZE: 10pt; FONT-FAMILY:Tahoma } typos tpyos.. 1) Be sure to use -dAP in the cvs upd command. cd /home/daniel/code/cm3 cvs -z3 upd -dAP 2) I'll likely do this soon unless there is a counterpoint or someone beats me to it. It's about three lines of sh/cmd/python (I can get all three, though I'm tempted to delete cmds instead; also, all three can now share one PKGSDB file, they all use relative paths from $CVSROOT and all always use forward slashes, so maybe I'll fix that too). - Jay --------------------------------- From: jayk123 at hotmail.com To: dabenavidesd at yahoo.es; m3devel at elegosoft.com Date: Wed, 19 Mar 2008 23:23:45 +0000 Subject: Re: [M3devel] do-cm3-std.sh buildship -> package sysutils not found .ExternalClass .EC_hmmessage P {padding:0px;} .ExternalClass EC_body.hmmessage {font-size:10pt;font-family:Tahoma;} 1) be sure us -dAP in the cvs upd command. cd /home/daniel/code/cm3 cvs -z3 -dAP Otherwise you miss out on new stuff. Or at least not delete deleted stuff. ? cvs --help to double check this. 2) delete pkgsdb rm /home/daniel/code/cm3/scripts/PKGSDB I wonder if maybe the file should have a version suffix, and when a rebuild is needed, checkin an update to bump up the version suffix, and before any rebuild, rm PKGSDB*? That should fix this problem reasonably well, simply seamless, right? Just that occasionally some folks would have to remember to update the version. The existing version numbers could do. Deleting PKGSDB when there is any doubt is always safe. It will be recreated if it doesn't exist. Recreation is slow though and worth optimizing almost all of the time. - Jay --------------------------------- Date: Wed, 19 Mar 2008 22:35:06 +0100 From: dabenavidesd at yahoo.es To: m3devel at elegosoft.com Subject: [M3devel] do-cm3-std.sh buildship -> package sysutils not found Hi all: I'm getting that error, when launching the script with the latests sources. Can you see what is missing, I think the sources tree is complete. Thanks adminfbd at fbd-desktop:/home/daniel/code/cm3/scripts$ sudo ./do-cm3-std.sh buildship /home/daniel/code/cm3/scripts/pkgmap.sh -c "/usr/local/cm3/bin/cm3 -build -DROOT='/home/daniel/code/cm3' -DCM3_VERSION_TEXT='d5.7.0' -DCM3_VERSION_NUMBER='050700' -DCM3_LAST_CHANGED='2008-03-16' && /usr/local/cm3/bin/cm3 -ship -DROOT='/home/daniel/code/cm3' -DCM3_VERSION_TEXT='d5.7.0' -DCM3_VERSION_NUMBER='050700' -DCM3_LAST_CHANGED='2008-03-16' " m3bundle === package /home/daniel/code/cm3/m3-tools/m3bundle === +++ /usr/local/cm3/bin/cm3 -build -DROOT='/home/daniel/code/cm3' -DCM3_VERSION_TEXT='d5.7.0' -DCM3_VERSION_NUMBER='050700' -DCM3_LAST_CHANGED='2008-03-16' && /usr/local/cm3/bin/cm3 -ship -DROOT='/home/daniel/code/cm3' -DCM3_VERSION_TEXT='d5.7.0' -DCM3_VERSION_NUMBER='050700' -DCM3_LAST_CHANGED='2008-03-16' +++ --- building in LINUXLIBC6 --- ignoring ../src/m3overrides --- shipping from LINUXLIBC6 --- . => /usr/local/cm3/pkg/m3bundle/LINUXLIBC6 .M3EXPORTS .M3WEB ../src => /usr/local/cm3/pkg/m3bundle/src m3bundle.m3 . => /usr/local/cm3/bin m3bundle . => /usr/local/cm3/man/man1 m3bundle.1 ==> /home/daniel/code/cm3/m3-tools/m3bundle done /home/daniel/code/cm3/scripts/pkgmap.sh -c "/usr/local/cm3/bin/cm3 -build -DROOT='/home/daniel/code/cm3' -DCM3_VERSION_TEXT='d5.7.0' -DCM3_VERSION_NUMBER='050700' -DCM3_LAST_CHANGED='2008-03-16' && /usr/local/cm3/bin/cm3 -ship -DROOT='/home/daniel/code/cm3' -DCM3_VERSION_TEXT='d5.7.0' -DCM3_VERSION_NUMBER='050700' -DCM3_LAST_CHANGED='2008-03-16' " import-libs m3core libm3 patternmatching sysutils unittest m3middle m3quake m3scanner m3tools m3cgcat m3cggen m3gdb m3bundle mklib fix_nl libdump arithmetic unittest-numeric bitvector digraph parseparams realgeometry set slisp sortedtableextras table-list tempfiles tcl tcp udp libsio libbuf debug listfuncs embutils m3tk-misc http binIO commandrw tapi serial m3tk mtex m3totex m3tohtml m3scan m3markup m3browser cmpdir cmpfp dirfp uniq netobj netobjd stubgen events rdwr sharedobj sharedobjgen odbc postgres95 db smalldb stablegen stable X11R4 ui PEX vbtkit cmvbt jvideo videovbt web formsvbtpixmaps formsvbt formsview formsedit codeview mg mgkit opengl anim3D zeus m3zume synloc synex metasyn obliqrt obliqparse obliqprint obliq obliqlibemb obliqlibm3 obliqlibui obliqlibanim obliqsrvstd obliqsrvui obliqbinmin obliqbinstd obliqbinui obliqbinanim visualobliq vocgi voquery vorun webvbt recordheap rehearsecode replayheap showheap shownew showthread pkl-fonts juno-machine juno-compiler juno-app cube calculator fisheye mentor package sysutils not found --------------------------------- Enviado desde Correo Yahoo! Disfruta de una bandeja de entrada m?s inteligente.. --------------------------------- ?Con Mascota por primera vez? - S? un mejor Amigo Entra en Yahoo! Respuestas. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Sat Mar 22 17:26:41 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sat, 22 Mar 2008 12:26:41 -0400 Subject: [M3devel] blue sky proposal :) In-Reply-To: <1206191868.16885.55.camel@hias.m3w.org> References: <1206191868.16885.55.camel@hias.m3w.org> Message-ID: <712302B7-3312-464D-AB97-316E3F07DA10@cs.purdue.edu> Messing with VM protection mechanisms *will* impact performance, both to set things up and to maintain. For example, the GC needs to write to those pages as it operates, so you are going to put extra work on the GC. Yuck! Here is the germ of an alternative idea -- let add a READONLY declaration, with enforcement by the compiler. Should READONLY be a property of types? i.e., READONLY RECORD f: T; g: T; END (* records of this type can never be modified (i.e., f, g can only be initialized in a constructor) *) Or should READONLY be a property of designators? i.e., RECORD READONLY f: T; g: T END (* READONLY fields can never be modified (i.e., f cannot be modified, but g can be modified) *) This needs fleshing out. For example, if READONLY attaches to types, then what are the assignment rules for non-readonly and readonly? More complicated still, what are the subtype rules? Food for thought... On Mar 22, 2008, at 9:17 AM, Dragi?a Duri? wrote: > Now that VM mechanism is fully obsoleted by Mr. Hoskings GC... Why > would > not we make something like this... > > <* READONLY object *> - to move object to VM protected pages of > traced > heap, so we can have RuntimeError.E when some code tries to write into > object > > <* WRITEABLE object *> - to return it back to normal traced heap pages > > Pragmas... Can be ignored by non-compatible compiler, and totally are > not changing language semantics. They just offer great debugging tool > and help us define/enforce some static properties of our structures. > > This, of course, is something for GC people to do... As I see current > situation (as deep as I went into it) it will not affect performance > at > all, and it will (IMHO) help straighten GC code even more :). > > dd > -- > Dragi?a Duri? From lemming at henning-thielemann.de Sat Mar 22 23:12:17 2008 From: lemming at henning-thielemann.de (Henning Thielemann) Date: Sat, 22 Mar 2008 23:12:17 +0100 (CET) Subject: [M3devel] blue sky proposal :) In-Reply-To: <712302B7-3312-464D-AB97-316E3F07DA10@cs.purdue.edu> References: <1206191868.16885.55.camel@hias.m3w.org> <712302B7-3312-464D-AB97-316E3F07DA10@cs.purdue.edu> Message-ID: On Sat, 22 Mar 2008, Tony Hosking wrote: > Messing with VM protection mechanisms *will* impact performance, both to set > things up and to maintain. For example, the GC needs to write to those pages > as it operates, so you are going to put extra work on the GC. Yuck! > > Here is the germ of an alternative idea -- let add a READONLY declaration, > with enforcement by the compiler. Should READONLY be a property of types? > i.e., > > READONLY RECORD f: T; g: T; END (* records of this type can never be > modified (i.e., f, g can only be initialized in a constructor) *) > > Or should READONLY be a property of designators? i.e., > > RECORD READONLY f: T; g: T END (* READONLY fields can never be modified > (i.e., f cannot be modified, but g can be modified) *) > > This needs fleshing out. For example, if READONLY attaches to types, then > what are the assignment rules for non-readonly and readonly? More > complicated still, what are the subtype rules? Similar things are discussed in "Systems programming with Modula-3" by Greg Nelson et.al., aren't they? From dragisha at m3w.org Sun Mar 23 09:43:15 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 23 Mar 2008 09:43:15 +0100 Subject: [M3devel] blue sky proposal :) In-Reply-To: <712302B7-3312-464D-AB97-316E3F07DA10@cs.purdue.edu> References: <1206191868.16885.55.camel@hias.m3w.org> <712302B7-3312-464D-AB97-316E3F07DA10@cs.purdue.edu> Message-ID: <1206261795.18073.25.camel@hias.m3w.org> On Sat, 2008-03-22 at 12:26 -0400, Tony Hosking wrote: > Messing with VM protection mechanisms *will* impact performance, both > to set things up and to maintain. For example, the GC needs to write > to those pages as it operates, so you are going to put extra work on > the GC. Yuck! Ok. > > Here is the germ of an alternative idea -- let add a READONLY > declaration, with enforcement by the compiler. Should READONLY be a > property of types? i.e., > > READONLY RECORD f: T; g: T; END (* records of this type can never be > modified (i.e., f, g can only be initialized in a constructor) *) Constructor, or better - in "owning" module. It would be one exporting interface where READONLY something is declared. Idea is - ADT. External users can work only through strict interfaces. IMPLEMENTATION owner is responsible for these interfaces, and also one who can access these fields. Any "friends" mechanism would be witness of bad initial design. If some new module needs really bad to write to these fields, I would propose library level private INTERFACE where methods writing to fields are exported. Need to write to such fields is much rarer than need to read - this way we speed up what we use most. > > Or should READONLY be a property of designators? i.e., > > RECORD READONLY f: T; g: T END (* READONLY fields can never be > modified (i.e., f cannot be modified, but g can be modified) *) I think this is better. This way we can use compiler instead of methods that only return value of some private field. > This needs fleshing out. For example, if READONLY attaches to types, > then what are the assignment rules for non-readonly and readonly? > More complicated still, what are the subtype rules? And again, designators are better. Readonly fields are readonly to clients. Subtype is client. Above mentioned exception in reserve for real needs. > > Food for thought... > > On Mar 22, 2008, at 9:17 AM, Dragi?a Duri? wrote: > > > Now that VM mechanism is fully obsoleted by Mr. Hoskings GC... Why > > would > > not we make something like this... > > > > <* READONLY object *> - to move object to VM protected pages of > > traced > > heap, so we can have RuntimeError.E when some code tries to write into > > object > > > > <* WRITEABLE object *> - to return it back to normal traced heap pages > > > > Pragmas... Can be ignored by non-compatible compiler, and totally are > > not changing language semantics. They just offer great debugging tool > > and help us define/enforce some static properties of our structures. > > > > This, of course, is something for GC people to do... As I see current > > situation (as deep as I went into it) it will not affect performance > > at > > all, and it will (IMHO) help straighten GC code even more :). > > > > dd > > -- > > Dragi?a Duri? > -- Dragi?a Duri? From dragisha at m3w.org Sun Mar 23 10:07:19 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 23 Mar 2008 10:07:19 +0100 Subject: [M3devel] blue sky proposal :) In-Reply-To: References: <1206191868.16885.55.camel@hias.m3w.org> <712302B7-3312-464D-AB97-316E3F07DA10@cs.purdue.edu> Message-ID: <1206263239.18073.29.camel@hias.m3w.org> On Sat, 2008-03-22 at 23:12 +0100, Henning Thielemann wrote: > > This needs fleshing out. For example, if READONLY attaches to types, then > > what are the assignment rules for non-readonly and readonly? More > > complicated still, what are the subtype rules? > > Similar things are discussed in "Systems programming with Modula-3" by > Greg Nelson et.al., aren't they? Section 8.4? Or something else? -- Dragi?a Duri? From hosking at cs.purdue.edu Mon Mar 24 15:29:23 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 24 Mar 2008 10:29:23 -0400 Subject: [M3devel] Checkins Message-ID: I woke up to a bunch of failed tinderbox builds this morning. Please, before checking anything in, make sure that things at least will build first. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Mon Mar 24 15:36:56 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 24 Mar 2008 10:36:56 -0400 Subject: [M3devel] Checkins Message-ID: <7CD425CD-B4F4-4002-A728-79E84BD6DE0B@cs.purdue.edu> It seems that the reason things failed is because a checkin of multiple related changes was not done as a single atomic commit. This meant that partial checkout of inconsistent code occurred while bits and pieces of a coherent set of changes were being singly checked in. Please try to bulk checkin related changes so as to avoid these sorts of inconsistencies. cvs does a great job of bundling an entire set of changes as one commit. It also has the nice side-effect of not bombarding folks with zillions of commit e-mails -- instead the bundled commit comes in one message. If we use the tools properly, then everyone can play nicely together. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Mon Mar 24 17:37:12 2008 From: jayk123 at hotmail.com (Jay) Date: Mon, 24 Mar 2008 16:37:12 +0000 Subject: [M3devel] Checkins In-Reply-To: <7CD425CD-B4F4-4002-A728-79E84BD6DE0B@cs.purdue.edu> References: <7CD425CD-B4F4-4002-A728-79E84BD6DE0B@cs.purdue.edu> Message-ID: Tony, The problem was I forgot to add a file, right? That's something the tools are setup to make you fail at as I understand. Right? I have to remember to cvs add whenever I create a file. Building works either way. After commiting, I updated another machine, noticed the missing file, added it. I thought other than the forgotten file, the changes were coherent, at least for non-NT386GNU. - Jay From: hosking at cs.purdue.eduTo: m3devel at elegosoft.comDate: Mon, 24 Mar 2008 10:36:56 -0400CC: jkrell at elego.deSubject: [M3devel] Checkins It seems that the reason things failed is because a checkin of multiple related changes was not done as a single atomic commit. This meant that partial checkout of inconsistent code occurred while bits and pieces of a coherent set of changes were being singly checked in. Please try to bulk checkin related changes so as to avoid these sorts of inconsistencies. cvs does a great job of bundling an entire set of changes as one commit. It also has the nice side-effect of not bombarding folks with zillions of commit e-mails -- instead the bundled commit comes in one message. If we use the tools properly, then everyone can play nicely together. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Mon Mar 24 17:43:31 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 24 Mar 2008 12:43:31 -0400 Subject: [M3devel] Checkins In-Reply-To: References: <7CD425CD-B4F4-4002-A728-79E84BD6DE0B@cs.purdue.edu> Message-ID: I didn't take the time to look to closely at what caused the failure, but I did notice a smear of checkins spread over time. Better to roll it all into one checkin. I usually do a cvs release to see if there are files (marked by ?) that I forgot to add, and add them, before doing the big checkin as one top-level command. On Mar 24, 2008, at 12:37 PM, Jay wrote: > Tony, The problem was I forgot to add a file, right? > That's something the tools are setup to make you fail at as I > understand. Right? I have to remember to cvs add whenever I create a > file. Building works either way. > After commiting, I updated another machine, noticed the missing > file, added it. > > I thought other than the forgotten file, the changes were coherent, > at least for non-NT386GNU. > > - Jay > From: hosking at cs.purdue.edu > To: m3devel at elegosoft.com > Date: Mon, 24 Mar 2008 10:36:56 -0400 > CC: jkrell at elego.de > Subject: [M3devel] Checkins > > It seems that the reason things failed is because a checkin of > multiple related changes was not done as a single atomic commit. > This meant that partial checkout of inconsistent code occurred while > bits and pieces of a coherent set of changes were being singly > checked in. Please try to bulk checkin related changes so as to > avoid these sorts of inconsistencies. cvs does a great job of > bundling an entire set of changes as one commit. It also has the > nice side-effect of not bombarding folks with zillions of commit e- > mails -- instead the bundled commit comes in one message. > > If we use the tools properly, then everyone can play nicely together. > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Mon Mar 24 17:54:40 2008 From: jayk123 at hotmail.com (Jay) Date: Mon, 24 Mar 2008 16:54:40 +0000 Subject: [M3devel] Checkins In-Reply-To: References: <7CD425CD-B4F4-4002-A728-79E84BD6DE0B@cs.purdue.edu> Message-ID: hm, cvs release, didn't know about that. I thought I had been much slower actually. I added a file first, to better show history, then one well tested, on one platform, thought to be coherent checkin, then updated my Mac and found the missing file, fixed it, etc. There was also a missing file for NT386GNU that nobody would notice. I also set up my testing to build the more-Posix path on NT386GNU to be sure it built and ran. I am not much into the habit of testing on more than one machine/platform before submitting, though sometimes, and actually NT386GNU makes that easier -- two platforms on one machine. I am more in the habit of testing other machines right after commit. I haven't found a convenient way to move changes..cvs diff into a file, copy that, and patch? Or maybe share out the tree over the network and build in one tree directly? I should probably try that.. And I can't checkin from the top as I have other definitely unrelated unfinished un-affecting changes, true that sets a trap for myself. Without the commit mail, what do you run to see recent commits? cvs filelog likes to default to showing the full history. I'd like to say, just show the last 10 commits, or the last 20 commits.. I'm operating relatively blindly wrt cvs skills I'm afraid. And you manually check the tinderbox pages? They don't send mail? I do run cvs diff but I have a bunch of junk..gotta clean it up.. ? bin? caltech-parser/parserlib/.parsertop? caltech-parser/parserlib/html/SeekRd.html? caltech-parser/parserlib/html/calc.html? caltech-parser/parserlib/html/index.html? caltech-parser/parserlib/html/kext.html? caltech-parser/parserlib/html/klex.html? caltech-parser/parserlib/html/ktok.html? caltech-parser/parserlib/html/kyacc.html? caltech-parser/parserlib/html/lr.html? caltech-parser/parserlib/html/m3build.html? caltech-parser/parserlib/html/macroizations? m3-db/smalldb/src/mine? m3-db/smalldb/src/orig? m3-libs/arithmetic/NT386? m3-libs/arithmetic/NT386GNU? m3-libs/commandrw/NT386? m3-libs/commandrw/SOLgnu? m3-libs/deepcopy/NT386? m3-libs/libm3/src/os/NT386GNU? m3-libs/libm3/src/os/POSIX/ProcessCygwin.m3 - Jay CC: m3devel at elegosoft.com; jkrell at elego.deFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] CheckinsDate: Mon, 24 Mar 2008 12:43:31 -0400 I didn't take the time to look to closely at what caused the failure, but I did notice a smear of checkins spread over time. Better to roll it all into one checkin. I usually do a cvs release to see if there are files (marked by ?) that I forgot to add, and add them, before doing the big checkin as one top-level command. On Mar 24, 2008, at 12:37 PM, Jay wrote: Tony, The problem was I forgot to add a file, right?That's something the tools are setup to make you fail at as I understand. Right? I have to remember to cvs add whenever I create a file. Building works either way.After commiting, I updated another machine, noticed the missing file, added it. I thought other than the forgotten file, the changes were coherent, at least for non-NT386GNU. - Jay From: hosking at cs.purdue.eduTo: m3devel at elegosoft.comDate: Mon, 24 Mar 2008 10:36:56 -0400CC: jkrell at elego.deSubject: [M3devel] Checkins It seems that the reason things failed is because a checkin of multiple related changes was not done as a single atomic commit. This meant that partial checkout of inconsistent code occurred while bits and pieces of a coherent set of changes were being singly checked in. Please try to bulk checkin related changes so as to avoid these sorts of inconsistencies. cvs does a great job of bundling an entire set of changes as one commit. It also has the nice side-effect of not bombarding folks with zillions of commit e-mails -- instead the bundled commit comes in one message. If we use the tools properly, then everyone can play nicely together. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Mon Mar 24 18:13:49 2008 From: jayk123 at hotmail.com (Jay) Date: Mon, 24 Mar 2008 17:13:49 +0000 Subject: [M3devel] Checkins In-Reply-To: References: <7CD425CD-B4F4-4002-A728-79E84BD6DE0B@cs.purdue.edu> Message-ID: I checked the web pages. The one error I saw a few times was: 4549 NEXT Fatal Error: unable to open file for reading: ../src/os/POSIX/ProcessPosixCommon.i3: errno=2 which I fixed a while ago, had forgotten cvs add. Or, oh, actually..I might have been bitten by my own attempt to do things efficiently and correctly (er..). cvs dif > \diff.txt findstr RCS \diff.txt > \opened.txt paste those files into a cvs commit line..but adds aren't there darn. My PPC_DARWIN build is working. - Jay From: jayk123 at hotmail.com [snip] From: hosking at cs.purdue.edu[snip] -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Mon Mar 24 18:39:29 2008 From: jayk123 at hotmail.com (Jay) Date: Mon, 24 Mar 2008 17:39:29 +0000 Subject: [M3devel] possible cygwin createprocess/fork cost measurements.. In-Reply-To: <20080321121953.39e767bl9ccgw0co@mail.elegosoft.com> References: <20080320090948.sbqj156m800s080o@mail.elegosoft.com> <20080321121953.39e767bl9ccgw0co@mail.elegosoft.com> Message-ID: I'm using q_exec now, except where my command line has a semicolon within a parameter. I'm using spawn now too generally.scripts\upgrade.py takes about a minute for NT386, about 3:45 for NT386GNU. I don't know how long it took before, but it does feel faster, and I building just m3core I had numbers for. About the only things left now for NT386GNU are: allow building a "Windows" app to work run the tests and tinderbox MAYBE see if batching cm3cg/as can save more time. Other folks have complained on the gcc mailing list that gcc should just output .o files directly.. - Jay From: jayk123 at hotmail.com [snip] > Have you had a look at System.ExecuteList and q_exec? Sorry, not yet.Yes it sounds like it will fix the sh wrap issue. But not the fork/exec vs. spawn, which I will deal with. > compiler backend. Integrated backends can be developped independently> of gcc though.Yes, I think more of these will be good. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Mon Mar 24 18:52:25 2008 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Mon, 24 Mar 2008 18:52:25 +0100 (CET) Subject: [M3devel] [M3commit] Test: Mail list down In-Reply-To: <1206375543.3767.0.camel@faramir.m3w.org> Message-ID: <898271.83754.qm@web27105.mail.ukl.yahoo.com> Hi: I remember some days ago I sent an email to m3devel but it didn't get mailed, it seems reasonable to ask for a ticket or receipt of the sent mails to the senders from the mailer? Thanks Dragi??a Duri?? wrote: m3devel is also down? On Mon, 2008-03-24 at 10:37 -0400, Tony Hosking wrote: > I haven't been seeing commits... > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > -- Dragi??a Duri?? --------------------------------- Enviado desde Correo Yahoo! Disfruta de una bandeja de entrada m?s inteligente.. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Mon Mar 24 19:14:44 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Mon, 24 Mar 2008 19:14:44 +0100 Subject: [M3devel] Checkins In-Reply-To: References: <7CD425CD-B4F4-4002-A728-79E84BD6DE0B@cs.purdue.edu> Message-ID: <1206382484.3767.7.camel@faramir.m3w.org> I would use cvs .... status 2>/dev/null to know what was changed so I can plan next steps when updating. Usually I do cvs ... diff also. This way I know what and how was changed before I commit anything. Works well... Most of time :) dd On Mon, 2008-03-24 at 16:37 +0000, Jay wrote: > Tony, The problem was I forgot to add a file, right? > That's something the tools are setup to make you fail at as I > understand. Right? I have to remember to cvs add whenever I create a > file. Building works either way. > After commiting, I updated another machine, noticed the missing file, > added it. > > I thought other than the forgotten file, the changes were coherent, at > least for non-NT386GNU. > > - Jay > > ______________________________________________________________ > From: hosking at cs.purdue.edu > To: m3devel at elegosoft.com > Date: Mon, 24 Mar 2008 10:36:56 -0400 > CC: jkrell at elego.de > Subject: [M3devel] Checkins > > It seems that the reason things failed is because a checkin of > multiple related changes was not done as a single atomic > commit. This meant that partial checkout of inconsistent code > occurred while bits and pieces of a coherent set of changes > were being singly checked in. Please try to bulk checkin > related changes so as to avoid these sorts of > inconsistencies. cvs does a great job of bundling an entire > set of changes as one commit. It also has the nice > side-effect of not bombarding folks with zillions of commit > e-mails -- instead the bundled commit comes in one message. > > > If we use the tools properly, then everyone can play nicely > together. > > Antony Hosking | Associate Professor | Computer Science | > Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > -- Dragi?a Duri? From dabenavidesd at yahoo.es Mon Mar 24 19:43:03 2008 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Mon, 24 Mar 2008 19:43:03 +0100 (CET) Subject: [M3devel] [M3commit] Test: Mail list down In-Reply-To: <1206382222.3767.2.camel@faramir.m3w.org> Message-ID: <237912.81476.qm@web27108.mail.ukl.yahoo.com> Hi: I meant, the sender should send a prior confirmation to the m3devel user account (email) besides the mail for the users of m3devel. That's like a little wait for moderator approval or something like that. Thanks Dragi??a Duri?? escribi?: Sender gets email back together with others. Maybe there's some setting other than default you can change to affect this, but I don't think you tweaked anything. Neither did I. dd On Mon, 2008-03-24 at 18:51 +0100, Daniel Alejandro Benavides D. wrote: > Hi: > I remember some days ago I sent an email to m3devel but it didn't get > mailed, it seems reasonable to ask for a ticket or receipt of the sent > mails to the senders from the mailer? > Thanks > > Dragi? ??a Duri????? escribi??: > m3devel is also down? > > On Mon, 2008-03-24 at 10:37 -0400, Tony Hosking wrote: > > I haven't been seeing commits... > > > > Antony Hosking | Associate Professor | Computer Science | > Purdue > > University > > 305 N. University Street | West Lafayette | IN 47907 | USA > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > -- > Dragi? ??a Duri????? > > > > > > ______________________________________________________________________ > > Enviado desde Correo Yahoo! > Disfruta de una bandeja de entrada m??s inteligente.. -- Dragi??a Duri?? --------------------------------- Enviado desde Correo Yahoo! Disfruta de una bandeja de entrada m?s inteligente.. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Mon Mar 24 20:07:55 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 24 Mar 2008 15:07:55 -0400 Subject: [M3devel] possible cygwin createprocess/fork cost measurements.. In-Reply-To: References: <20080320090948.sbqj156m800s080o@mail.elegosoft.com> <20080321121953.39e767bl9ccgw0co@mail.elegosoft.com> Message-ID: <944E5C40-B1CB-419B-A3B8-5C2372DD038C@cs.purdue.edu> > MAYBE see if batching cm3cg/as can save more time. > Other folks have complained on the gcc mailing list that gcc should > just output .o files directly.. gcc itself is a top-level driver program that generates as and forks an as. I don't think it's possible to do what you suggest. > > > - Jay > > > From: jayk123 at hotmail.com > > [snip] > > > Have you had a look at System.ExecuteList and q_exec? > > Sorry, not yet. > Yes it sounds like it will fix the sh wrap issue. But not the fork/ > exec vs. spawn, which I will deal with. > > > compiler backend. Integrated backends can be developped > independently > > of gcc though. > > Yes, I think more of these will be good. > > - Jay > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Mon Mar 24 20:25:00 2008 From: jayk123 at hotmail.com (Jay) Date: Mon, 24 Mar 2008 19:25:00 +0000 Subject: [M3devel] possible cygwin createprocess/fork cost measurements.. In-Reply-To: <944E5C40-B1CB-419B-A3B8-5C2372DD038C@cs.purdue.edu> References: <20080320090948.sbqj156m800s080o@mail.elegosoft.com> <20080321121953.39e767bl9ccgw0co@mail.elegosoft.com> <944E5C40-B1CB-419B-A3B8-5C2372DD038C@cs.purdue.edu> Message-ID: I understand, but it need not be this way. http://gcc.gnu.org/ml/gcc/2004-05/msg00318.html GCC is the last compiler I'm using that doesn't have an integrated object file generator. It's way pasttime to integrate BFD, opcodes, etc. or whatever other solution to replace this generate-text-and-pipe-it-to-the-assembler '70s technology.That means one less thing on the host that may not match GCC's expectations. (e.g. AltiVec register names and the host assemblers) - Jay CC: wagner at elegosoft.com; m3devel at elegosoft.com [snip] gcc itself is a top-level driver program that generates as and forks an as. I don't think it's possible to do what you suggest. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Mon Mar 24 20:58:07 2008 From: jayk123 at hotmail.com (Jay) Date: Mon, 24 Mar 2008 19:58:07 +0000 Subject: [M3devel] mentor or Trestle bug? Message-ID: Saw this on NT386GNU: Jay at jay-win9 /cygdrive/c/cm3/bin $ *** *** runtime error: *** Exception "Filter.FatalError" not in RAISES list *** file "../src/split/Filter.m3", line 41 *** [3] Segmentation fault (core dumped) ./mentor not sure if these are from the same process or not. Mentor didn't seem to bring up any animations, though it did bring up a fair amount of GUI. I'm not likely to get around to really exercising/debugging the gui stuff in NT386GNU. If anyone else wants to... I'm uploading current min and std NT386GNU packages right now. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Mon Mar 24 21:03:31 2008 From: jayk123 at hotmail.com (Jay) Date: Mon, 24 Mar 2008 20:03:31 +0000 Subject: [M3devel] NT386GNU naming conventions? Message-ID: I switched NT386GNU to use: foo for executables foo.so for shared libraries libfoo.a for static or import libraries libfoo.a.sa for standalone libraries when there is an import lib I'm thinking maybe foo and foo.so are taking things too far. "Out Posixing Cygwin". They use foo.exe and either foo.dll or cygfoo.dll. Anyone care? I'm thinking of wrapping up NT386GNU work for a bit. Is anyone insterested in trying it? "min" and "std" distributions will be on the web shortly. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Mon Mar 24 21:05:51 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 24 Mar 2008 16:05:51 -0400 Subject: [M3devel] possible cygwin createprocess/fork cost measurements.. In-Reply-To: References: <20080320090948.sbqj156m800s080o@mail.elegosoft.com> <20080321121953.39e767bl9ccgw0co@mail.elegosoft.com> <944E5C40-B1CB-419B-A3B8-5C2372DD038C@cs.purdue.edu> Message-ID: <3064DA82-8130-41B4-90CF-E40BD2143876@cs.purdue.edu> Blurg! Tools should do well what they are supposed to do and do no more than necessary. Assemblers are good at assembling, so why reinvent the wheel? See: http://linux.oreillynet.com/lpt/a/302 The counterpoint to this argument is Windoze... On Mar 24, 2008, at 3:25 PM, Jay wrote: > I understand, but it need not be this way. > > http://gcc.gnu.org/ml/gcc/2004-05/msg00318.html > > GCC is the last compiler I'm using that doesn't have an integrated > object file generator. It's way past > time to integrate BFD, opcodes, etc. or whatever other solution to > replace this generate-text-and-pipe-it-to-the-assembler '70s > technology. > That means one less thing on the host that may not match GCC's > expectations. (e.g. AltiVec register names and the host assemblers) > > - Jay > > > CC: wagner at elegosoft.com; m3devel at elegosoft.com > [snip] > gcc itself is a top-level driver program that generates as and forks > an as. I don't think it's possible to do what you suggest. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Mon Mar 24 21:08:45 2008 From: jayk123 at hotmail.com (Jay) Date: Mon, 24 Mar 2008 20:08:45 +0000 Subject: [M3devel] new NT386GNU releases Message-ID: http://modula3.elegosoft.com/cm3/uploaded-archives => http://modula3.elegosoft.com/cm3/uploaded-archives/cm3-std-POSIX-NT386GNU-d5.7.0.tar.bz2 http://modula3.elegosoft.com/cm3/uploaded-archives/cm3-min-POSIX-NT386GNU-d5.7.0.tar.bz2 This is the first with X Windows stuff. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Mon Mar 24 21:38:50 2008 From: jayk123 at hotmail.com (Jay) Date: Mon, 24 Mar 2008 20:38:50 +0000 Subject: [M3devel] possible cygwin createprocess/fork cost measurements.. In-Reply-To: <3064DA82-8130-41B4-90CF-E40BD2143876@cs.purdue.edu> References: <20080320090948.sbqj156m800s080o@mail.elegosoft.com> <20080321121953.39e767bl9ccgw0co@mail.elegosoft.com> <944E5C40-B1CB-419B-A3B8-5C2372DD038C@cs.purdue.edu> <3064DA82-8130-41B4-90CF-E40BD2143876@cs.purdue.edu> Message-ID: I understand. But there are valid counterpoints. >From many people's point of view, a compiler outputs an .obj file. Where the layers are, as long as you don't use the intermediate layers, and/or as long as they aren't "sealed against debugging", don't care. And the more layers, the slower. I realize though, that producing an .obj file maybe is actually a significant amount of work and best left to another layer. But the code is out there, both in gas to be linked into cm3cg, and e.g. in the integrated NT386 backend. As well, generating /textual/ assembly code really does seem like a waste. The NT386 compiler is ROUGHLY 4 times the speed of NT386GNU. That must be coming from SOMEWHERE, though I don't really know where. It could be that cm3cg merely spends a lot of CPU trying to improve the code and hardly achieving anything. - Jay CC: wagner at elegosoft.com; m3devel at elegosoft.com From: hosking at cs.purdue.edu To: jayk123 at hotmail.com Subject: Re: [M3devel] possible cygwin createprocess/fork cost measurements.. Date: Mon, 24 Mar 2008 16:05:51 -0400 Blurg! Tools should do well what they are supposed to do and do no more than necessary. Assemblers are good at assembling, so why reinvent the wheel? See: http://linux.oreillynet.com/lpt/a/302 The counterpoint to this argument is Windoze... On Mar 24, 2008, at 3:25 PM, Jay wrote:I understand, but it need not be this way. http://gcc.gnu.org/ml/gcc/2004-05/msg00318.html GCC is the last compiler I'm using that doesn't have an integrated object file generator. It's way past time to integrate BFD, opcodes, etc. or whatever other solution to replace this generate-text-and-pipe-it-to-the-assembler '70s technology. That means one less thing on the host that may not match GCC's expectations. (e.g. AltiVec register names and the host assemblers) - Jay CC: wagner at elegosoft.com; m3devel at elegosoft.com [snip] gcc itself is a top-level driver program that generates as and forks an as. I don't think it's possible to do what you suggest. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Mon Mar 24 22:03:53 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 24 Mar 2008 17:03:53 -0400 Subject: [M3devel] possible cygwin createprocess/fork cost measurements.. In-Reply-To: References: <20080320090948.sbqj156m800s080o@mail.elegosoft.com> <20080321121953.39e767bl9ccgw0co@mail.elegosoft.com> <944E5C40-B1CB-419B-A3B8-5C2372DD038C@cs.purdue.edu> <3064DA82-8130-41B4-90CF-E40BD2143876@cs.purdue.edu> Message-ID: On Mar 24, 2008, at 4:38 PM, Jay wrote: > The NT386 compiler is ROUGHLY 4 times the speed of NT386GNU. It doesn't pass through the 4 stages of IR that the gcc backend uses: m3cg => .mo => gcc trees => gcc rtl => .s => .o. Hence the speed. The integrated backend compiler does m3cg => .o. > That must be coming from SOMEWHERE, though I don't really know where. > It could be that cm3cg merely spends a lot of CPU trying to improve > the code > and hardly achieving anything. > > - Jay > > CC: wagner at elegosoft.com; m3devel at elegosoft.com > From: hosking at cs.purdue.edu > To: jayk123 at hotmail.com > Subject: Re: [M3devel] possible cygwin createprocess/fork cost > measurements.. > Date: Mon, 24 Mar 2008 16:05:51 -0400 > > Blurg! Tools should do well what they are supposed to do and do no > more than necessary. Assemblers are good at assembling, so why > reinvent the wheel? > > See: http://linux.oreillynet.com/lpt/a/302 > > The counterpoint to this argument is Windoze... > > On Mar 24, 2008, at 3:25 PM, Jay wrote: > I understand, but it need not be this way. > > http://gcc.gnu.org/ml/gcc/2004-05/msg00318.html > > GCC is the last compiler I'm using that doesn't have an integrated > object file generator. It's way past > time to integrate BFD, opcodes, etc. or whatever other solution to > replace this generate-text-and-pipe-it-to-the-assembler '70s > technology. > That means one less thing on the host that may not match GCC's > expectations. (e.g. AltiVec register names and the host assemblers) > > - Jay > > > CC: wagner at elegosoft.com; m3devel at elegosoft.com > [snip] > gcc itself is a top-level driver program that generates as and forks > an as. I don't think it's possible to do what you suggest. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Tue Mar 25 02:30:50 2008 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Tue, 25 Mar 2008 02:30:50 +0100 (CET) Subject: [M3devel] [M3commit] Test: Mail list down In-Reply-To: <237912.81476.qm@web27108.mail.ukl.yahoo.com> Message-ID: <297026.57884.qm@web27109.mail.ukl.yahoo.com> Hi: I also didn't mean that, I'm sorry. What I mean is that if it's possible to the m3devel daemon send a prior confirmation to the m3devel subscript (user) prior publishing his email and sending all the subscripts the desired email. I think I saw this kind of functionality on a sourceforge mail list. This serves for indicating the moderator approval and gives you a feedback from the part of the daemon immediately you send a email to it. Thanks. "Daniel Alejandro Benavides D." wrote: Hi: I meant, the sender should send a prior confirmation to the m3devel user account (email) besides the mail for the users of m3devel. That's like a little wait for moderator approval or something like that. Thanks Dragi??a Duri?? escribi?: Sender gets email back together with others. Maybe there's some setting other than default you can change to affect this, but I don't think you tweaked anything. Neither did I. dd On Mon, 2008-03-24 at 18:51 +0100, Daniel Alejandro Benavides D. wrote: > Hi: > I remember some days ago I sent an email to m3devel but it didn't get > mailed, it seems reasonable to ask for a ticket or receipt of the sent > mails to the senders from the mailer? > Thanks > > Dragi? ??a Duri????? escribi??: > m3devel is also down? > > On Mon, 2008-03-24 at 10:37 -0400, Tony Hosking wrote: > > I haven't been seeing commits... > > > > Antony Hosking | Associate Professor | Computer Science | > Purdue > > University > > 305 N. University Street | West Lafayette | IN 47907 | USA > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > -- > Dragi? ??a Duri????? > > > > > > ______________________________________________________________________ > > Enviado desde Correo Yahoo! > Disfruta de una bandeja de entrada m??s inteligente.. -- Dragi??a Duri?? --------------------------------- Enviado desde Correo Yahoo! Disfruta de una bandeja de entrada m?s inteligente.. --------------------------------- Enviado desde Correo Yahoo! Disfruta de una bandeja de entrada m?s inteligente.. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Tue Mar 25 23:24:35 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 25 Mar 2008 23:24:35 +0100 Subject: [M3devel] possible cygwin createprocess/fork cost measurements.. In-Reply-To: References: <20080320090948.sbqj156m800s080o@mail.elegosoft.com> <20080321121953.39e767bl9ccgw0co@mail.elegosoft.com> <944E5C40-B1CB-419B-A3B8-5C2372DD038C@cs.purdue.edu> <3064DA82-8130-41B4-90CF-E40BD2143876@cs.purdue.edu> Message-ID: <20080325232435.p6esq8jj40soowwo@mail.elegosoft.com> Quoting Tony Hosking : > On Mar 24, 2008, at 4:38 PM, Jay wrote: >> The NT386 compiler is ROUGHLY 4 times the speed of NT386GNU. > > It doesn't pass through the 4 stages of IR that the gcc backend uses: > m3cg => .mo => gcc trees => gcc rtl => .s => .o. Hence the speed. > The integrated backend compiler does m3cg => .o. That's probably the main reason; another is its flexibility and portability, which will cost some performance, too. I agree that the gcc backend is really slow in comparison. It's really great wrt. all the different targets though ;-) Maybe avoiding some intermediate files may help a bit? I remember there was the -pipe option for gcc: -pipe Use pipes rather than temporary files for communication between the various stages of compilation. This fails to work on some systems where the assembler is unable to read from a pipe; but the GNU assembler has no trouble. I don't know how easily this can be used with cm3cg though. Olaf >> That must be coming from SOMEWHERE, though I don't really know where. >> It could be that cm3cg merely spends a lot of CPU trying to improve >> the code >> and hardly achieving anything. -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From dragisha at m3w.org Fri Mar 28 09:11:11 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Fri, 28 Mar 2008 09:11:11 +0100 Subject: [M3devel] proedure closures? Message-ID: <1206691871.23390.55.camel@faramir.m3w.org> I would like to construct a call to arbitrary procedure/method whose argument list and address I know. Is there a lowlevel way to do this, or its only possible throught stubgen like and/or enumerating all possible procedure types and loopholing to them through some kind of CASE? dd -- Dragi?a Duri? From wagner at elegosoft.com Fri Mar 28 10:24:25 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Fri, 28 Mar 2008 10:24:25 +0100 Subject: [M3devel] proedure closures? In-Reply-To: <1206691871.23390.55.camel@faramir.m3w.org> References: <1206691871.23390.55.camel@faramir.m3w.org> Message-ID: <20080328102425.mnu9f53cgo4s8wss@mail.elegosoft.com> Quoting Dragi?a Duri? : > I would like to construct a call to arbitrary procedure/method whose > argument list and address I know. Is there a lowlevel way to do this, or > its only possible throught stubgen like and/or enumerating all possible > procedure types and loopholing to them through some kind of CASE? TYPECASE on the procedure type followed by NARROW comes to mind; I have never tried that with procedures though. Another option might be to declare the procedures as methods in sub-closures and perform the NARROW on the closure class. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From lemming at henning-thielemann.de Fri Mar 28 10:50:17 2008 From: lemming at henning-thielemann.de (Henning Thielemann) Date: Fri, 28 Mar 2008 10:50:17 +0100 (CET) Subject: [M3devel] proedure closures? In-Reply-To: <20080328102425.mnu9f53cgo4s8wss@mail.elegosoft.com> References: <1206691871.23390.55.camel@faramir.m3w.org> <20080328102425.mnu9f53cgo4s8wss@mail.elegosoft.com> Message-ID: On Fri, 28 Mar 2008, Olaf Wagner wrote: > Quoting Dragi?a Duri? : > >> I would like to construct a call to arbitrary procedure/method whose >> argument list and address I know. Is there a lowlevel way to do this, or >> its only possible throught stubgen like and/or enumerating all possible >> procedure types and loopholing to them through some kind of CASE? > > TYPECASE on the procedure type followed by NARROW comes to mind; > I have never tried that with procedures though. I suspect this won't work because the 'arrow rule' which defines the subtype relation of procedures was discussed in length in "Systems programming with Modula-3" but was finally dropped due to implementation issues. From jayk123 at hotmail.com Sat Mar 29 20:46:20 2008 From: jayk123 at hotmail.com (Jay) Date: Sat, 29 Mar 2008 19:46:20 +0000 Subject: [M3devel] new NT386GNU releases Message-ID: I moved these away to "funny" names..that don't show on the web page oops..and replaced them with archives that use Win32 naming conventions (foo.lib, foo.lib.sa, foo.exe, foo.dll). Switching this stuff around is pretty quick and easy. Olaf, I'm using 140meg of diskspace in /var/www/modula3.elegosoft.com/cm3/uploaded-archivesMaybe should a) prune out older archives b) see about making "std" smaller, such as not including all the static .libs, like maybe just m3core.lib.sa and libm3.core.sa in order to build cm3 and mklib. Maybe not making all the static .libs by default as well. Maybe stripping all/many/most of the binaries too. I like having symbols, but most of them go unused most of the time, and they aren't hard to regenerate, as long as you have a consistent repro and don't have to debug something that is "precious". I think I'll try deleting all *.lib.sa files except m3core.lib.sa/libm3core.sa -- whatever mklib and cm3 use. And sysutils. Those three files can be crucial in bootstrapping, when an older compiler can't build a newer runtime, but can build a newer compiler, as was the case recently. The native NT386 archives I make are similarly bloated, and even the cminstall archives might be? btw, these are my form of archive. There is no cminstall and nested archive. They extract to cm3-min-posix-5.2.7\bin\cm3.exe and such, which you can then rename or copy into \cm3\bin\cm3.exe or /usr/local/bin/cm3.exe or whatnot, and the config files should just work for pretty much any configuration, as well, you should be able to edit cm3.cfg by just one line to retarget to any of the three NT386/NT386GNU/NT386MINGNU targets, as long as you have the appropriate C build environment setup. Any chance anyone is going to ever use this stuff? :) - Jay From: jayk123 at hotmail.comTo: m3devel at elegosoft.comSubject: new NT386GNU releasesDate: Mon, 24 Mar 2008 20:08:45 +0000 http://modula3.elegosoft.com/cm3/uploaded-archives=> http://modula3.elegosoft.com/cm3/uploaded-archives/cm3-std-POSIX-NT386GNU-d5.7.0.tar.bz2 http://modula3.elegosoft.com/cm3/uploaded-archives/cm3-min-POSIX-NT386GNU-d5.7.0.tar.bz2This is the first with X Windows stuff. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Sat Mar 29 21:05:39 2008 From: jayk123 at hotmail.com (Jay) Date: Sat, 29 Mar 2008 20:05:39 +0000 Subject: [M3devel] OLPC "XO" laptop Message-ID: So, I got an OLPC through G1G1. It's just a very low end Linux/x86 machine.(AMD Geode or such.) 1gig of flash, about 1/3rd full upon delivery256 meg RAM.Running Fedora Core 7, with modifications.X Windows.A window manager I'd never heard of before -- "matchbox" I think. It comes with ld, as, awk, sed, python, gdb, nano, vi, smalltalk? (eToys) it can record sound, still pictures, video, but the recorded video quality is terrible it has a Mozilla-based browser It has a decent gui text editor, but with indecent file open/save behavior, so not usable It comes withOUT gcc, perl, headers, libs, emacs, kate, nedit, make, ruby, tcl (kate is the most tolerable Unix editor I have found so far, used it some on Linux/PPC.) cminstall runs ok and finds ld and as.It turns out, you can point ld at .so files and it seems to mostlywork, but the resulting programs crash early on, I thinkfor lack of the usual startup code, since argv is 0 and argcand envp are "large negative numbers" -- I'm pretty sureenvp wasn't a valid address, and the actual error was due to argcbeing negative. Oh and you have to give ld the -I for interpretname command. But you can just do: su yum install gcc and easily/quickly/cheaply fill out the development environment, besidesdoing development on another machine and copying the files over, or besidesjust not caring at all about these machines, so catering to the missinggcc is probably not worthwhile. I'm still curious to look how thecommand line is usually retrieved. You can go at least go so far as: su yum install kdevelop automake autoconf emacs ruby tcl tk make perl nedit gcc The machine is quite slow.The keyboard is really small and /incredibly/ uncomfortable to use. I don't think mind it being "membrane", but the small keys are really hard on adult hands.I must try an external one. And an external drive to avoid writing too much and for more space,so that I might try building the Modula-3 system here. The screen has a high DPI so the text console is very hard to read.Even nedit, xterm, kdevelop suffer from the high DPI. There must be a setting to fix thisbut I havn't found it yet. (They have a "Terminal" activity that is perfectly readable/usable.)I haven't booted into single user mode yet. I think I need a developer key, have to waita day for that. It uses OpenFirmware I believe. If you run too many of the "activities", bogs down and must power cycle. I think it has no swap, so 256meg is it.(swap would probably kill the flash, and be even slower than usual) I originally "used", barely, Modula-3 I think on a system with 64meg.Or maybe 16 or 32.Not to mention much smaller systems. I don't YET have an EEPC or whatever it is called but it is probably much better. There's plenty of information at www.laptop.org. I know this is only barely on topic, but I thought fellow geeks might be interested. :) - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Sat Mar 29 21:19:54 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Sat, 29 Mar 2008 21:19:54 +0100 Subject: [M3devel] new NT386GNU releases In-Reply-To: References: Message-ID: <20080329211954.aj4tsikq8s8gk8g4@mail.elegosoft.com> Quoting Jay : > I moved these away to "funny" names..that don't show on the web page > oops..and replaced them with archives that use Win32 naming > conventions (foo.lib, foo.lib.sa, foo.exe, foo.dll). Switching this > stuff around is pretty quick and easy. > > Olaf, I'm using 140meg of diskspace in > /var/www/modula3.elegosoft.com/cm3/uploaded-archivesMaybe should a) > prune out older archives b) see about making "std" smaller, such as > not including all the static .libs, like maybe just m3core.lib.sa > and libm3.core.sa in order to build cm3 and mklib. Maybe not making > all the static .libs by default as well. Maybe stripping > all/many/most of the binaries too. I like having symbols, but most > of them go unused most of the time, and they aren't hard to > regenerate, as long as you have a consistent repro and don't have to > debug something that is "precious". Don't worry too much about the disk space on birch; there's still 98 GB free. If you are providing full binary installation archives, they will sure be large. Thus said, providing something smaller may be nice for those who keep the sources anyway; perhaps just distribute the core packages? > I think I'll try deleting all *.lib.sa files except > m3core.lib.sa/libm3core.sa -- whatever mklib and cm3 use. > And sysutils. > Those three files can be crucial in bootstrapping, when an older > compiler can't build a newer runtime, but can build a newer > compiler, as was the case recently. > The native NT386 archives I make are similarly bloated, and even the > cminstall archives might be? > > btw, these are my form of archive. There is no cminstall and nested > archive. They extract to cm3-min-posix-5.2.7\bin\cm3.exe and such, > which you can then rename or copy into \cm3\bin\cm3.exe or > /usr/local/bin/cm3.exe or whatnot, and the config files should just > work for pretty much any configuration, as well, you should be able > to edit cm3.cfg by just one line to retarget to any of the three > NT386/NT386GNU/NT386MINGNU targets, as long as you have the > appropriate C build environment setup. > > Any chance anyone is going to ever use this stuff? :) I certainly think so. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jayk123 at hotmail.com Sat Mar 29 21:54:43 2008 From: jayk123 at hotmail.com (Jay) Date: Sat, 29 Mar 2008 20:54:43 +0000 Subject: [M3devel] new NT386GNU releases In-Reply-To: <20080329211954.aj4tsikq8s8gk8g4@mail.elegosoft.com> References: <20080329211954.aj4tsikq8s8gk8g4@mail.elegosoft.com> Message-ID: > Thus said, providing something smaller may be nice for those> who keep the sources anyway; perhaps just distribute the core> packages? I provide "min" and "std"."min" is libm3, m3core, maybe sysutils, and cm3/cm3cg, and mklib (used on all NT386 targets) It is enough to build a) small non-gui prorams b) the whole system from source. I haven't studied the core/base lists to come to some "comfortable" feeling that they are a useful compromise. I'll probably just find them to be "arbitrary". Or are they some rational list? They are almost identical, which I find suspicious, both in their English names, and in their content. I offer the bare minimum or the kitchen sink. :) The "min" NT386GNU also has symbols in it. For NT386, they are separate. This is a reasonable way to separate symbols with the GNU tools. I should maybe start doing that. The "symbol" file is just the original file, I think even with all the code (wasteful, but easy), and the symbols are removed from the "main" executable in the usual strip way, and the file name of the other put in it. I don't know if there's also a timestamp/hash/guid. I'll still see about trimming out the static .libs from "std", though I suppose folks might really be interested in "standalone", and using lots of .libs, and not building the system themselves?? (I always suffer from thinking anything I have learned is then easy for anyone else, such as building this system from source to re-derive any of the files that I pruned out of the archive). I did prune out all the *.m3 files.. Thanks, - Jay > Date: Sat, 29 Mar 2008 21:19:54 +0100> From: wagner at elegosoft.com> To: m3devel at elegosoft.com> Subject: Re: [M3devel] new NT386GNU releases> > Quoting Jay :> > > I moved these away to "funny" names..that don't show on the web page > > oops..and replaced them with archives that use Win32 naming > > conventions (foo.lib, foo.lib.sa, foo.exe, foo.dll). Switching this > > stuff around is pretty quick and easy.> >> > Olaf, I'm using 140meg of diskspace in > > /var/www/modula3.elegosoft.com/cm3/uploaded-archivesMaybe should a) > > prune out older archives b) see about making "std" smaller, such as > > not including all the static .libs, like maybe just m3core.lib.sa > > and libm3.core.sa in order to build cm3 and mklib. Maybe not making > > all the static .libs by default as well. Maybe stripping > > all/many/most of the binaries too. I like having symbols, but most > > of them go unused most of the time, and they aren't hard to > > regenerate, as long as you have a consistent repro and don't have to > > debug something that is "precious".> > Don't worry too much about the disk space on birch; there's still> 98 GB free. If you are providing full binary installation archives,> they will sure be large.> > Thus said, providing something smaller may be nice for those> who keep the sources anyway; perhaps just distribute the core> packages?> > > I think I'll try deleting all *.lib.sa files except > > m3core.lib.sa/libm3core.sa -- whatever mklib and cm3 use.> > And sysutils.> > Those three files can be crucial in bootstrapping, when an older > > compiler can't build a newer runtime, but can build a newer > > compiler, as was the case recently.> > The native NT386 archives I make are similarly bloated, and even the > > cminstall archives might be?> >> > btw, these are my form of archive. There is no cminstall and nested > > archive. They extract to cm3-min-posix-5.2.7\bin\cm3.exe and such, > > which you can then rename or copy into \cm3\bin\cm3.exe or > > /usr/local/bin/cm3.exe or whatnot, and the config files should just > > work for pretty much any configuration, as well, you should be able > > to edit cm3.cfg by just one line to retarget to any of the three > > NT386/NT386GNU/NT386MINGNU targets, as long as you have the > > appropriate C build environment setup.> >> > Any chance anyone is going to ever use this stuff? :)> > I certainly think so.> > Olaf> -- > Olaf Wagner -- elego Software Solutions GmbH> Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany> phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95> http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin> Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194> -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Sat Mar 29 22:34:23 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Sat, 29 Mar 2008 22:34:23 +0100 Subject: [M3devel] new NT386GNU releases In-Reply-To: References: <20080329211954.aj4tsikq8s8gk8g4@mail.elegosoft.com> Message-ID: <20080329223423.uw4rg7fihcccg04g@mail.elegosoft.com> Quoting Jay : >> Thus said, providing something smaller may be nice for those> who >> keep the sources anyway; perhaps just distribute the core> packages? > I provide "min" and "std"."min" is libm3, m3core, maybe sysutils, > and cm3/cm3cg, and mklib (used on all NT386 targets) > It is enough to build a) small non-gui prorams b) the whole system > from source. Min is OK, I was confused and thinking there was only a full distribution. No need for core. > I haven't studied the core/base lists to come to some "comfortable" > feeling that they are a useful compromise. > I'll probably just find them to be "arbitrary". Or are they some > rational list? > They are almost identical, which I find suspicious, both in their > English names, and in their content. base was just used in the beginning of the porting, I think; we could/should remove it now. I'm mostly using core now for small but useful systems. > I offer the bare minimum or the kitchen sink. :) That's OK. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jayk123 at hotmail.com Sat Mar 29 22:59:46 2008 From: jayk123 at hotmail.com (Jay) Date: Sat, 29 Mar 2008 21:59:46 +0000 Subject: [M3devel] target naming again..DJGPP? Message-ID: Does anyone else find the idea of a DJGPP port "interesting"? I think I'll go ahead and do it. I know its useless but I find it "interesting".A few things will come before it. The DJGPP runtime appears to have the adequate support for alarm/setitimer,so it /should/ be fairly easy.I've already demonstrated to myself cooperating threading, and alarm/setitimer can be used for preemption, with the user posix threads. The easy work I did pruning down NT386GNU's *.i3 file to a minimum will be useful. What to call it? Some combination of (x86 or 386 or i386) and (DOS or MSDOS) and (DJGPP and/or GNU)? DJGPP MSDOS X86_MSDOS X86_DJGPP X86_MSDOS_DJGPP X86_MSDOS_DJGPP_GNU X86_MSDOS_GNU It really is reasonable to have a quadruple sometimes. architecture-operatingsystem-C runtime-toolset though C runtime doesn't very often. ?I don't want to open the whole can of worms, just need a name for DJGPP. DJGPP and MSDOS are fairly unambiguous here..but...Open Watcom is also a viable runtime/toolset.Maybe fit into the naming. X86_MSDOS_WATCOM X86_MSDOS_GNU X86_MSDOS_DJGPP Open Watcom really puts a monkey wrench into it.X86_OPENWATCOM is among the most ambiguous hypothetical names, because OpenWatcom targetsa bajillion x86 variants (and even made progress toward Alpha and PowerPC).Open Watcom is actively developed and targets at least: 16 bit real mode MS-DOS 32 bit protected mode MS-DOS 16 bit Windows 3.1 (enhanced mode?) 32 bit code in 16 bit Windows 32 bit protected mode Windows (NT) 32 bit protected mode OS/2 other OS/2 variants? 16 bit? Or only 16 bit? Novell Netware I believe! x86 Linux or just DJGPP MSDOS_OPENWATCOM or MSDOS_DJGPP MSDOS_OPENWATCOM MSDOS implies 32 bit x86. ? PERHAPS some more names should be settled for some actual practical targets that might come up soon, to guide the discussion? X86_SOLARIS ? AMD64_SOLARIS ? SPARC64_SOLARIS ? Solaris always has two toolsets though, ? Sun and GNU? AMD64_DARWIN ? PPC64_DARWIN ? (obsolete before it ever caught on?) PPC64_AIX ? (this exists as an OS, right?) AMD64_NT -- ambiguous ? AMD64_NT_GNU ? AMD64_NT_MINGNU ? -- this toolset is already out there AMD64_NT_CYGWIN ? -- This isn't likely, Cygwin is very x86-specific AMD64_LINUX ? IA64_VMS ? IA64_LINUX ? two or more toolsets, right? GCC, Intel, SGI? IA64_LINUX_GNU ? IA64_LINUX_INTEL ? IA64_LINUX_SGI ? I'm skeptical there will be any IA64 ports, but I expect there will be AMD64. (I have seen IA64 for around $500 on eBay, tempting, for a port..) And consider LLVM in the naming exercise? As a toolset, right? AMD64_LINUX_GNU AMD64_LINUX_LLVM ? Are people wedded to "I386" instead of "x86"? To indicate clearly it isn't 286 and the ilk? I am somewhat wedded to "AMD64", though other names include "X64" and X86-64. X86-64 is too long, and the dash would probably go away, X8664, getting to be a long run of gibberish numbers. X64...is that less than X86? :) AMD64 is the value of %PROCESSOR_ARCHITECTURE% on NT and the install directory name on the pre-Vista CDs (along with i386), and it is available as an #ifdef (_AMD64_, _M_AMD64). Intel was calling it EM64T, but that gone away in favor of Intel64, to not be confused with IA64, which is Itanium, stands for Intel Architecture 64 hm... Intel64 != Intel Architecture 64. Admittedly they were kind of stuck. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Sun Mar 30 12:53:10 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 30 Mar 2008 12:53:10 +0200 Subject: [M3devel] target naming again..DJGPP? In-Reply-To: References: Message-ID: <1206874390.7023.16.camel@faramir.m3w.org> djgpp being piece of archaeology, it surely is interesting. Would be much better to invest time in 64bit ports these days, though :). dd On Sat, 2008-03-29 at 21:59 +0000, Jay wrote: > Does anyone else find the idea of a DJGPP port "interesting"? > > > I think I'll go ahead and do it. I know its useless but I find it > "interesting". > A few things will come before it. > > > The DJGPP runtime appears to have the adequate support for > alarm/setitimer, > so it /should/ be fairly easy. > I've already demonstrated to myself cooperating threading, and > alarm/setitimer > can be used for preemption, with the user posix threads. > The easy work I did pruning down NT386GNU's *.i3 file to a minimum > will be useful. > > > What to call it? > > > Some combination of > > > (x86 or 386 or i386) and (DOS or MSDOS) and (DJGPP and/or GNU)? > > > DJGPP > MSDOS > X86_MSDOS > X86_DJGPP > X86_MSDOS_DJGPP > X86_MSDOS_DJGPP_GNU > X86_MSDOS_GNU > > > It really is reasonable to have a quadruple sometimes. > architecture-operatingsystem-C runtime-toolset > though C runtime doesn't very often. > > > ?I don't want to open the whole can of worms, just need a name for > DJGPP. > > > DJGPP and MSDOS are fairly unambiguous here..but... > Open Watcom is also a viable runtime/toolset. > Maybe fit into the naming. > X86_MSDOS_WATCOM > X86_MSDOS_GNU > X86_MSDOS_DJGPP > > > Open Watcom really puts a monkey wrench into it. > X86_OPENWATCOM is among the most ambiguous hypothetical names, because > OpenWatcom targets > a bajillion x86 variants (and even made progress toward Alpha and > PowerPC). > Open Watcom is actively developed and targets at least: > 16 bit real mode MS-DOS > 32 bit protected mode MS-DOS > 16 bit Windows 3.1 (enhanced mode?) > 32 bit code in 16 bit Windows > 32 bit protected mode Windows (NT) > 32 bit protected mode OS/2 > other OS/2 variants? 16 bit? Or only 16 bit? > Novell Netware I believe! > x86 Linux > > or just > DJGPP > MSDOS_OPENWATCOM > or > > MSDOS_DJGPP > MSDOS_OPENWATCOM > > MSDOS implies 32 bit x86. ? > > > PERHAPS some more names should be settled for some actual practical > targets that might come up soon, to guide the discussion? > > X86_SOLARIS ? > AMD64_SOLARIS ? > SPARC64_SOLARIS ? > Solaris always has two toolsets though, ? Sun and GNU? > AMD64_DARWIN ? > PPC64_DARWIN ? (obsolete before it ever caught on?) > PPC64_AIX ? (this exists as an OS, right?) > AMD64_NT -- ambiguous ? > AMD64_NT_GNU ? > AMD64_NT_MINGNU ? -- this toolset is already out there > AMD64_NT_CYGWIN ? -- This isn't likely, Cygwin is very x86-specific > AMD64_LINUX ? > IA64_VMS ? > IA64_LINUX ? > two or more toolsets, right? GCC, Intel, SGI? > IA64_LINUX_GNU ? > IA64_LINUX_INTEL ? > IA64_LINUX_SGI ? > > > I'm skeptical there will be any IA64 ports, but I expect there will be > AMD64. > (I have seen IA64 for around $500 on eBay, tempting, for a port..) > > > And consider LLVM in the naming exercise? As a toolset, right? > AMD64_LINUX_GNU > AMD64_LINUX_LLVM ? > > > Are people wedded to "I386" instead of "x86"? > To indicate clearly it isn't 286 and the ilk? > > > I am somewhat wedded to "AMD64", though other names include "X64" and > X86-64. > X86-64 is too long, and the dash would probably go away, X8664, > getting to be a long run of gibberish numbers. > X64...is that less than X86? :) > AMD64 is the value of %PROCESSOR_ARCHITECTURE% on NT and the install > directory name on the pre-Vista CDs (along with i386), and it is > available as an #ifdef (_AMD64_, _M_AMD64). > > > Intel was calling it EM64T, but that gone away in favor of Intel64, to > not be confused with IA64, which is Itanium, stands for Intel > Architecture 64 hm... > Intel64 != Intel Architecture 64. > Admittedly they were kind of stuck. > > > - Jay > -- Dragi?a Duri? From jayk123 at hotmail.com Sun Mar 30 16:15:39 2008 From: jayk123 at hotmail.com (Jay) Date: Sun, 30 Mar 2008 14:15:39 +0000 Subject: [M3devel] target naming again..DJGPP? In-Reply-To: <1206874390.7023.16.camel@faramir.m3w.org> References: <1206874390.7023.16.camel@faramir.m3w.org> Message-ID: Right, I plan to do that too. Please let's also chose names for the following two targets: NTAMD64, AMD64_NT, AMD64_MSWIN, AMD64_WIN, etc. NTAMD64MINGNU, AMD64_NT_GNU, AMD64_MSWIN_GNU, AMD64_WIN_GCC, AMD64_NT_MINGWIN, MING64, MINGWIN64, etc. You can work NTAMD64GNU/AMD64_NT_CYGWIN into the scheme, but it is less likely to happen. Also Services for Unix, AMD64_NT_INTERIX, AMD64_INTERIX, AMD64_MSU, etc. Also consider the following likely targets: X86_SOLARIS, I386_SOLARIS, X86_SOLARIS X86_SOLARIS_SUN X86_SOLARIS_GNU AMD64_SOLARIS AMD64_SOLARIS_SUN (Sun tools. Do they exist?) AMD64_SOLARIS_GNU AMD64_DARWIN AMD64_FREEBSD, AMD64_FBSD, FREEBSDAMD64, FBSDAMD64 AMD64_OPENBSD AMD64_NETBSD and let me know which are actually "likely". I think there are a surprisingly large number of likely new targets -- at LEAST AMD64 for Windows and MacOSX, probably also some AMD64 BSD and maybe AMD64/x86 Solaris. It's easy to work on getting these to compile actually, they all share the same likely small set of problems, in a cross compile. I'll have enough hardware soon enough, though my current primary home machine is only 32 bit. We should also possibly think more about how cross compilation looks and works, since I think cross compilation is getting more common. Could be that it mostly already is worked out though. The main thing maybe is to have \cm3\bin\cm3.cmd, \cm3\bin\cm3, \cm3\bin\cm3cg that are actually cmd and sh that delegate off to \cm3\pkg\cm3\host\cm3 and \cm3\pkg\m3cc\host\target\cm3cg, usually via some very unambiguous parameter for the scripts' use or a very fast sniff for interactive use, possibly the scripts are actually host dependenent in their default, rather than ever sniffing, you know, they are like about two lines each, and actually host specific. cm3.cmd: setlocal set host=%1 set target=%2 if not defined target set host=NT386 if not defined target set target=NT386 call %~dp0\..\pkg\cm3\%host%\%target%\cm3 %* But then NT386MINGNU might actually ship a cm3 with different defaults. cm3: #!/bin/sh host=${1:-PPC_DARWIN} target=${2:-PPC_DARWIN} # how to get your own path? exec $0/../pkg/cm3/$host/$target/cm3 $@ m3cc: #!/bin/sh host=${1:-PPC_DARWIN} target=${2:-PPC_DARWIN} # how to get your own path? exec $0/../pkg/m3cc/$host/$target/cm3 $@ they don't necessarily delegate to the package store. It migth actually be \bin\cm3 \bin\host\target\cm3 \bin\cm3cg \bin\host\target\cm3cg or maybe just \bin\cm3 \bin\cm3cg \bin\target\cm3cg host might be implied by "\bin" or what is above it (nothing here, but could be something). Not delegating out to the package store has a possible advantage to being more "relocatable" into other file system contexts. I'm a fair amount interested in working out a solid well adhered to convention here. In particular, scripts should be able to depend on it so they can skip the wrapper. It should, perhaps, if I'm not crazy, be possible and obvious how to create a "very full" tree/distribution for use "anywhere". You know, in a file system shared by multiple host systems, or in a distribution that is super piggy on the download and maybe prunes itself down upon install. Or maybe one of those stub installers that downloads just what the user requests. It might also be by file name, cm3, vs. cm3-nt386.exe. On the other hand, this maybe isn't all that important. Furthermore..it'd be nice to work gcc and ld and headers/libs into this structure if possible. I like fewer larger pieces of functionality so I don't have to go and setup so many varying inconsistent things, and then it becomes possible to build any Modula-3 target on any host. I guess I should just get into doing my own builds and cross builds of gcc/ld/glibc and setup config files appropriately, though I don't think everything is licensed "appropriately" and/or it'd still be some "aggregation", since it isn't always GNU ld (for example on Darwin) or glibc (many examples -- Cygwin, Darwin, Solaris). - Jay > Subject: Re: [M3devel] target naming again..DJGPP?> From: dragisha at m3w.org> To: jayk123 at hotmail.com> CC: m3devel at elegosoft.com> Date: Sun, 30 Mar 2008 12:53:10 +0200> > djgpp being piece of archaeology, it surely is interesting. Would be> much better to invest time in 64bit ports these days, though :).> > dd> > On Sat, 2008-03-29 at 21:59 +0000, Jay wrote:> > Does anyone else find the idea of a DJGPP port "interesting"?> > > > > > I think I'll go ahead and do it. I know its useless but I find it> > "interesting".> > A few things will come before it.> > > > > > The DJGPP runtime appears to have the adequate support for> > alarm/setitimer,> > so it /should/ be fairly easy.> > I've already demonstrated to myself cooperating threading, and> > alarm/setitimer> > can be used for preemption, with the user posix threads.> > The easy work I did pruning down NT386GNU's *.i3 file to a minimum> > will be useful.> > > > > > What to call it?> > > > > > Some combination of> > > > > > (x86 or 386 or i386) and (DOS or MSDOS) and (DJGPP and/or GNU)?> > > > > > DJGPP > > MSDOS > > X86_MSDOS > > X86_DJGPP > > X86_MSDOS_DJGPP > > X86_MSDOS_DJGPP_GNU > > X86_MSDOS_GNU > > > > > > It really is reasonable to have a quadruple sometimes.> > architecture-operatingsystem-C runtime-toolset > > though C runtime doesn't very often.> > > > > > ?I don't want to open the whole can of worms, just need a name for> > DJGPP.> > > > > > DJGPP and MSDOS are fairly unambiguous here..but...> > Open Watcom is also a viable runtime/toolset.> > Maybe fit into the naming.> > X86_MSDOS_WATCOM > > X86_MSDOS_GNU > > X86_MSDOS_DJGPP > > > > > > Open Watcom really puts a monkey wrench into it.> > X86_OPENWATCOM is among the most ambiguous hypothetical names, because> > OpenWatcom targets> > a bajillion x86 variants (and even made progress toward Alpha and> > PowerPC).> > Open Watcom is actively developed and targets at least:> > 16 bit real mode MS-DOS> > 32 bit protected mode MS-DOS > > 16 bit Windows 3.1 (enhanced mode?) > > 32 bit code in 16 bit Windows > > 32 bit protected mode Windows (NT) > > 32 bit protected mode OS/2 > > other OS/2 variants? 16 bit? Or only 16 bit? > > Novell Netware I believe! > > x86 Linux> > > > or just> > DJGPP> > MSDOS_OPENWATCOM > > or> > > > MSDOS_DJGPP> > MSDOS_OPENWATCOM > > > > MSDOS implies 32 bit x86. ?> > > > > > PERHAPS some more names should be settled for some actual practical> > targets that might come up soon, to guide the discussion? > > > > X86_SOLARIS ? > > AMD64_SOLARIS ? > > SPARC64_SOLARIS ?> > Solaris always has two toolsets though, ? Sun and GNU?> > AMD64_DARWIN ? > > PPC64_DARWIN ? (obsolete before it ever caught on?) > > PPC64_AIX ? (this exists as an OS, right?) > > AMD64_NT -- ambiguous ? > > AMD64_NT_GNU ? > > AMD64_NT_MINGNU ? -- this toolset is already out there > > AMD64_NT_CYGWIN ? -- This isn't likely, Cygwin is very x86-specific > > AMD64_LINUX ? > > IA64_VMS ?> > IA64_LINUX ?> > two or more toolsets, right? GCC, Intel, SGI?> > IA64_LINUX_GNU ?> > IA64_LINUX_INTEL ?> > IA64_LINUX_SGI ?> > > > > > I'm skeptical there will be any IA64 ports, but I expect there will be> > AMD64.> > (I have seen IA64 for around $500 on eBay, tempting, for a port..)> > > > > > And consider LLVM in the naming exercise? As a toolset, right?> > AMD64_LINUX_GNU > > AMD64_LINUX_LLVM ? > > > > > > Are people wedded to "I386" instead of "x86"?> > To indicate clearly it isn't 286 and the ilk?> > > > > > I am somewhat wedded to "AMD64", though other names include "X64" and> > X86-64.> > X86-64 is too long, and the dash would probably go away, X8664,> > getting to be a long run of gibberish numbers.> > X64...is that less than X86? :)> > AMD64 is the value of %PROCESSOR_ARCHITECTURE% on NT and the install> > directory name on the pre-Vista CDs (along with i386), and it is> > available as an #ifdef (_AMD64_, _M_AMD64).> > > > > > Intel was calling it EM64T, but that gone away in favor of Intel64, to> > not be confused with IA64, which is Itanium, stands for Intel> > Architecture 64 hm...> > Intel64 != Intel Architecture 64.> > Admittedly they were kind of stuck.> > > > > > - Jay> > > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Sun Mar 30 16:34:27 2008 From: jayk123 at hotmail.com (Jay) Date: Sun, 30 Mar 2008 14:34:27 +0000 Subject: [M3devel] correct Alpha/OSF headers? Message-ID: correct Alpha/OSF headers? Can anyone confirm stuff like: http://www.nanobit.net/quake3_doxy/alpha_2osf_2limits_8h-source.html http://www.nanobit.net/quake3_doxy/alpha_2osf_2setjmp_8h-source.html Does anyone have a working machine? That I can ssh into? (or working Alpha anything else? Linux/FreeBSD/OpenBSD/FreeBSD/NT (I do have NT Alpha/Mips/PowerPC tools :) ?) oh but NT Alpha was only 32 bit.) - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Sun Mar 30 16:49:17 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 30 Mar 2008 10:49:17 -0400 Subject: [M3devel] target naming again..DJGPP? In-Reply-To: References: <1206874390.7023.16.camel@faramir.m3w.org> Message-ID: I would prefer to use x86_64 instead of AMD64 to describe these platforms as there is now a degree of commonality between AMD and Intel's implementations. I have had it in the the back of my queue to do/complete (someone started it at some point) a Linux x86_64 port sometime. On Mar 30, 2008, at 10:15 AM, Jay wrote: > Right, I plan to do that too. > Please let's also chose names for the following two targets: > > NTAMD64, AMD64_NT, AMD64_MSWIN, AMD64_WIN, etc. > NTAMD64MINGNU, AMD64_NT_GNU, AMD64_MSWIN_GNU, AMD64_WIN_GCC, > AMD64_NT_MINGWIN, MING64, MINGWIN64, etc. > > You can work NTAMD64GNU/AMD64_NT_CYGWIN into the scheme, but it is > less likely to happen. > Also Services for Unix, AMD64_NT_INTERIX, AMD64_INTERIX, AMD64_MSU, > etc. > > Also consider the following likely targets: > X86_SOLARIS, I386_SOLARIS, X86_SOLARIS > X86_SOLARIS_SUN > X86_SOLARIS_GNU > AMD64_SOLARIS > AMD64_SOLARIS_SUN (Sun tools. Do they exist?) > AMD64_SOLARIS_GNU > AMD64_DARWIN > AMD64_FREEBSD, AMD64_FBSD, FREEBSDAMD64, FBSDAMD64 > AMD64_OPENBSD > AMD64_NETBSD > > > and let me know which are actually "likely". > I think there are a surprisingly large number of likely new targets > -- at LEAST AMD64 for Windows and MacOSX, probably also some AMD64 > BSD and maybe AMD64/x86 Solaris. > > It's easy to work on getting these to compile actually, they all > share the same likely small set of problems, in a cross compile. > I'll have enough hardware soon enough, though my current primary > home machine is only 32 bit. > > We should also possibly think more about how cross compilation looks > and works, since I think cross compilation is getting more common. > Could be that it mostly already is worked out though. > The main thing maybe is to have \cm3\bin\cm3.cmd, \cm3\bin\cm3, > \cm3\bin\cm3cg that are actually cmd and sh that delegate off to > \cm3\pkg\cm3\host\cm3 and \cm3\pkg\m3cc\host\target\cm3cg, usually > via some very unambiguous parameter for the scripts' use or a very > fast sniff for interactive use, possibly the scripts are actually > host dependenent in their default, rather than ever sniffing, you > know, they are like about two lines each, and actually host specific. > > cm3.cmd: > setlocal > set host=%1 > set target=%2 > if not defined target set host=NT386 > if not defined target set target=NT386 > call %~dp0\..\pkg\cm3\%host%\%target%\cm3 %* > > But then NT386MINGNU might actually ship a cm3 with different > defaults. > > cm3: > #!/bin/sh > host=${1:-PPC_DARWIN} > target=${2:-PPC_DARWIN} > # how to get your own path? > exec $0/../pkg/cm3/$host/$target/cm3 $@ > > m3cc: > #!/bin/sh > host=${1:-PPC_DARWIN} > target=${2:-PPC_DARWIN} > # how to get your own path? > exec $0/../pkg/m3cc/$host/$target/cm3 $@ > > they don't necessarily delegate to the package store. > It migth actually be > \bin\cm3 > \bin\host\target\cm3 > \bin\cm3cg > \bin\host\target\cm3cg > > or maybe just > > \bin\cm3 > \bin\cm3cg > \bin\target\cm3cg > > host might be implied by "\bin" or what is above it (nothing here, > but could be something). > > Not delegating out to the package store has a possible advantage to > being more "relocatable" into other file system contexts. > > I'm a fair amount interested in working out a solid well adhered to > convention here. > In particular, scripts should be able to depend on it so they can > skip the wrapper. > It should, perhaps, if I'm not crazy, be possible and obvious how to > create a "very full" tree/distribution for use "anywhere". > You know, in a file system shared by multiple host systems, or in a > distribution that is super piggy on the download and maybe prunes > itself down upon install. > Or maybe one of those stub installers that downloads just what the > user requests. > > It might also be by file name, cm3, vs. cm3-nt386.exe. > > On the other hand, this maybe isn't all that important. > > Furthermore..it'd be nice to work gcc and ld and headers/libs into > this structure if possible. I like fewer larger pieces of > functionality so I don't have to go and setup so many varying > inconsistent things, and then it becomes possible to build any > Modula-3 target on any host. I guess I should just get into doing my > own builds and cross builds of gcc/ld/glibc and setup config files > appropriately, though I don't think everything is licensed > "appropriately" and/or it'd still be some "aggregation", since it > isn't always GNU ld (for example on Darwin) or glibc (many examples > -- Cygwin, Darwin, Solaris). > > > - Jay > > > > > Subject: Re: [M3devel] target naming again..DJGPP? > > From: dragisha at m3w.org > > To: jayk123 at hotmail.com > > CC: m3devel at elegosoft.com > > Date: Sun, 30 Mar 2008 12:53:10 +0200 > > > > djgpp being piece of archaeology, it surely is interesting. Would be > > much better to invest time in 64bit ports these days, though :). > > > > dd > > > > On Sat, 2008-03-29 at 21:59 +0000, Jay wrote: > > > Does anyone else find the idea of a DJGPP port "interesting"? > > > > > > > > > I think I'll go ahead and do it. I know its useless but I find it > > > "interesting". > > > A few things will come before it. > > > > > > > > > The DJGPP runtime appears to have the adequate support for > > > alarm/setitimer, > > > so it /should/ be fairly easy. > > > I've already demonstrated to myself cooperating threading, and > > > alarm/setitimer > > > can be used for preemption, with the user posix threads. > > > The easy work I did pruning down NT386GNU's *.i3 file to a minimum > > > will be useful. > > > > > > > > > What to call it? > > > > > > > > > Some combination of > > > > > > > > > (x86 or 386 or i386) and (DOS or MSDOS) and (DJGPP and/or GNU)? > > > > > > > > > DJGPP > > > MSDOS > > > X86_MSDOS > > > X86_DJGPP > > > X86_MSDOS_DJGPP > > > X86_MSDOS_DJGPP_GNU > > > X86_MSDOS_GNU > > > > > > > > > It really is reasonable to have a quadruple sometimes. > > > architecture-operatingsystem-C runtime-toolset > > > though C runtime doesn't very often. > > > > > > > > > ?I don't want to open the whole can of worms, just need a name for > > > DJGPP. > > > > > > > > > DJGPP and MSDOS are fairly unambiguous here..but... > > > Open Watcom is also a viable runtime/toolset. > > > Maybe fit into the naming. > > > X86_MSDOS_WATCOM > > > X86_MSDOS_GNU > > > X86_MSDOS_DJGPP > > > > > > > > > Open Watcom really puts a monkey wrench into it. > > > X86_OPENWATCOM is among the most ambiguous hypothetical names, > because > > > OpenWatcom targets > > > a bajillion x86 variants (and even made progress toward Alpha and > > > PowerPC). > > > Open Watcom is actively developed and targets at least: > > > 16 bit real mode MS-DOS > > > 32 bit protected mode MS-DOS > > > 16 bit Windows 3.1 (enhanced mode?) > > > 32 bit code in 16 bit Windows > > > 32 bit protected mode Windows (NT) > > > 32 bit protected mode OS/2 > > > other OS/2 variants? 16 bit? Or only 16 bit? > > > Novell Netware I believe! > > > x86 Linux > > > > > > or just > > > DJGPP > > > MSDOS_OPENWATCOM > > > or > > > > > > MSDOS_DJGPP > > > MSDOS_OPENWATCOM > > > > > > MSDOS implies 32 bit x86. ? > > > > > > > > > PERHAPS some more names should be settled for some actual > practical > > > targets that might come up soon, to guide the discussion? > > > > > > X86_SOLARIS ? > > > AMD64_SOLARIS ? > > > SPARC64_SOLARIS ? > > > Solaris always has two toolsets though, ? Sun and GNU? > > > AMD64_DARWIN ? > > > PPC64_DARWIN ? (obsolete before it ever caught on?) > > > PPC64_AIX ? (this exists as an OS, right?) > > > AMD64_NT -- ambiguous ? > > > AMD64_NT_GNU ? > > > AMD64_NT_MINGNU ? -- this toolset is already out there > > > AMD64_NT_CYGWIN ? -- This isn't likely, Cygwin is very x86- > specific > > > AMD64_LINUX ? > > > IA64_VMS ? > > > IA64_LINUX ? > > > two or more toolsets, right? GCC, Intel, SGI? > > > IA64_LINUX_GNU ? > > > IA64_LINUX_INTEL ? > > > IA64_LINUX_SGI ? > > > > > > > > > I'm skeptical there will be any IA64 ports, but I expect there > will be > > > AMD64. > > > (I have seen IA64 for around $500 on eBay, tempting, for a port..) > > > > > > > > > And consider LLVM in the naming exercise? As a toolset, right? > > > AMD64_LINUX_GNU > > > AMD64_LINUX_LLVM ? > > > > > > > > > Are people wedded to "I386" instead of "x86"? > > > To indicate clearly it isn't 286 and the ilk? > > > > > > > > > I am somewhat wedded to "AMD64", though other names include > "X64" and > > > X86-64. > > > X86-64 is too long, and the dash would probably go away, X8664, > > > getting to be a long run of gibberish numbers. > > > X64...is that less than X86? :) > > > AMD64 is the value of %PROCESSOR_ARCHITECTURE% on NT and the > install > > > directory name on the pre-Vista CDs (along with i386), and it is > > > available as an #ifdef (_AMD64_, _M_AMD64). > > > > > > > > > Intel was calling it EM64T, but that gone away in favor of > Intel64, to > > > not be confused with IA64, which is Itanium, stands for Intel > > > Architecture 64 hm... > > > Intel64 != Intel Architecture 64. > > > Admittedly they were kind of stuck. > > > > > > > > > - Jay > > > > > -- > > Dragi?a Duri? > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Sun Mar 30 16:51:43 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 30 Mar 2008 10:51:43 -0400 Subject: [M3devel] correct Alpha/OSF headers? In-Reply-To: References: Message-ID: <2CEE2E1D-A942-4BA6-B7BF-781E10F1BFD1@cs.purdue.edu> I have a ALPHA/OSF machine that's been down since it got hacked a few years ago. It was happily running Modula-3 at the time. Should not be too much work to resurrect, just time, and I don't have a lot of that. The ALPHA_OSF stuff in CM3 and PM3 is probably still accurate. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 30, 2008, at 10:34 AM, Jay wrote: > correct Alpha/OSF headers? > Can anyone confirm stuff like: > > http://www.nanobit.net/quake3_doxy/alpha_2osf_2limits_8h-source.html > http://www.nanobit.net/quake3_doxy/alpha_2osf_2setjmp_8h-source.html > > Does anyone have a working machine? > That I can ssh into? > > (or working Alpha anything else? Linux/FreeBSD/OpenBSD/FreeBSD/NT (I > do have NT Alpha/Mips/PowerPC tools :) ?) oh but NT Alpha was only > 32 bit.) > > - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Sun Mar 30 17:16:36 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 30 Mar 2008 17:16:36 +0200 Subject: [M3devel] target naming again..DJGPP? In-Reply-To: References: <1206874390.7023.16.camel@faramir.m3w.org> Message-ID: <1206890197.7023.20.camel@faramir.m3w.org> Is that non-finished port available somehwere? dd On Sun, 2008-03-30 at 10:49 -0400, Tony Hosking wrote: > I would prefer to use x86_64 instead of AMD64 to describe these > platforms as there is now a degree of commonality between AMD and > Intel's implementations. I have had it in the the back of my queue to > do/complete (someone started it at some point) a Linux x86_64 port > sometime. > > On Mar 30, 2008, at 10:15 AM, Jay wrote: > > Right, I plan to do that too. > > Please let's also chose names for the following two targets: > > > > NTAMD64, AMD64_NT, AMD64_MSWIN, AMD64_WIN, etc. > > NTAMD64MINGNU, AMD64_NT_GNU, AMD64_MSWIN_GNU, AMD64_WIN_GCC, > > AMD64_NT_MINGWIN, MING64, MINGWIN64, etc. > > > > You can work NTAMD64GNU/AMD64_NT_CYGWIN into the scheme, but it is > > less likely to happen. > > Also Services for Unix, AMD64_NT_INTERIX, AMD64_INTERIX, AMD64_MSU, > > etc. > > > > Also consider the following likely targets: > > X86_SOLARIS, I386_SOLARIS, X86_SOLARIS > > X86_SOLARIS_SUN > > X86_SOLARIS_GNU > > AMD64_SOLARIS > > AMD64_SOLARIS_SUN (Sun tools. Do they exist?) > > AMD64_SOLARIS_GNU > > AMD64_DARWIN > > AMD64_FREEBSD, AMD64_FBSD, FREEBSDAMD64, FBSDAMD64 > > AMD64_OPENBSD > > AMD64_NETBSD > > > > > > and let me know which are actually "likely". > > I think there are a surprisingly large number of likely new targets > > -- at LEAST AMD64 for Windows and MacOSX, probably also some AMD64 > > BSD and maybe AMD64/x86 Solaris. > > > > It's easy to work on getting these to compile actually, they all > > share the same likely small set of problems, in a cross compile. > > I'll have enough hardware soon enough, though my current primary > > home machine is only 32 bit. > > > > We should also possibly think more about how cross compilation looks > > and works, since I think cross compilation is getting more common. > > Could be that it mostly already is worked out though. > > The main thing maybe is to have \cm3\bin\cm3.cmd, \cm3\bin\cm3, \cm3 > > \bin\cm3cg that are actually cmd and sh that delegate off to \cm3 > > \pkg\cm3\host\cm3 and \cm3\pkg\m3cc\host\target\cm3cg, usually via > > some very unambiguous parameter for the scripts' use or a very fast > > sniff for interactive use, possibly the scripts are actually host > > dependenent in their default, rather than ever sniffing, you know, > > they are like about two lines each, and actually host specific. > > > > cm3.cmd: > > setlocal > > set host=%1 > > set target=%2 > > if not defined target set host=NT386 > > if not defined target set target=NT386 > > call %~dp0\..\pkg\cm3\%host%\%target%\cm3 %* > > > > But then NT386MINGNU might actually ship a cm3 with different > > defaults. > > > > cm3: > > #!/bin/sh > > host=${1:-PPC_DARWIN} > > target=${2:-PPC_DARWIN} > > # how to get your own path? > > exec $0/../pkg/cm3/$host/$target/cm3 $@ > > > > m3cc: > > #!/bin/sh > > host=${1:-PPC_DARWIN} > > target=${2:-PPC_DARWIN} > > # how to get your own path? > > exec $0/../pkg/m3cc/$host/$target/cm3 $@ > > > > they don't necessarily delegate to the package store. > > It migth actually be > > \bin\cm3 > > \bin\host\target\cm3 > > \bin\cm3cg > > \bin\host\target\cm3cg > > > > or maybe just > > > > \bin\cm3 > > \bin\cm3cg > > \bin\target\cm3cg > > > > host might be implied by "\bin" or what is above it (nothing here, > > but could be something). > > > > Not delegating out to the package store has a possible advantage to > > being more "relocatable" into other file system contexts. > > > > I'm a fair amount interested in working out a solid well adhered to > > convention here. > > In particular, scripts should be able to depend on it so they can > > skip the wrapper. > > It should, perhaps, if I'm not crazy, be possible and obvious how to > > create a "very full" tree/distribution for use "anywhere". > > You know, in a file system shared by multiple host systems, or in a > > distribution that is super piggy on the download and maybe prunes > > itself down upon install. > > Or maybe one of those stub installers that downloads just what the > > user requests. > > > > It might also be by file name, cm3, vs. cm3-nt386.exe. > > > > On the other hand, this maybe isn't all that important. > > > > Furthermore..it'd be nice to work gcc and ld and headers/libs into > > this structure if possible. I like fewer larger pieces of > > functionality so I don't have to go and setup so many varying > > inconsistent things, and then it becomes possible to build any > > Modula-3 target on any host. I guess I should just get into doing my > > own builds and cross builds of gcc/ld/glibc and setup config files > > appropriately, though I don't think everything is licensed > > "appropriately" and/or it'd still be some "aggregation", since it > > isn't always GNU ld (for example on Darwin) or glibc (many examples > > -- Cygwin, Darwin, Solaris). > > > > > > - Jay > > > > > > > > ____________________________________________________________________ > > > > > Subject: Re: [M3devel] target naming again..DJGPP? > > > From: dragisha at m3w.org > > > To: jayk123 at hotmail.com > > > CC: m3devel at elegosoft.com > > > Date: Sun, 30 Mar 2008 12:53:10 +0200 > > > > > > djgpp being piece of archaeology, it surely is interesting. Would > > be > > > much better to invest time in 64bit ports these days, though :). > > > > > > dd > > > > > > On Sat, 2008-03-29 at 21:59 +0000, Jay wrote: > > > > Does anyone else find the idea of a DJGPP port "interesting"? > > > > > > > > > > > > I think I'll go ahead and do it. I know its useless but I find > > it > > > > "interesting". > > > > A few things will come before it. > > > > > > > > > > > > The DJGPP runtime appears to have the adequate support for > > > > alarm/setitimer, > > > > so it /should/ be fairly easy. > > > > I've already demonstrated to myself cooperating threading, and > > > > alarm/setitimer > > > > can be used for preemption, with the user posix threads. > > > > The easy work I did pruning down NT386GNU's *.i3 file to a > > minimum > > > > will be useful. > > > > > > > > > > > > What to call it? > > > > > > > > > > > > Some combination of > > > > > > > > > > > > (x86 or 386 or i386) and (DOS or MSDOS) and (DJGPP and/or GNU)? > > > > > > > > > > > > DJGPP > > > > MSDOS > > > > X86_MSDOS > > > > X86_DJGPP > > > > X86_MSDOS_DJGPP > > > > X86_MSDOS_DJGPP_GNU > > > > X86_MSDOS_GNU > > > > > > > > > > > > It really is reasonable to have a quadruple sometimes. > > > > architecture-operatingsystem-C runtime-toolset > > > > though C runtime doesn't very often. > > > > > > > > > > > > ?I don't want to open the whole can of worms, just need a name > > for > > > > DJGPP. > > > > > > > > > > > > DJGPP and MSDOS are fairly unambiguous here..but... > > > > Open Watcom is also a viable runtime/toolset. > > > > Maybe fit into the naming. > > > > X86_MSDOS_WATCOM > > > > X86_MSDOS_GNU > > > > X86_MSDOS_DJGPP > > > > > > > > > > > > Open Watcom really puts a monkey wrench into it. > > > > X86_OPENWATCOM is among the most ambiguous hypothetical names, > > because > > > > OpenWatcom targets > > > > a bajillion x86 variants (and even made progress toward Alpha > > and > > > > PowerPC). > > > > Open Watcom is actively developed and targets at least: > > > > 16 bit real mode MS-DOS > > > > 32 bit protected mode MS-DOS > > > > 16 bit Windows 3.1 (enhanced mode?) > > > > 32 bit code in 16 bit Windows > > > > 32 bit protected mode Windows (NT) > > > > 32 bit protected mode OS/2 > > > > other OS/2 variants? 16 bit? Or only 16 bit? > > > > Novell Netware I believe! > > > > x86 Linux > > > > > > > > or just > > > > DJGPP > > > > MSDOS_OPENWATCOM > > > > or > > > > > > > > MSDOS_DJGPP > > > > MSDOS_OPENWATCOM > > > > > > > > MSDOS implies 32 bit x86. ? > > > > > > > > > > > > PERHAPS some more names should be settled for some actual > > practical > > > > targets that might come up soon, to guide the discussion? > > > > > > > > X86_SOLARIS ? > > > > AMD64_SOLARIS ? > > > > SPARC64_SOLARIS ? > > > > Solaris always has two toolsets though, ? Sun and GNU? > > > > AMD64_DARWIN ? > > > > PPC64_DARWIN ? (obsolete before it ever caught on?) > > > > PPC64_AIX ? (this exists as an OS, right?) > > > > AMD64_NT -- ambiguous ? > > > > AMD64_NT_GNU ? > > > > AMD64_NT_MINGNU ? -- this toolset is already out there > > > > AMD64_NT_CYGWIN ? -- This isn't likely, Cygwin is very > > x86-specific > > > > AMD64_LINUX ? > > > > IA64_VMS ? > > > > IA64_LINUX ? > > > > two or more toolsets, right? GCC, Intel, SGI? > > > > IA64_LINUX_GNU ? > > > > IA64_LINUX_INTEL ? > > > > IA64_LINUX_SGI ? > > > > > > > > > > > > I'm skeptical there will be any IA64 ports, but I expect there > > will be > > > > AMD64. > > > > (I have seen IA64 for around $500 on eBay, tempting, for a > > port..) > > > > > > > > > > > > And consider LLVM in the naming exercise? As a toolset, right? > > > > AMD64_LINUX_GNU > > > > AMD64_LINUX_LLVM ? > > > > > > > > > > > > Are people wedded to "I386" instead of "x86"? > > > > To indicate clearly it isn't 286 and the ilk? > > > > > > > > > > > > I am somewhat wedded to "AMD64", though other names include > > "X64" and > > > > X86-64. > > > > X86-64 is too long, and the dash would probably go away, X8664, > > > > getting to be a long run of gibberish numbers. > > > > X64...is that less than X86? :) > > > > AMD64 is the value of %PROCESSOR_ARCHITECTURE% on NT and the > > install > > > > directory name on the pre-Vista CDs (along with i386), and it is > > > > available as an #ifdef (_AMD64_, _M_AMD64). > > > > > > > > > > > > Intel was calling it EM64T, but that gone away in favor of > > Intel64, to > > > > not be confused with IA64, which is Itanium, stands for Intel > > > > Architecture 64 hm... > > > > Intel64 != Intel Architecture 64. > > > > Admittedly they were kind of stuck. > > > > > > > > > > > > - Jay > > > > > > > -- > > > Dragi?a Duri? > > > > > > > > -- Dragi?a Duri? From jayk123 at hotmail.com Sun Mar 30 17:38:28 2008 From: jayk123 at hotmail.com (Jay) Date: Sun, 30 Mar 2008 15:38:28 +0000 Subject: [M3devel] target naming again..DJGPP? In-Reply-To: References: <1206874390.7023.16.camel@faramir.m3w.org> Message-ID: The environment variable is still "AMD64" on Windows machines with Intel CPUs. It can't change, for compatibility. The headers mention AMD64 repeatedly. winnt.h: #if defined(_M_MRX000) || defined(_M_ALPHA) || defined(_M_PPC) || defined(_M_IA64) || defined(_M_AMD64)#define UNALIGNED __unaligned#if defined(_WIN64)#define UNALIGNED64 __unaligned#else#define UNALIGNED64#endif#else#define UNALIGNED#define UNALIGNED64#endif #if defined(_AMD64_)#define PROBE_ALIGNMENT( _s ) TYPE_ALIGNMENT( DWORD )#elif defined(_IA64_)#define PROBE_ALIGNMENT( _s ) (TYPE_ALIGNMENT( _s ) > TYPE_ALIGNMENT( DWORD ) ? \... #ifndef SYSTEM_CACHE_ALIGNMENT_SIZE#if defined(_AMD64_) || defined(_X86_)#define SYSTEM_CACHE_ALIGNMENT_SIZE 64#else#define SYSTEM_CACHE_ALIGNMENT_SIZE 128#endif#endif excpt.h: #elif defined(_M_AMD64) /* * Declarations to keep AMD64 compiler happy */struct _EXCEPTION_RECORD;struct _CONTEXT; #endif This oddity though, winnt.h: #define PROCESSOR_INTEL_386 386#define PROCESSOR_INTEL_486 486#define PROCESSOR_INTEL_PENTIUM 586#define PROCESSOR_INTEL_IA64 2200#define PROCESSOR_AMD_X8664 8664#define PROCESSOR_MIPS_R4000 4000 // incl R4101 & R3910 for Windows CE and: #define PROCESSOR_ARCHITECTURE_INTEL 0#define PROCESSOR_ARCHITECTURE_MIPS 1#define PROCESSOR_ARCHITECTURE_ALPHA 2#define PROCESSOR_ARCHITECTURE_PPC 3#define PROCESSOR_ARCHITECTURE_SHX 4#define PROCESSOR_ARCHITECTURE_ARM 5#define PROCESSOR_ARCHITECTURE_IA64 6#define PROCESSOR_ARCHITECTURE_ALPHA64 7#define PROCESSOR_ARCHITECTURE_MSIL 8#define PROCESSOR_ARCHITECTURE_AMD64 9 - Jay CC: dragisha at m3w.org; m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] target naming again..DJGPP?Date: Sun, 30 Mar 2008 10:49:17 -0400 I would prefer to use x86_64 instead of AMD64 to describe these platforms as there is now a degree of commonality between AMD and Intel's implementations. I have had it in the the back of my queue to do/complete (someone started it at some point) a Linux x86_64 port sometime. On Mar 30, 2008, at 10:15 AM, Jay wrote: Right, I plan to do that too.Please let's also chose names for the following two targets: NTAMD64, AMD64_NT, AMD64_MSWIN, AMD64_WIN, etc. NTAMD64MINGNU, AMD64_NT_GNU, AMD64_MSWIN_GNU, AMD64_WIN_GCC, AMD64_NT_MINGWIN, MING64, MINGWIN64, etc. You can work NTAMD64GNU/AMD64_NT_CYGWIN into the scheme, but it is less likely to happen.Also Services for Unix, AMD64_NT_INTERIX, AMD64_INTERIX, AMD64_MSU, etc. Also consider the following likely targets: X86_SOLARIS, I386_SOLARIS, X86_SOLARIS X86_SOLARIS_SUN X86_SOLARIS_GNU AMD64_SOLARIS AMD64_SOLARIS_SUN (Sun tools. Do they exist?) AMD64_SOLARIS_GNU AMD64_DARWIN AMD64_FREEBSD, AMD64_FBSD, FREEBSDAMD64, FBSDAMD64 AMD64_OPENBSD AMD64_NETBSD and let me know which are actually "likely".I think there are a surprisingly large number of likely new targets -- at LEAST AMD64 for Windows and MacOSX, probably also some AMD64 BSD and maybe AMD64/x86 Solaris. It's easy to work on getting these to compile actually, they all share the same likely small set of problems, in a cross compile.I'll have enough hardware soon enough, though my current primary home machine is only 32 bit. We should also possibly think more about how cross compilation looks and works, since I think cross compilation is getting more common.Could be that it mostly already is worked out though.The main thing maybe is to have \cm3\bin\cm3.cmd, \cm3\bin\cm3, \cm3\bin\cm3cg that are actually cmd and sh that delegate off to \cm3\pkg\cm3\host\cm3 and \cm3\pkg\m3cc\host\target\cm3cg, usually via some very unambiguous parameter for the scripts' use or a very fast sniff for interactive use, possibly the scripts are actually host dependenent in their default, rather than ever sniffing, you know, they are like about two lines each, and actually host specific. cm3.cmd:setlocalset host=%1set target=%2if not defined target set host=NT386if not defined target set target=NT386call %~dp0\..\pkg\cm3\%host%\%target%\cm3 %* But then NT386MINGNU might actually ship a cm3 with different defaults. cm3:#!/bin/shhost=${1:-PPC_DARWIN}target=${2:-PPC_DARWIN}# how to get your own path?exec $0/../pkg/cm3/$host/$target/cm3 $@ m3cc:#!/bin/shhost=${1:-PPC_DARWIN}target=${2:-PPC_DARWIN}# how to get your own path?exec $0/../pkg/m3cc/$host/$target/cm3 $@ they don't necessarily delegate to the package store.It migth actually be \bin\cm3 \bin\host\target\cm3 \bin\cm3cg \bin\host\target\cm3cg or maybe just \bin\cm3 \bin\cm3cg \bin\target\cm3cg host might be implied by "\bin" or what is above it (nothing here, but could be something). Not delegating out to the package store has a possible advantage to being more "relocatable" into other file system contexts. I'm a fair amount interested in working out a solid well adhered to convention here.In particular, scripts should be able to depend on it so they can skip the wrapper.It should, perhaps, if I'm not crazy, be possible and obvious how to create a "very full" tree/distribution for use "anywhere".You know, in a file system shared by multiple host systems, or in a distribution that is super piggy on the download and maybe prunes itself down upon install.Or maybe one of those stub installers that downloads just what the user requests. It might also be by file name, cm3, vs. cm3-nt386.exe. On the other hand, this maybe isn't all that important. Furthermore..it'd be nice to work gcc and ld and headers/libs into this structure if possible. I like fewer larger pieces of functionality so I don't have to go and setup so many varying inconsistent things, and then it becomes possible to build any Modula-3 target on any host. I guess I should just get into doing my own builds and cross builds of gcc/ld/glibc and setup config files appropriately, though I don't think everything is licensed "appropriately" and/or it'd still be some "aggregation", since it isn't always GNU ld (for example on Darwin) or glibc (many examples -- Cygwin, Darwin, Solaris). - Jay > Subject: Re: [M3devel] target naming again..DJGPP?> From: dragisha at m3w.org> To: jayk123 at hotmail.com> CC: m3devel at elegosoft.com> Date: Sun, 30 Mar 2008 12:53:10 +0200> > djgpp being piece of archaeology, it surely is interesting. Would be> much better to invest time in 64bit ports these days, though :).> > dd> > On Sat, 2008-03-29 at 21:59 +0000, Jay wrote:> > Does anyone else find the idea of a DJGPP port "interesting"?> > > > > > I think I'll go ahead and do it. I know its useless but I find it> > "interesting".> > A few things will come before it.> > > > > > The DJGPP runtime appears to have the adequate support for> > alarm/setitimer,> > so it /should/ be fairly easy.> > I've already demonstrated to myself cooperating threading, and> > alarm/setitimer> > can be used for preemption, with the user posix threads.> > The easy work I did pruning down NT386GNU's *.i3 file to a minimum> > will be useful.> > > > > > What to call it?> > > > > > Some combination of> > > > > > (x86 or 386 or i386) and (DOS or MSDOS) and (DJGPP and/or GNU)?> > > > > > DJGPP > > MSDOS > > X86_MSDOS > > X86_DJGPP > > X86_MSDOS_DJGPP > > X86_MSDOS_DJGPP_GNU > > X86_MSDOS_GNU > > > > > > It really is reasonable to have a quadruple sometimes.> > architecture-operatingsystem-C runtime-toolset > > though C runtime doesn't very often.> > > > > > ?I don't want to open the whole can of worms, just need a name for> > DJGPP.> > > > > > DJGPP and MSDOS are fairly unambiguous here..but...> > Open Watcom is also a viable runtime/toolset.> > Maybe fit into the naming.> > X86_MSDOS_WATCOM > > X86_MSDOS_GNU > > X86_MSDOS_DJGPP > > > > > > Open Watcom really puts a monkey wrench into it.> > X86_OPENWATCOM is among the most ambiguous hypothetical names, because> > OpenWatcom targets> > a bajillion x86 variants (and even made progress toward Alpha and> > PowerPC).> > Open Watcom is actively developed and targets at least:> > 16 bit real mode MS-DOS> > 32 bit protected mode MS-DOS > > 16 bit Windows 3.1 (enhanced mode?) > > 32 bit code in 16 bit Windows > > 32 bit protected mode Windows (NT) > > 32 bit protected mode OS/2 > > other OS/2 variants? 16 bit? Or only 16 bit? > > Novell Netware I believe! > > x86 Linux> > > > or just> > DJGPP> > MSDOS_OPENWATCOM > > or> > > > MSDOS_DJGPP> > MSDOS_OPENWATCOM > > > > MSDOS implies 32 bit x86. ?> > > > > > PERHAPS some more names should be settled for some actual practical> > targets that might come up soon, to guide the discussion? > > > > X86_SOLARIS ? > > AMD64_SOLARIS ? > > SPARC64_SOLARIS ?> > Solaris always has two toolsets though, ? Sun and GNU?> > AMD64_DARWIN ? > > PPC64_DARWIN ? (obsolete before it ever caught on?) > > PPC64_AIX ? (this exists as an OS, right?) > > AMD64_NT -- ambiguous ? > > AMD64_NT_GNU ? > > AMD64_NT_MINGNU ? -- this toolset is already out there > > AMD64_NT_CYGWIN ? -- This isn't likely, Cygwin is very x86-specific > > AMD64_LINUX ? > > IA64_VMS ?> > IA64_LINUX ?> > two or more toolsets, right? GCC, Intel, SGI?> > IA64_LINUX_GNU ?> > IA64_LINUX_INTEL ?> > IA64_LINUX_SGI ?> > > > > > I'm skeptical there will be any IA64 ports, but I expect there will be> > AMD64.> > (I have seen IA64 for around $500 on eBay, tempting, for a port..)> > > > > > And consider LLVM in the naming exercise? As a toolset, right?> > AMD64_LINUX_GNU > > AMD64_LINUX_LLVM ? > > > > > > Are people wedded to "I386" instead of "x86"?> > To indicate clearly it isn't 286 and the ilk?> > > > > > I am somewhat wedded to "AMD64", though other names include "X64" and> > X86-64.> > X86-64 is too long, and the dash would probably go away, X8664,> > getting to be a long run of gibberish numbers.> > X64...is that less than X86? :)> > AMD64 is the value of %PROCESSOR_ARCHITECTURE% on NT and the install> > directory name on the pre-Vista CDs (along with i386), and it is> > available as an #ifdef (_AMD64_, _M_AMD64).> > > > > > Intel was calling it EM64T, but that gone away in favor of Intel64, to> > not be confused with IA64, which is Itanium, stands for Intel> > Architecture 64 hm...> > Intel64 != Intel Architecture 64.> > Admittedly they were kind of stuck.> > > > > > - Jay> > > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Sun Mar 30 17:52:06 2008 From: jayk123 at hotmail.com (Jay) Date: Sun, 30 Mar 2008 15:52:06 +0000 Subject: [M3devel] correct Alpha/OSF headers? In-Reply-To: <2CEE2E1D-A942-4BA6-B7BF-781E10F1BFD1@cs.purdue.edu> References: <2CEE2E1D-A942-4BA6-B7BF-781E10F1BFD1@cs.purdue.edu> Message-ID: I was thinking jmp_buf was double the necessary but it is correct. I thought long was 64 bits. - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] correct Alpha/OSF headers?Date: Sun, 30 Mar 2008 10:51:43 -0400 I have a ALPHA/OSF machine that's been down since it got hacked a few years ago. It was happily running Modula-3 at the time. Should not be too much work to resurrect, just time, and I don't have a lot of that. The ALPHA_OSF stuff in CM3 and PM3 is probably still accurate. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 30, 2008, at 10:34 AM, Jay wrote: correct Alpha/OSF headers?Can anyone confirm stuff like: http://www.nanobit.net/quake3_doxy/alpha_2osf_2limits_8h-source.html http://www.nanobit.net/quake3_doxy/alpha_2osf_2setjmp_8h-source.html Does anyone have a working machine?That I can ssh into? (or working Alpha anything else? Linux/FreeBSD/OpenBSD/FreeBSD/NT (I do have NT Alpha/Mips/PowerPC tools :) ?) oh but NT Alpha was only 32 bit.) - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Sun Mar 30 18:56:13 2008 From: jayk123 at hotmail.com (Jay) Date: Sun, 30 Mar 2008 16:56:13 +0000 Subject: [M3devel] the quake tests on NT386GNU In-Reply-To: <2CEE2E1D-A942-4BA6-B7BF-781E10F1BFD1@cs.purdue.edu> References: <2CEE2E1D-A942-4BA6-B7BF-781E10F1BFD1@cs.purdue.edu> Message-ID: The quake tests on NT386GNU. m3-sys\m3quake\test There are the following failures. I can partly explain all of them, and I can get them all to pass, but some of my changes are dubious. % % Executables certainly can be and are writable -- how did they get created in the first place? % if not equal ($OS, "Windows_NT") f = "res = fs_writable( more )" check( "f013", f, "" ) end so I added: fs_rmfile("readonly") > "readonly" in end exec("chmod -w readonly") f = "res = fs_writable( \"readonly\" )" check( "f013b", f, "" ) which works. f = "res = fs_lsfiles( \"a\", \"T\" )" check( "f036", f, "b apple" ) fails because it assumes the file system will return files in a certain order. This kind of thing varies legimimately. I added a check2 function that expects either of two outputs and it succeeds with: % some file system sort, some hash, some retain creation order, etc % allow either order f = "res = fs_lsfiles( \"a\", \"T\" )" check2( "f036", f, "b apple", "apple b" ) In this case I think the test is wrong. PPC_DARWIN fails the same here (and otherwise passes; so I can debug) Tests 8, 9, 10, 11 that verify they can run some pipelines, including sed, do succeed, but they get the wrong output, as if sed didn't run. If I wrap the whole thing in sh, it works. But that shouldn't be necessary as I understand. As I understand q_exec's job is to emulate a bunch of sh behavior, esp. around |, ||, &, &&, >, <, without running sh. Tests 19 and 20 use q_exec and echo. Again a sh wrapper helps, but shouldn't be necessary, as well echo 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16>x yields 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 instead of the expected 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 If you put a space between the 16 and x, and again the sh wrapper, it works. I'd say the 16 is being interpreted as a file handle, but the output does go to the right place, so it's half and half. In this case the test might be partly wrong. Lastly, simply, when you pushd/pop around, you expect BUILD_DIR, not TARGET. In this case the test is simply subtely wrong. echo.exe does exist on Cygwin and it should be what gets run by these. echo exists on Darwin (not a shell built in). I think the ls fix, BUILD_DIR fix, readonly fix, should be taken as I have them, after I test them on PPC_DARWIN. The others might be a failure of the Quake code to be consistent across platforms. Someone (maybe me) needs to debug them. I also need to test native NT386, but with sed and guess chmod in the path. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Sun Mar 30 19:54:26 2008 From: jayk123 at hotmail.com (Jay) Date: Sun, 30 Mar 2008 17:54:26 +0000 Subject: [M3devel] the quake tests on NT386GNU In-Reply-To: References: <2CEE2E1D-A942-4BA6-B7BF-781E10F1BFD1@cs.purdue.edu> Message-ID: nevermind, I had broken redirection in Cygwin Process.Create, fixed Nice tests! - Jay > From: jayk123 at hotmail.com> To: m3devel at elegosoft.com> Date: Sun, 30 Mar 2008 16:56:13 +0000> Subject: [M3devel] the quake tests on NT386GNU> > > The quake tests on NT386GNU.> > m3-sys\m3quake\test> > There are the following failures. I can partly explain all of them, and I can get them all to pass, but some of my changes are dubious.> > %> % Executables certainly can be and are writable -- how did they get created in the first place?> %> if not equal ($OS, "Windows_NT")> f = "res = fs_writable( more )"> check( "f013", f, "" )> end> > so I added:> > fs_rmfile("readonly")> > "readonly" in> end> exec("chmod -w readonly")> f = "res = fs_writable( \"readonly\" )"> check( "f013b", f, "" )> > which works.> > f = "res = fs_lsfiles( \"a\", \"T\" )"> check( "f036", f, "b apple" )> > fails because it assumes the file system will return files in a certain order.> This kind of thing varies legimimately. I added a check2 function that expects> either of two outputs and it succeeds with:> > % some file system sort, some hash, some retain creation order, etc> % allow either order> f = "res = fs_lsfiles( \"a\", \"T\" )"> check2( "f036", f, "b apple", "apple b" )> > In this case I think the test is wrong. PPC_DARWIN fails the same here (and otherwise passes; so I can debug)> > Tests 8, 9, 10, 11 that verify they can run some pipelines, including sed, do succeed, but they get the wrong output, as if sed didn't run.> If I wrap the whole thing in sh, it works.> But that shouldn't be necessary as I understand.> As I understand q_exec's job is to emulate a bunch of sh behavior, esp. around |, ||, &, &&, >, <, without running sh.> > Tests 19 and 20 use q_exec and echo.> Again a sh wrapper helps, but shouldn't be necessary, as well> > echo 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16>x> yields 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15> instead of the expected 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16> If you put a space between the 16 and x, and again the sh wrapper, it works -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Sun Mar 30 19:42:42 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 30 Mar 2008 19:42:42 +0200 Subject: [M3devel] Text structure bites again... Message-ID: <1206898962.7023.24.camel@faramir.m3w.org> Imagine string made by 50,000 & operations. After that, we have very unbalanced tree of TextCat.T's. Imagine get_chars() invocation by unsuspecting code. 10MB of default stack goes crash. -- Dragi?a Duri? From jayk123 at hotmail.com Sun Mar 30 20:22:37 2008 From: jayk123 at hotmail.com (Jay) Date: Sun, 30 Mar 2008 18:22:37 +0000 Subject: [M3devel] Text structure bites again... In-Reply-To: <1206898962.7023.24.camel@faramir.m3w.org> References: <1206898962.7023.24.camel@faramir.m3w.org> Message-ID: Personally I'd something more like Windows -- every type and function is doubled, and one set is implemented on top of the others as long as they are reasonably expensive otherwise. And then tend to do everything in Unicode. An array of bytes is an array of bytes. An array of "characters" is usually WCHAR. 10meg default stack! Geez that's crazy. How are you going to create many threads like that? On Windows the default stacks are usually 1meg or 256k, but it is customizable. - Jay > From: dragisha at m3w.org> To: m3devel at elegosoft.com> Date: Sun, 30 Mar 2008 19:42:42 +0200> Subject: [M3devel] Text structure bites again...> > Imagine string made by 50,000 & operations.> > After that, we have very unbalanced tree of TextCat.T's.> > Imagine get_chars() invocation by unsuspecting code.> > 10MB of default stack goes crash.> -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Sun Mar 30 20:18:45 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 30 Mar 2008 20:18:45 +0200 Subject: [M3devel] Text structure bites again... In-Reply-To: References: <1206898962.7023.24.camel@faramir.m3w.org> Message-ID: <1206901125.7023.27.camel@faramir.m3w.org> On Sun, 2008-03-30 at 18:22 +0000, Jay wrote: > Personally I'd something more like Windows -- every type and function > is doubled, and one set is implemented on top of the others as long as > they are reasonably expensive otherwise. And then tend to do > everything in Unicode. An array of bytes is an array of bytes. An > array of "characters" is usually WCHAR. And this would mean what exactly? > > 10meg default stack! > Geez that's crazy. How are you going to create many threads like that? > On Windows the default stacks are usually 1meg or 256k, but it is > customizable. No problem, of course. You set stack on shell level, for example. ulimit -s 64 is what I do when I expect many threads and when I know actual depth would be nothing special in every one of them. It makes 64kbyte default. > > - Jay > > > > ______________________________________________________________________ > > > From: dragisha at m3w.org > > To: m3devel at elegosoft.com > > Date: Sun, 30 Mar 2008 19:42:42 +0200 > > Subject: [M3devel] Text structure bites again... > > > > Imagine string made by 50,000 & operations. > > > > After that, we have very unbalanced tree of TextCat.T's. > > > > Imagine get_chars() invocation by unsuspecting code. > > > > 10MB of default stack goes crash. > > -- > > Dragi?a Duri? > > > -- Dragi?a Duri? From rodney.bates at wichita.edu Sun Mar 30 21:05:16 2008 From: rodney.bates at wichita.edu (Rodney M. Bates) Date: Sun, 30 Mar 2008 13:05:16 -0600 Subject: [M3devel] Text structure bites again... In-Reply-To: <1206898962.7023.24.camel@faramir.m3w.org> References: <1206898962.7023.24.camel@faramir.m3w.org> Message-ID: <47EFE46C.7010006@wichita.edu> Just some thoughts: Are all the intermediate TEXT values during all the concatenations unused (other than for the next concatenation) until the whole string is built, and only then get_chars? If so, a possible workaround might be to use a TextWr.T to build the string, then TextWr.ToText. It will keep a more efficient internal representation, (expandable array of pointers to mutable buffers of almost 1000 chars each,) and ToText is iterative. Even if some of the intermediate strings are needed, but not too often, it might be possible to do a TextWr.ToText and Wr.Put to put the result right back into the TextWr.T and get started building the next longer string. If you like it better to fix the TEXT implementation rather than change your application, I can imagine a better version of TextCat's override of get_chars that recurses on only one of the two component strings and iterates on the other. I think it could iterate on either component, as long as that is done temporally last. It could recurse on the component that is not another TextCat.T, if there is one, otherwise, recurse on the component that is shorter. This would only be a guess at which one is shallower, but might have a reasonable probability of being the right guess. Dragi?a Duri? wrote: > Imagine string made by 50,000 & operations. > > After that, we have very unbalanced tree of TextCat.T's. > > Imagine get_chars() invocation by unsuspecting code. > > 10MB of default stack goes crash. -- ------------------------------------------------------------- Rodney M. Bates, retired assistant professor Dept. of Computer Science, Wichita State University Wichita, KS 67260-0083 316-978-3922 rodney.bates at wichita.edu From dragisha at m3w.org Sun Mar 30 21:12:00 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 30 Mar 2008 21:12:00 +0200 Subject: [M3devel] Text structure bites again... In-Reply-To: <47EFE46C.7010006@wichita.edu> References: <1206898962.7023.24.camel@faramir.m3w.org> <47EFE46C.7010006@wichita.edu> Message-ID: <1206904320.7023.33.camel@faramir.m3w.org> On Sun, 2008-03-30 at 13:05 -0600, Rodney M. Bates wrote: > Just some thoughts: > > Are all the intermediate TEXT values during all the concatenations > unused (other than for the next concatenation) until the whole > string is built, and only then get_chars? If so, a possible > workaround might be to use a TextWr.T to build the string, then > TextWr.ToText. It will keep a more efficient internal representation, > (expandable array of pointers to mutable buffers of almost 1000 chars > each,) and ToText is iterative. Exellent idea. And working :). > Even if some of the intermediate strings are needed, but not too > often, it might be possible to do a TextWr.ToText and Wr.Put > to put the result right back into the TextWr.T and get started > building the next longer string. > > If you like it better to fix the TEXT implementation rather than > change your application, I can imagine a better version of TextCat's > override of get_chars that recurses on only one of the two > component strings and iterates on the other. I think it could > iterate on either component, as long as that is done temporally > last. This implementation makes & and Text.Length() blindingly fast, and & copy free. Any traversal of "TEXT tree" is potential efficiency problem. What I was thinking is to implement AVL tree instead of current one, thus making traversals as efficient as tree can be, while not burdening &. > > It could recurse on the component that is not another TextCat.T, > if there is one, otherwise, recurse on the component that is shorter. > This would only be a guess at which one is shallower, but might have > a reasonable probability of being the right guess. > > Dragi?a Duri? wrote: > > Imagine string made by 50,000 & operations. > > > > After that, we have very unbalanced tree of TextCat.T's. > > > > Imagine get_chars() invocation by unsuspecting code. > > > > 10MB of default stack goes crash. > -- Dragi?a Duri? From wagner at elegosoft.com Sat Mar 1 15:43:32 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Sat, 01 Mar 2008 15:43:32 +0100 Subject: [M3devel] state of m3gdb on LINUXLIBC6? In-Reply-To: <47C83637.6010602@wichita.edu> References: <1204226001.13580.64.camel@hias.m3w.org> <47C749B8.1060906@wichita.edu> <20080229011936.fs7gutizgggg8s44@mail.elegosoft.com> <47C83637.6010602@wichita.edu> Message-ID: <20080301154332.jk9jnfgzcw48g04s@mail.elegosoft.com> Quoting "Rodney M. Bates" : > Olaf Wagner wrote: >> Quoting "Rodney M. Bates" : >> >>> As for other platforms, I don't have any around, and I'm not aware >>> of anybody >>> trying it. >> >> It compiles without problems on FreeBSD 6.3 and works as long as >> programs are linked with libthr.so (not with libpthread.so). > > Any more information on what the difference in these libraries is? AFAIK, there are three different pthreads implementations on FreeBSD: libc_r, libpthread, libthr. libc_r is completely user-level, while pthread and thr use system level threads (with support and callbacks from the system scheduler). In FreeBSD 6.3, pthread seems to be the default, while in 7.0 (which has been released just a few days ago) they have switched to thr as default. thr is a 1:1 mapping from user to system level threads AFAIK. I switched the CM3 thread default for FreeBSD from POSIX to Antony's PTHREADS some months ago; I think we should go with the libthr as default now (since this is what the FreeBSD folks do themselves). (BTW. they promise dramatic performance increases in FreeBSD 7.0 wrt. concurrency.) In FreeBSD, dynamic library mapping for threads can be switched easily via /etc/libmap.conf. >> I haven't used it much yet, and some things seemed to be strange; >> thread stack switching or viewing didn't work at all. > > m3gdb needs work on threads. I have never done anything on it. > Unless Tony did some when he was working on it several years ago, > I think it is pretty much as it came from SRC, and that was fairly > primitive. Plus, the new pthreads implementation is not supported. > Once I glanced at the code and it looked immediately like what was > there would not work at all. This has been on my list a while. If, as the others say, stock gdb has good support for pthreads, then it is probably a FreeBSD specific problem. The BSD folks often do customizations of the GNU tools on their own, that the FSF does not accept / maintain. One would probably need to base a m3gdb for FreeBSD on the FreeBSD gdb sources :-/ >> If you want to try anything on FreeBSD, let me know, I can provide >> a remote login. > > Yes, that would be helpful. I'm currently not at home, but I'll setup a login on my FreeBSD system for you at the start of next week. I assume your ssh key is already on birch.elegosoft.com (if not, just send it to me). Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From dragisha at m3w.org Sun Mar 2 00:41:19 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 02 Mar 2008 00:41:19 +0100 Subject: [M3devel] random-acces I/O In-Reply-To: <20080229224126.GA21397@topoi.pooq.com> References: <20080229224126.GA21397@topoi.pooq.com> Message-ID: <1204414879.10030.8.camel@hias.m3w.org> On Fri, 2008-02-29 at 17:41 -0500, hendrik at topoi.pooq.com wrote: > I need to do direct-access I/O from a Modula 3 program with a file > format that is externally specified. It is necessary to both read and > write in random fashion during a run of the program, prefereably without > having to close or open the file each time. > > I expect the answers to the following questions are "Yes", but I'd > appreciate confirmation: > > ? If I have a File.T (obtained from FS.OpenFile) can I have both a > Rd.T and a Wr.T active on it simultaeously? Yes > > ? Will things written using the Wr.T be immediately available > for reading with the Rd.T, with no buffereing problems? Most probably you'll have to use unbuffered IO here. > > ? Are there methods for locking the file -- or portions of it -- > against simutaneous access by other programs, including ones written > in other languages? There are. But you have to make some rules, and follow them from all users of your file. > > -- hendrik -- Dragi?a Duri? From mika at async.caltech.edu Sat Mar 1 23:49:23 2008 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 01 Mar 2008 14:49:23 -0800 Subject: [M3devel] random-acces I/O In-Reply-To: Your message of "Fri, 29 Feb 2008 17:41:26 EST." <20080229224126.GA21397@topoi.pooq.com> Message-ID: <200803012249.m21MnOnF018386@camembert.async.caltech.edu> Hi Hendrik, The answer is, "it depends." As a matter of fact this was my biggest area of trouble when porting a program from PM3 on Unix to PM3/Klagenfurt for NT386GNU. On NT386GNU, the default is for Modula-3 to avail itself of Windows's mandatory file locking, whereas POSIX has no such thing. This means that a Modula-3 program on NT386GNU can't even open a file for reading if you are "lessing" it for instance. I am sure it can be worked around, but I never bothered to figure out how, sorry (because the behavior I needed was in fact strict locking of this kind, which I had to go to special lengths to get on Unix!) On Unix, things are a bit more as you describe them, but not exactly. You can definitely have an Rd.T and a Wr.T on a single file, but I do not believe that the Rd.T and Wr.T will be aware of each other's existence (I could be wrong). However, there is a package called rdwr(?) that I think may have the sort of functionality you're looking for. Maybe. File locking is strictly an OS issue as far as I know and that is not something that Rd and Wr concern themselves with. If you need this kind of low-level access I have a feeling Rd and Wr are too high-level for you. I would suspect you want to build something with rdwr (or some other means of getting an Rd out of a Wr or vice versa) and then extend it with routines for maintaining a faithful disk-based copy of what you want... using lower-level access methods. Oh yes Rd and Wr are documented in detail in chapter 6 of the Green Book. Mika hendrik at topoi.pooq.com writes: >I need to do direct-access I/O from a Modula 3 program with a file >format that is externally specified. It is necessary to both read and >write in random fashion during a run of the program, prefereably without >having to close or open the file each time. > >I expect the answers to the following questions are "Yes", but I'd >appreciate confirmation: > > ? If I have a File.T (obtained from FS.OpenFile) can I have both a > Rd.T and a Wr.T active on it simultaeously? > > ? Will things written using the Wr.T be immediately available > for reading with the Rd.T, with no buffereing problems? > > ? Are there methods for locking the file -- or portions of it -- > against simutaneous access by other programs, including ones written > in other languages? > >-- hendrik From dragisha at m3w.org Sun Mar 2 01:12:44 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 02 Mar 2008 01:12:44 +0100 Subject: [M3devel] random-acces I/O In-Reply-To: <200803012249.m21MnOnF018386@camembert.async.caltech.edu> References: <200803012249.m21MnOnF018386@camembert.async.caltech.edu> Message-ID: <1204416764.10030.12.camel@hias.m3w.org> rdwr is package by Blair MacIntyre and it's not related to anything File.* except indirectly, ie if you make File Wr part of TeeWr. On Sat, 2008-03-01 at 14:49 -0800, Mika Nystrom wrote: > Hi Hendrik, > > The answer is, "it depends." As a matter of fact this was my biggest > area of trouble when porting a program from PM3 on Unix to > PM3/Klagenfurt for NT386GNU. On NT386GNU, the default is for > Modula-3 to avail itself of Windows's mandatory file locking, whereas > POSIX has no such thing. This means that a Modula-3 program on > NT386GNU can't even open a file for reading if you are "lessing" > it for instance. I am sure it can be worked around, but I never > bothered to figure out how, sorry (because the behavior I needed > was in fact strict locking of this kind, which I had to go to special > lengths to get on Unix!) > > On Unix, things are a bit more as you describe them, but not exactly. > You can definitely have an Rd.T and a Wr.T on a single file, but I > do not believe that the Rd.T and Wr.T will be aware of each other's > existence (I could be wrong). However, there is a package called > rdwr(?) that I think may have the sort of functionality you're > looking for. Maybe. > > File locking is strictly an OS issue as far as I know and that is not > something that Rd and Wr concern themselves with. > > If you need this kind of low-level access I have a feeling Rd and Wr > are too high-level for you. I would suspect you want to build something > with rdwr (or some other means of getting an Rd out of a Wr or vice > versa) and then extend it with routines for maintaining a faithful > disk-based copy of what you want... using lower-level access methods. > > Oh yes Rd and Wr are documented in detail in chapter 6 of the Green Book. > > Mika > > hendrik at topoi.pooq.com writes: > >I need to do direct-access I/O from a Modula 3 program with a file > >format that is externally specified. It is necessary to both read and > >write in random fashion during a run of the program, prefereably without > >having to close or open the file each time. > > > >I expect the answers to the following questions are "Yes", but I'd > >appreciate confirmation: > > > > ? If I have a File.T (obtained from FS.OpenFile) can I have both a > > Rd.T and a Wr.T active on it simultaeously? > > > > ? Will things written using the Wr.T be immediately available > > for reading with the Rd.T, with no buffereing problems? > > > > ? Are there methods for locking the file -- or portions of it -- > > against simutaneous access by other programs, including ones written > > in other languages? > > > >-- hendrik -- Dragi?a Duri? From mika at async.caltech.edu Sun Mar 2 03:24:02 2008 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 01 Mar 2008 18:24:02 -0800 Subject: [M3devel] random-acces I/O In-Reply-To: Your message of "Sun, 02 Mar 2008 01:12:44 +0100." <1204416764.10030.12.camel@hias.m3w.org> Message-ID: <200803020224.m222O3bP023511@camembert.async.caltech.edu> Yes, I am aware of that. My point was simply that if you want a reader and writer accessing the same data it's probably easiest to connect together the reader and writer directly, somehow, and write some sort of "watcher" that flushes modifications to disk "on the side" of the interaction between the reader and writer. Btw, I do have some code that does this sort of thing (not entirely random access), and to run both on Unix and NT386GNU, yes, I do close the file every time I "turn it around" from reading to writing or vice versa. Mika =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: >rdwr is package by Blair MacIntyre and it's not related to anything >File.* except indirectly, ie if you make File Wr part of TeeWr. > >On Sat, 2008-03-01 at 14:49 -0800, Mika Nystrom wrote: >> Hi Hendrik, >> >> The answer is, "it depends." As a matter of fact this was my biggest >> area of trouble when porting a program from PM3 on Unix to >> PM3/Klagenfurt for NT386GNU. On NT386GNU, the default is for >> Modula-3 to avail itself of Windows's mandatory file locking, whereas >> POSIX has no such thing. This means that a Modula-3 program on >> NT386GNU can't even open a file for reading if you are "lessing" >> it for instance. I am sure it can be worked around, but I never >> bothered to figure out how, sorry (because the behavior I needed >> was in fact strict locking of this kind, which I had to go to special >> lengths to get on Unix!) >> >> On Unix, things are a bit more as you describe them, but not exactly. >> You can definitely have an Rd.T and a Wr.T on a single file, but I >> do not believe that the Rd.T and Wr.T will be aware of each other's >> existence (I could be wrong). However, there is a package called >> rdwr(?) that I think may have the sort of functionality you're >> looking for. Maybe. >> >> File locking is strictly an OS issue as far as I know and that is not >> something that Rd and Wr concern themselves with. >> >> If you need this kind of low-level access I have a feeling Rd and Wr >> are too high-level for you. I would suspect you want to build something > with rdwr (or some other means of getting an Rd out of a Wr or vice >> versa) and then extend it with routines for maintaining a faithful >> disk-based copy of what you want... using lower-level access methods. >> >> Oh yes Rd and Wr are documented in detail in chapter 6 of the Green Book. >> >> Mika >> >> hendrik at topoi.pooq.com writes: >> >I need to do direct-access I/O from a Modula 3 program with a file >> >format that is externally specified. It is necessary to both read and >> >write in random fashion during a run of the program, prefereably without >> >having to close or open the file each time. >> > >> >I expect the answers to the following questions are "Yes", but I'd >> >appreciate confirmation: >> > >> > ? If I have a File.T (obtained from FS.OpenFile) can I have both a >> > Rd.T and a Wr.T active on it simultaeously? >> > >> > ? Will things written using the Wr.T be immediately available >> > for reading with the Rd.T, with no buffereing problems? >> > >> > ? Are there methods for locking the file -- or portions of it -- >> > against simutaneous access by other programs, including ones written >> > in other languages? >> > >> >-- hendrik >-- >Dragi??a Duri?? From wagner at elegosoft.com Sun Mar 2 11:53:57 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Sun, 02 Mar 2008 11:53:57 +0100 Subject: [M3devel] random-acces I/O In-Reply-To: <1204414879.10030.8.camel@hias.m3w.org> References: <20080229224126.GA21397@topoi.pooq.com> <1204414879.10030.8.camel@hias.m3w.org> Message-ID: <20080302115357.2mv96dyas4so04o0@mail.elegosoft.com> Quoting Dragi?a Duri? : > On Fri, 2008-02-29 at 17:41 -0500, hendrik at topoi.pooq.com wrote: >> ? Are there methods for locking the file -- or portions of it -- >> against simutaneous access by other programs, including ones written >> in other languages? > > There are. But you have to make some rules, and follow them from all > users of your file. File.T provides lock() and unlock() methods. You must be aware though that file locking is inherently depending on operating system and file system, and as far as I know there is not the slightest chance of a general applicable specification of locks. In CM3, the POSIX target use the POSIX advisory file locking, as far as I know, which means that only those interested in heeding the lock will see it. Nobody will prevent others from reading or writing the file. On Windows, the default behaviour for NTFS is completely different; on FAT file systems there are probably no locks at all. If your files are located on an NFS share, locking depends on the implementation of the rpc.lockd on the server and on the quality of the NFS client; for SMBFS/SAMBA, it depends both on the onderlying file system and on the configuration of SAMBA. If anybody knows of a commonly available locking abstraction, I'd be interested in it, too. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From hendrik at topoi.pooq.com Sun Mar 2 14:12:47 2008 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Sun, 2 Mar 2008 08:12:47 -0500 Subject: [M3devel] random-acces I/O In-Reply-To: <20080302115357.2mv96dyas4so04o0@mail.elegosoft.com> References: <20080229224126.GA21397@topoi.pooq.com> <1204414879.10030.8.camel@hias.m3w.org> <20080302115357.2mv96dyas4so04o0@mail.elegosoft.com> Message-ID: <20080302131247.GA26816@topoi.pooq.com> On Sun, Mar 02, 2008 at 11:53:57AM +0100, Olaf Wagner wrote: > Quoting Dragi?a Duri? : > > >On Fri, 2008-02-29 at 17:41 -0500, hendrik at topoi.pooq.com wrote: > >> ? Are there methods for locking the file -- or portions of it -- > >> against simutaneous access by other programs, including ones written > >> in other languages? > > > > There are. But you have to make some rules, and follow them from all > >users of your file. > > File.T provides lock() and unlock() methods. You must be aware though > that file locking is inherently depending on operating system and > file system, and as far as I know there is not the slightest chance > of a general applicable specification of locks. In CM3, the POSIX > target use the POSIX advisory file locking, as far as I know, which > means that only those interested in heeding the lock will see it. > Nobody will prevent others from reading or writing the file. Advisory is fine. > > On Windows, the default behaviour for NTFS is completely different; > on FAT file systems there are probably no locks at all. > > If your files are located on an NFS share, locking depends on > the implementation of the rpc.lockd on the server and on the quality > of the NFS client; for SMBFS/SAMBA, it depends both on the onderlying > file system and on the configuration of SAMBA. In Linux, locking via fcntl is NFS-safe. locking via flock is apparently not. lockf is implemented via fcntl, so it is presumably NFS-safe, too, although the spoecifications don't actually require it to be implemented via fcntl. When I had NFS misconfigured (and I could find no information indicating how reconfigure it) one program (monotone) could not lock any NFS files, knwew it, and refused to operate as a result. That's the safe way to deal with NFS misconfiguration. I was annoyed, but overall I was quite impressed. I suspect it used fcntl, though maybe I should ask. The NFS configuration problem went away on its own during upgrades of etch (which at the time was Debian testing). I have no idea what the problem was. What primitive does Modula 3 use on Linux systems to perform locking? Let me hope for fcntl. Or lockf. But not flock. -- hendrik From wagner at elegosoft.com Sun Mar 2 15:11:42 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Sun, 02 Mar 2008 15:11:42 +0100 Subject: [M3devel] random-acces I/O In-Reply-To: <20080302131247.GA26816@topoi.pooq.com> References: <20080229224126.GA21397@topoi.pooq.com> <1204414879.10030.8.camel@hias.m3w.org> <20080302115357.2mv96dyas4so04o0@mail.elegosoft.com> <20080302131247.GA26816@topoi.pooq.com> Message-ID: <20080302151142.qes8lz26qs8gwo44@mail.elegosoft.com> Quoting hendrik at topoi.pooq.com: > On Sun, Mar 02, 2008 at 11:53:57AM +0100, Olaf Wagner wrote: >> Quoting Dragi?a Duri? : > In Linux, locking via fcntl is NFS-safe. locking via flock is > apparently not. lockf is implemented via fcntl, so it is presumably > NFS-safe, too, although the spoecifications don't actually require it to > be implemented via fcntl. > > When I had NFS misconfigured (and I could find no information indicating > how reconfigure it) one program (monotone) could not lock any NFS files, > knwew it, and refused to operate as a result. That's the safe way > to deal with NFS misconfiguration. I was annoyed, but overall I was > quite impressed. I suspect it used fcntl, though maybe I should ask. > The NFS configuration problem went away on its own during upgrades of > etch (which at the time was Debian testing). I have no idea what the > problem was. > > What primitive does Modula 3 use on Linux systems to perform locking? > Let me hope for fcntl. Or lockf. But not flock. fcntl. See http://www.opencm3.net/doc/help/gen_html/libm3/src/os/POSIX/FilePosix.m3.html Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jayk123 at hotmail.com Sun Mar 2 15:54:02 2008 From: jayk123 at hotmail.com (Jay) Date: Sun, 2 Mar 2008 14:54:02 +0000 Subject: [M3devel] full paths to source files in debug info In-Reply-To: <200802292119.m1TLJhcB080697@camembert.async.caltech.edu> References: Your message of "Fri, 29 Feb 2008 11:11:35 CST." <47C83CC7.9060905@wichita.edu> <200802292119.m1TLJhcB080697@camembert.async.caltech.edu> Message-ID: I probably only originally tested changes involving only parse.c, and then the final parse.c change with the m3front changes. I probably did not previously test only changing m3front. I have now tested only changing m3front and the behavior is as I desire. I put back parse.c and left m3front asis. And I believe has a good chance of satisfying everyone else. The assembly source does not have the full path to any FILE, but it does have the current working directory, or directory of the input *c file -- these are always the same and so it is ambiguous, and I didn't track down the code or try where they differ. The two m3front implementation in question, are, given: ../src/Foo.m3 ../src/POSIX/FooPosix.m3 should gcc be given the paths as show above or Foo.m3 FooPosix.m3 Historically the answer was Foo.m3 and FooPosix.m3. This does not lead to what I expect for an "easier" debugging experience (oxymoron?) Changing to the ../src path does. As well I once tested setting a breakpoint by file and line number in gdb. With ../src/foo.m3, you can still break on Foo.m3:123. ok? I did not check that given Foo.m3, if the symbols still have that directory path. It is POSSIBLE that "wrecking" the source file path causes gcc to omit the directory name, but I would be surprised. It is definitely possible to have gdb always check for "../src", however this isn't general and I'm sure would not be accepted "upstream", and doesn't handle m3core, etc. - Jay > CC: m3devel at elegosoft.com> Date: Fri, 29 Feb 2008 13:19:43 -0800> From: mika at async.caltech.edu> Subject: Re: [M3devel] FW: put full paths to source files in debug info]> > "Rodney M. Bates" writes:> ...> >People can and often do move executables to different machines and also> >sometimes move source trees around on the original machine. Using simple> >file names within an executable (of which there is always exactly one)> >is invariant to all this, and a lot shorter to type/read.> ...> > Arbitrary paths are also not trustworthy on many Unix systems that run> automounters. The path you get from getcwd is likely not to be permanently> accessible on those. (You access the files via a different path.)> > Mika _________________________________________________________________ Climb to the top of the charts!?Play the word scramble challenge with star power. http://club.live.com/star_shuffle.aspx?icid=starshuffle_wlmailtextlink_jan -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Sun Mar 2 17:06:07 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 02 Mar 2008 17:06:07 +0100 Subject: [M3devel] random-acces I/O In-Reply-To: <200803020224.m222O3bP023511@camembert.async.caltech.edu> References: <200803020224.m222O3bP023511@camembert.async.caltech.edu> Message-ID: <1204473967.10030.32.camel@hias.m3w.org> On Sat, 2008-03-01 at 18:24 -0800, Mika Nystrom wrote: > Yes, I am aware of that. My point was simply that if you want > a reader and writer accessing the same data it's probably easiest > to connect together the reader and writer directly, somehow, and write > some sort of "watcher" that flushes modifications to disk "on the side" > of the interaction between the reader and writer. But not back to back, reader to writer, as in RdWrPipe. > Btw, I do have some code that does this sort of thing (not entirely > random access), and to run both on Unix and NT386GNU, yes, I do > close the file every time I "turn it around" from reading to writing > or vice versa. I have database library doint exactly that, with (of course) OO, transactions, multinode access and much more. Locking mechanisms are old hat, there is lot of options. Skip buffers, even readers/writers, and you don't have to close/reopen file. Point is, in Hendrik's case, he must define usage patterns for both Modula-3 and non-Modula-3 code, and follow them. > > Mika > > > =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: > >rdwr is package by Blair MacIntyre and it's not related to anything > >File.* except indirectly, ie if you make File Wr part of TeeWr. > > > >On Sat, 2008-03-01 at 14:49 -0800, Mika Nystrom wrote: > >> Hi Hendrik, > >> > >> The answer is, "it depends." As a matter of fact this was my biggest > >> area of trouble when porting a program from PM3 on Unix to > >> PM3/Klagenfurt for NT386GNU. On NT386GNU, the default is for > >> Modula-3 to avail itself of Windows's mandatory file locking, whereas > >> POSIX has no such thing. This means that a Modula-3 program on > >> NT386GNU can't even open a file for reading if you are "lessing" > >> it for instance. I am sure it can be worked around, but I never > >> bothered to figure out how, sorry (because the behavior I needed > >> was in fact strict locking of this kind, which I had to go to special > >> lengths to get on Unix!) > >> > >> On Unix, things are a bit more as you describe them, but not exactly. > >> You can definitely have an Rd.T and a Wr.T on a single file, but I > >> do not believe that the Rd.T and Wr.T will be aware of each other's > >> existence (I could be wrong). However, there is a package called > >> rdwr(?) that I think may have the sort of functionality you're > >> looking for. Maybe. > >> > >> File locking is strictly an OS issue as far as I know and that is not > >> something that Rd and Wr concern themselves with. > >> > >> If you need this kind of low-level access I have a feeling Rd and Wr > >> are too high-level for you. I would suspect you want to build something > > with rdwr (or some other means of getting an Rd out of a Wr or vice > >> versa) and then extend it with routines for maintaining a faithful > >> disk-based copy of what you want... using lower-level access methods. > >> > >> Oh yes Rd and Wr are documented in detail in chapter 6 of the Green Book. > >> > >> Mika > >> > >> hendrik at topoi.pooq.com writes: > >> >I need to do direct-access I/O from a Modula 3 program with a file > >> >format that is externally specified. It is necessary to both read and > >> >write in random fashion during a run of the program, prefereably without > >> >having to close or open the file each time. > >> > > >> >I expect the answers to the following questions are "Yes", but I'd > >> >appreciate confirmation: > >> > > >> > ? If I have a File.T (obtained from FS.OpenFile) can I have both a > >> > Rd.T and a Wr.T active on it simultaeously? > >> > > >> > ? Will things written using the Wr.T be immediately available > >> > for reading with the Rd.T, with no buffereing problems? > >> > > >> > ? Are there methods for locking the file -- or portions of it -- > >> > against simutaneous access by other programs, including ones written > >> > in other languages? > >> > > >> >-- hendrik > >-- > >Dragi?a Duri? -- Dragi?a Duri? From dragisha at m3w.org Sun Mar 2 22:12:39 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 02 Mar 2008 22:12:39 +0100 Subject: [M3devel] DEBUG pragma Message-ID: <1204492359.10030.40.camel@hias.m3w.org> I saw this old post by Mr. Hosking about DEBUG pragma and a way to turn it off. Is there more detailed info on it? Argument to cm3 to turn it off? Runtime switch? TIA, dd -- Dragi?a Duri? From jayk123 at hotmail.com Mon Mar 3 01:16:22 2008 From: jayk123 at hotmail.com (Jay) Date: Mon, 3 Mar 2008 00:16:22 +0000 Subject: [M3devel] full paths to source files in debug info In-Reply-To: References: Your message of "Fri, 29 Feb 2008 11:11:35 CST." <47C83CC7.9060905@wichita.edu> <200802292119.m1TLJhcB080697@camembert.async.caltech.edu> Message-ID: You do now get like: #0 Thread__Fork (M3_EMTrVz_closure=0x7ff9ef58) at ..\src\thread\WIN32\ThreadWin32.m3:653#1 0x643cc0d5 in FileBrowserVBT_M3 (M3_AcxOUs_mode=1) at ..\src\lego\FileBrowserVBT.m3:942#2 0x6bcd840e in RTLinker__RunMainBody (M3_DjPxE3_m=0x64455c60) at ../src/runtime/common/RTLinker.m3:399#3 0x6bcd82d6 in RTLinker__RunMainBody (M3_DjPxE3_m=0x6d840e20) at ../src/runtime/common/RTLinker.m3:379#4 0x6bcd82d6 in RTLinker__RunMainBody (M3_DjPxE3_m=0x6d845480) at ../src/runtime/common/RTLinker.m3:379#5 0x6bcd82d6 in RTLinker__RunMainBody (M3_DjPxE3_m=0x6d841d60) at ../src/runtime/common/RTLinker.m3:379#6 0x6bcd82d6 in RTLinker__RunMainBody (M3_DjPxE3_m=0x4614e0) at ../src/runtime/common/RTLinker.m3:379#7 0x6bcd7939 in RTLinker__AddUnitI (M3_DjPxE3_m=0x4614e0) at ../src/runtime/common/RTLinker.m3:113#8 0x6bcd79c0 in RTLinker__AddUnit (M3_DjPxE5_b=0x44a4d3)whereas presumably before you would get: #0 Thread__Fork (M3_EMTrVz_closure=0x7ff9ef58) at ThreadWin32.m3:653#1 0x643cc0d5 in FileBrowserVBT_M3 (M3_AcxOUs_mode=1) at FileBrowserVBT.m3:942#2 0x6bcd840e in RTLinker__RunMainBody (M3_DjPxE3_m=0x64455c60) at RTLinker.m3:399#3 0x6bcd82d6 in RTLinker__RunMainBody (M3_DjPxE3_m=0x6d840e20) at RTLinker.m3:379#4 0x6bcd82d6 in RTLinker__RunMainBody (M3_DjPxE3_m=0x6d845480) at RTLinker.m3:379#5 0x6bcd82d6 in RTLinker__RunMainBody (M3_DjPxE3_m=0x6d841d60) at RTLinker.m3:379#6 0x6bcd82d6 in RTLinker__RunMainBody (M3_DjPxE3_m=0x4614e0) at RTLinker.m3:379#7 0x6bcd7939 in RTLinker__AddUnitI (M3_DjPxE3_m=0x4614e0) at RTLinker.m3:113#8 0x6bcd79c0 in RTLinker__AddUnit (M3_DjPxE5_b=0x44a4d3) ok? I agree this medium level of verbosity in the ui is not useful. The gain is that gdb finds the source files automatically. - Jay From: jayk123 at hotmail.comTo: m3devel at elegosoft.comDate: Sun, 2 Mar 2008 14:54:02 +0000Subject: [M3devel] full paths to source files in debug info I probably only originally tested changes involving only parse.c, and then the final parse.c change with the m3front changes. I probably did not previously test only changing m3front. I have now tested only changing m3front and the behavior is as I desire.I put back parse.c and left m3front asis. And I believe has a good chance of satisfying everyone else. The assembly source does not have the full path to any FILE, but it does have the current working directory, or directory of the input *c file -- these are always the same and so it is ambiguous, and I didn't track down the code or try where they differ. The two m3front implementation in question, are, given: ../src/Foo.m3 ../src/POSIX/FooPosix.m3 should gcc be given the paths as show above or Foo.m3 FooPosix.m3 Historically the answer was Foo.m3 and FooPosix.m3.This does not lead to what I expect for an "easier" debugging experience (oxymoron?)Changing to the ../src path does. As well I once tested setting a breakpoint by file and line number in gdb.With ../src/foo.m3, you can still break on Foo.m3:123. ok? I did not check that given Foo.m3, if the symbols still have that directory path.It is POSSIBLE that "wrecking" the source file path causes gcc to omit the directory name,but I would be surprised. It is definitely possible to have gdb always check for "../src", however this isn't generaland I'm sure would not be accepted "upstream", and doesn't handle m3core, etc. - Jay > CC: m3devel at elegosoft.com> Date: Fri, 29 Feb 2008 13:19:43 -0800> From: mika at async.caltech.edu> Subject: Re: [M3devel] FW: put full paths to source files in debug info]> > "Rodney M. Bates" writes:> ...> >People can and often do move executables to different machines and also> >sometimes move source trees around on the original machine. Using simple> >file names within an executable (of which there is always exactly one)> >is invariant to all this, and a lot shorter to type/read.> ...> > Arbitrary paths are also not trustworthy on many Unix systems that run> automounters. The path you get from getcwd is likely not to be permanently> accessible on those. (You access the files via a different path.)> > Mika Climb to the top of the charts! Play the word scramble challenge with star power. Play now! _________________________________________________________________ Climb to the top of the charts!?Play the word scramble challenge with star power. http://club.live.com/star_shuffle.aspx?icid=starshuffle_wlmailtextlink_jan -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Mon Mar 3 01:23:28 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 2 Mar 2008 19:23:28 -0500 Subject: [M3devel] DEBUG pragma In-Reply-To: <1204492359.10030.40.camel@hias.m3w.org> References: <1204492359.10030.40.camel@hias.m3w.org> Message-ID: <2C2B845D-9A4E-439B-A81D-3F6E6F67380D@cs.purdue.edu> M3_OPTIONS("-NoDebug") should do it I think. On Mar 2, 2008, at 4:12 PM, Dragi?a Duri? wrote: > I saw this old post by Mr. Hosking about DEBUG pragma and a way to > turn > it off. Is there more detailed info on it? Argument to cm3 to turn it > off? Runtime switch? > > TIA, > dd > > -- > Dragi?a Duri? From mika at async.caltech.edu Mon Mar 3 03:34:41 2008 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 02 Mar 2008 18:34:41 -0800 Subject: [M3devel] [M3commit] tint In-Reply-To: Your message of "Mon, 03 Mar 2008 02:16:46 GMT." Message-ID: <200803030234.m232YfW8058063@camembert.async.caltech.edu> Language spec section 2.6.8, Green Book p. 52 Jay writes: >--_1ea12ebb-f3a7-4477-abce-de694783b438_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > >Tony, it is definitely confusing to claim the array has NUMBER(x) elements,= > and then initialize only some of them "correctly". >Trailing zeros I can see as maybe clear for positive values, but definitely= > confusing for negative values. > > - Jay From dragisha at m3w.org Mon Mar 3 08:56:03 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Mon, 03 Mar 2008 08:56:03 +0100 Subject: [M3devel] cminstall floating point exception Message-ID: <1204530963.10030.51.camel@hias.m3w.org> When trying to use cminstall on Fedora Core 4 box, I am getting this: Floating point exception./cminstall -q $INSTALLROOT Is this known problem with some older glibc? Something I can work around? dd -- Dragi?a Duri? From wagner at elegosoft.com Mon Mar 3 12:29:04 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Mon, 03 Mar 2008 12:29:04 +0100 Subject: [M3devel] Regression tests Message-ID: <20080303122904.qt7o6w6j8k4c04wg@mail.elegosoft.com> After noticing that some more external test results begin to appear in the CM3 tinderbox, I'd like to point out again the differences between the `lastok build' and `release build' tests. `lastok build' tests include simply building the complete standard set of packages with an already installed CM3. They indicate that o all standard packages compile o no imcompatibilities have been introduced since the last test (which would require a more sophisticated compiler bootstrap) `release build' tests include the following: o a compiler upgrade based on the last release (currently 5.4.0) including (a) two stage bootstrap and (b) forced configuration upgrade via cminstall o running of the m3-sys/m3tests test suite (compiler and runtime) --> shipping of the test report to www.opencm3.net o complete build of all packages (based on scripts/pkginfo.txt) and running of package tests --> shipping of build and test report to www.opencm3.net o building of binary distribution archives and their --> shipping to www.opencm3.net o generating the complete package documentation with m3tohtml `release build' will be extended as needed with further tests and archive generation etc. Personally, I don't think we need to run `lastok build' on all target platforms, but we should run `release build' on as many platforms as possible in order to have all the different test results listed above available. The explaining texts on the tinderbox page should probably be improved, too :-) Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From rodney.bates at wichita.edu Tue Mar 4 22:48:51 2008 From: rodney.bates at wichita.edu (Rodney M. Bates) Date: Tue, 04 Mar 2008 15:48:51 -0600 Subject: [M3devel] CG error messages on subranges of LONGINT Message-ID: <47CDC3C3.601@wichita.edu> I wrote the attached test program for testing compiler/debugger subrange bounds communication. It gets CG errors from the compiler: "../src/Subranges.m3", line 34: ** INTERNAL CG ERROR *** unaligned store_indirect type=6 s/a=32/32 "../src/Subranges.m3", line 35: ** INTERNAL CG ERROR *** unaligned store_indirect type=6 s/a=32/32 "../src/Subranges.m3", line 43: ** INTERNAL CG ERROR *** unaligned store_indirect type=6 s/a=32/32 Attempts to remove seemingly irrelevant declarations change the behavior. -- ------------------------------------------------------------- Rodney M. Bates, retired assistant professor Dept. of Computer Science, Wichita State University Wichita, KS 67260-0083 316-978-3922 rodney.bates at wichita.edu -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: Subranges.m3 URL: From hosking at cs.purdue.edu Wed Mar 5 02:45:12 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 4 Mar 2008 20:45:12 -0500 Subject: [M3devel] CG error messages on subranges of LONGINT In-Reply-To: <47CDC3C3.601@wichita.edu> References: <47CDC3C3.601@wichita.edu> Message-ID: <734E2A5D-AEA2-4C2F-AC03-B13632EB597C@cs.purdue.edu> Looks like alignment of LONGINT is not getting handled properly. I'll try to look into it. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 4, 2008, at 4:48 PM, Rodney M. Bates wrote: > I wrote the attached test program for testing compiler/debugger > subrange bounds communication. It gets CG errors from the compiler: > > "../src/Subranges.m3", line 34: ** INTERNAL CG ERROR *** unaligned > store_indirect type=6 s/a=32/32 > "../src/Subranges.m3", line 35: ** INTERNAL CG ERROR *** unaligned > store_indirect type=6 s/a=32/32 > "../src/Subranges.m3", line 43: ** INTERNAL CG ERROR *** unaligned > store_indirect type=6 s/a=32/32 > > > Attempts to remove seemingly irrelevant declarations change the > behavior. > > > -- > ------------------------------------------------------------- > Rodney M. Bates, retired assistant professor > Dept. of Computer Science, Wichita State University > Wichita, KS 67260-0083 > 316-978-3922 > rodney.bates at wichita.edu > MODULE Subranges EXPORTS Main > > ; TYPE U1 = [ 0 .. 16_7F ] > ; TYPE S1 = [ 0 .. 16_FF ] > ; TYPE U2 = [ 0 .. 16_7FFF ] > ; TYPE S2 = [ 0 .. 16_FFFF ] > ; TYPE U3 = [ 0 .. 16_7FFFFF ] > ; TYPE S3 = [ 0 .. 16_FFFFFF ] > ; TYPE U4 = [ 0 .. 16_7FFFFFFF ] > ; TYPE S4 = [ 16_FFFFFFFF .. 16_FFFFFFFF ] > > ; TYPE LU4 = [ 0L .. 16_7FFFFFFFL ] > ; TYPE LS4 = [ 0L .. 16_FFFFFFFFL ] > ; TYPE LU5 = [ 0L .. 16_7FFFFFFFFFL ] > ; TYPE LS5 = [ 0L .. 16_FFFFFFFFFFL ] > ; TYPE LU6 = [ 0L .. 16_7FFFFFFFFFFFL ] > ; TYPE LS6 = [ 0L .. 16_FFFFFFFFFFFFL ] > ; TYPE LU7 = [ 0L .. 16_7FFFFFFFFFFFFFL ] > ; TYPE LS7 = [ 0L .. 16_FFFFFFFFFFFFFFL ] > ; TYPE LU8 = [ 0L .. 16_7FFFFFFFFFFFFFFFL ] > ; TYPE LS8 = [ 0L .. 16_FFFFFFFFFFFFFFFFL ] > > ; PROCEDURE Work ( ) > > = VAR VU1 : U1 := 0 > ; VAR VS1 : S1 := 0 > ; VAR VU2 : U2 := 0 > ; VAR VS2 : S2 := 0 > ; VAR VU3 : U3 := 0 > ; VAR VS3 : S3 := 0 > ; VAR VU4 : U4 := 0 > ; VAR VS4 : S4 := 16_FFFFFFFF > > ; VAR VLU4 : LU4 := 0L > ; VAR VLS4 : LS4 := 0L > ; VAR VLU5 : LU5 := 0L > ; VAR VLS5 : LS5 := 0L > ; VAR VLU6 : LU6 := 0L > ; VAR VLS6 : LS6 := 0L > ; VAR VLU7 : LU7 := 0L > ; VAR VLS7 : LS7 := 0L > ; VAR VLU8 : LU8 := 0L > ; VAR VLS8 : LS8 := 0L > ; VAR X : INTEGER > > ; BEGIN > X := VU1 > END Work > > ; BEGIN > Work () > END Subranges > . -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Wed Mar 5 19:35:20 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Wed, 05 Mar 2008 19:35:20 +0100 Subject: [M3devel] CG error messages on subranges of LONGINT In-Reply-To: <47CDC3C3.601@wichita.edu> References: <47CDC3C3.601@wichita.edu> Message-ID: <20080305193520.3ypns4kao4gco0ss@mail.elegosoft.com> Quoting "Rodney M. Bates" : > I wrote the attached test program for testing compiler/debugger > subrange bounds communication. It gets CG errors from the compiler: > > "../src/Subranges.m3", line 34: ** INTERNAL CG ERROR *** unaligned > store_indirect type=6 s/a=32/32 > "../src/Subranges.m3", line 35: ** INTERNAL CG ERROR *** unaligned > store_indirect type=6 s/a=32/32 > "../src/Subranges.m3", line 43: ** INTERNAL CG ERROR *** unaligned > store_indirect type=6 s/a=32/32 > > Attempts to remove seemingly irrelevant declarations change the > behavior. I've added that test program as p207 to m3-sys/m3tests. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From wagner at elegosoft.com Wed Mar 5 19:39:47 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Wed, 05 Mar 2008 19:39:47 +0100 Subject: [M3devel] cminstall floating point exception In-Reply-To: <1204530963.10030.51.camel@hias.m3w.org> References: <1204530963.10030.51.camel@hias.m3w.org> Message-ID: <20080305193947.0bxs0my3wowgogow@mail.elegosoft.com> Quoting Dragi?a Duri? : > When trying to use cminstall on Fedora Core 4 box, I am getting this: > > Floating point exception./cminstall -q $INSTALLROOT > > Is this known problem with some older glibc? Something I can work > around? It's nothing that I remember and not documented on the Known Problems page. Can you get a stack backtrace with gcc and show against which libraries the program is linked (ldd)? Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From dragisha at m3w.org Wed Mar 5 20:47:15 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Wed, 05 Mar 2008 20:47:15 +0100 Subject: [M3devel] Promotion, anyone? Message-ID: <1204746435.8782.2.camel@hias.m3w.org> http://shootout.alioth.debian.org/ -- Dragi?a Duri? From dragisha at m3w.org Thu Mar 6 16:50:59 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 06 Mar 2008 16:50:59 +0100 Subject: [M3devel] What is this??? How's it possible? Message-ID: <1204818659.8782.16.camel@hias.m3w.org> *** *** runtime error: *** A compile-time type is missing. *** file "../src/runtime/common/RTLinker.i3" *** -- Dragi?a Duri? From hosking at cs.purdue.edu Thu Mar 6 17:15:30 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 6 Mar 2008 11:15:30 -0500 Subject: [M3devel] What is this??? How's it possible? In-Reply-To: <1204818659.8782.16.camel@hias.m3w.org> References: <1204818659.8782.16.camel@hias.m3w.org> Message-ID: <4AB9C403-1CF8-4621-8E90-9B5BB1F8B4B4@cs.purdue.edu> This usually means you did not rebuild a clean set of binaries. i.e., some old binaries were lying around that made old assumptions about type ids. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 6, 2008, at 10:50 AM, Dragi?a Duri? wrote: > *** > *** runtime error: > *** A compile-time type is missing. > *** file "../src/runtime/common/RTLinker.i3" > *** > > -- > Dragi?a Duri? -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Thu Mar 6 18:23:19 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Thu, 06 Mar 2008 18:23:19 +0100 Subject: [M3devel] What is this??? How's it possible? In-Reply-To: <4AB9C403-1CF8-4621-8E90-9B5BB1F8B4B4@cs.purdue.edu> References: <1204818659.8782.16.camel@hias.m3w.org> <4AB9C403-1CF8-4621-8E90-9B5BB1F8B4B4@cs.purdue.edu> Message-ID: <20080306182319.u2flyt1uskg0owkg@mail.elegosoft.com> Quoting Tony Hosking : > This usually means you did not rebuild a clean set of binaries. i.e., > some old binaries were lying around that made old assumptions about > type ids. > On Mar 6, 2008, at 10:50 AM, Dragi?a Duri? wrote: > >> *** >> *** runtime error: >> *** A compile-time type is missing. >> *** file "../src/runtime/common/RTLinker.i3" >> *** Well, we'd have to admit it _is_ a little bit cryptic and may not be very helpful and encouraging to the casual user :-) In theory, CM3 should be able to detect such version conflicts even at link time (using fingerprints for compiled units etc.). I seem to remember that older M3 versions (i.e. DEC SRC M3) were more reliable in detecting such things; but my memory may be wrong or the environment completely different (no shared objects for example). I'd like to better understand this kind of error: does it only show up in connection with old libraries, only in connection with old runtime modules (some special rules here), only when using shared libraries and so on. Conceptually, it's a kind of version error and should be notified as early as possible. Then we can think of ways to (a) detect it earlier (if possible) and (b) improve the user notification ;-) Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From dragisha at m3w.org Thu Mar 6 18:35:07 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 06 Mar 2008 18:35:07 +0100 Subject: [M3devel] What is this??? How's it possible? In-Reply-To: <4AB9C403-1CF8-4621-8E90-9B5BB1F8B4B4@cs.purdue.edu> References: <1204818659.8782.16.camel@hias.m3w.org> <4AB9C403-1CF8-4621-8E90-9B5BB1F8B4B4@cs.purdue.edu> Message-ID: <1204824907.8782.20.camel@hias.m3w.org> It was my mistake, of course. Dynamically loaded libraries were not up to date. And day was too hard on me :). dd On Thu, 2008-03-06 at 11:15 -0500, Tony Hosking wrote: > This usually means you did not rebuild a clean set of binaries. i.e., > some old binaries were lying around that made old assumptions about > type ids. > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > On Mar 6, 2008, at 10:50 AM, Dragi?a Duri? wrote: > > > *** > > *** runtime error: > > *** A compile-time type is missing. > > *** file "../src/runtime/common/RTLinker.i3" > > *** > > > > -- > > Dragi?a Duri? > > > -- Dragi?a Duri? From dragisha at m3w.org Thu Mar 6 19:30:33 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 06 Mar 2008 19:30:33 +0100 Subject: [M3devel] What is this??? How's it possible? In-Reply-To: <20080306182319.u2flyt1uskg0owkg@mail.elegosoft.com> References: <1204818659.8782.16.camel@hias.m3w.org> <4AB9C403-1CF8-4621-8E90-9B5BB1F8B4B4@cs.purdue.edu> <20080306182319.u2flyt1uskg0owkg@mail.elegosoft.com> Message-ID: <1204828233.8782.34.camel@hias.m3w.org> My situation was like this: In hm3-m3lib0 library I have two types: Killer.T Killer.Client That library is linked to xli binary, dynamically. hm3-web is one of libraries I am loading dynamically at runtime (using dlopen and co). Its FastCGI and WebServer modules are using above mentioned two types. I've changed hm3-m3lib0, and hm3-web remained unchanged. So, when I invoke this code: dl:=NEW(DL.T).init(pIn, global); FOR i:=1 TO r.size() DO RTLinker.AddUnit(LOOPHOLE(dl.sym(NARROW(r.get(i-1), TEXT) & "_M3").addr, RT0.Binder)); (* IO.Put("Adding: " & NARROW(r.get(i-1), TEXT) & "\n"); *) END; RTLinker.AddUnit fails. Message is one I've sent. When this error happens during "normal" initialization, runtime error is obviuosly normal behaviour. You can't ignore one of linked libraries and expect system to work. But, as some of these additional libraries (in my case scripting engine plugins) are optional, it's probably wise to allow interception of this runtime error. This is probably what RuntimeError.E is for? I'll try it tonight. dd On Thu, 2008-03-06 at 18:23 +0100, Olaf Wagner wrote: > Quoting Tony Hosking : > > > This usually means you did not rebuild a clean set of binaries. i.e., > > some old binaries were lying around that made old assumptions about > > type ids. > > > On Mar 6, 2008, at 10:50 AM, Dragi?a Duri? wrote: > > > >> *** > >> *** runtime error: > >> *** A compile-time type is missing. > >> *** file "../src/runtime/common/RTLinker.i3" > >> *** > > Well, we'd have to admit it _is_ a little bit cryptic and may not > be very helpful and encouraging to the casual user :-) > > In theory, CM3 should be able to detect such version conflicts > even at link time (using fingerprints for compiled units etc.). > I seem to remember that older M3 versions (i.e. DEC SRC M3) were > more reliable in detecting such things; but my memory may be wrong > or the environment completely different (no shared objects for example). > > I'd like to better understand this kind of error: does it only show > up in connection with old libraries, only in connection with old > runtime modules (some special rules here), only when using shared > libraries and so on. Conceptually, it's a kind of version error and > should be notified as early as possible. > > Then we can think of ways to (a) detect it earlier (if possible) and > (b) improve the user notification ;-) > > Olaf -- Dragi?a Duri? From dragisha at m3w.org Thu Mar 6 20:02:32 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 06 Mar 2008 20:02:32 +0100 Subject: [M3devel] What is this??? How's it possible? In-Reply-To: <20080306182319.u2flyt1uskg0owkg@mail.elegosoft.com> References: <1204818659.8782.16.camel@hias.m3w.org> <4AB9C403-1CF8-4621-8E90-9B5BB1F8B4B4@cs.purdue.edu> <20080306182319.u2flyt1uskg0owkg@mail.elegosoft.com> Message-ID: <1204830152.8782.39.camel@hias.m3w.org> Tried: TRY RTLinker.AddUnit(LOOPHOLE(dl.sym(NARROW(r.get(i-1), TEXT) & "_M3").addr, RT0.Binder)); EXCEPT RuntimeError.E(expl) => <* DEBUG "Error adding " & r.get(i-1) *> END; It still fails with: *** *** runtime error: *** A compile-time type is missing. *** file "../src/runtime/common/RTLinker.i3" *** And RTType.Fail procedure, only one getting RuntimeError.MisingType as argument, looks like this: PROCEDURE Fail (rte: RTE; m: RT0.ModulePtr; x, y: ADDRESS) = <*FATAL ANY*> ... RTException.Raise (a); END Fail; But, I am getting above mentioned error. dd On Thu, 2008-03-06 at 18:23 +0100, Olaf Wagner wrote: > Quoting Tony Hosking : > > > This usually means you did not rebuild a clean set of binaries. i.e., > > some old binaries were lying around that made old assumptions about > > type ids. > > > On Mar 6, 2008, at 10:50 AM, Dragi?a Duri? wrote: > > > >> *** > >> *** runtime error: > >> *** A compile-time type is missing. > >> *** file "../src/runtime/common/RTLinker.i3" > >> *** > > Well, we'd have to admit it _is_ a little bit cryptic and may not > be very helpful and encouraging to the casual user :-) > > In theory, CM3 should be able to detect such version conflicts > even at link time (using fingerprints for compiled units etc.). > I seem to remember that older M3 versions (i.e. DEC SRC M3) were > more reliable in detecting such things; but my memory may be wrong > or the environment completely different (no shared objects for example). > > I'd like to better understand this kind of error: does it only show > up in connection with old libraries, only in connection with old > runtime modules (some special rules here), only when using shared > libraries and so on. Conceptually, it's a kind of version error and > should be notified as early as possible. > > Then we can think of ways to (a) detect it earlier (if possible) and > (b) improve the user notification ;-) > > Olaf -- Dragi?a Duri? From jayk123 at hotmail.com Fri Mar 7 07:00:46 2008 From: jayk123 at hotmail.com (Jay) Date: Fri, 7 Mar 2008 06:00:46 +0000 Subject: [M3devel] thread.fork getting nil on nt386gnu? In-Reply-To: <6346DE25-6495-49D0-A990-F48B989218AC@cs.purdue.edu> References: <20080224134642.48e7fowms8ooowo8@mail.elegosoft.com> <20080224143053.fdtxd48fkskggws0@mail.elegosoft.com> <20080224153541.59c105ro8e88480g@mail.elegosoft.com> <6346DE25-6495-49D0-A990-F48B989218AC@cs.purdue.edu> Message-ID: This asssertion is absent from the two other thread implementations, and is dependent on a loophole to underlying details that I didn't track down to determine if they are corrrect. So I merely removed this. Still can't connect to the X server for some reason.. - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Tue, 26 Feb 2008 13:29:28 -0500Looks like a corrupted/uninitialized type (method table contains a NIL pointer...). Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 25, 2008, at 1:15 PM, Jay wrote: any obvious ideas? I'll have to pause for a bit again..Jay at jay-win1 /cygdrive/c/dev2/cm3.2/m3-ui/juno-2/juno-app/NT386GNU/Juno.exe****** runtime error:*** Thread client error: NIL apply method passed to Thread.Fork!*** file "ThreadWin32.m3", line 578*** 14837 [sig] Juno 3908 c:\dev2\cm3.2\m3-ui\juno-2\juno-app\NT386GNU\Juno.exe: *** fatal error - called with threadlist_ix -1Hangup - Jay Need to know the score, the latest news, or you need your Hotmail?-get your "fix". Check it out. _________________________________________________________________ Shed those extra pounds with MSN and The Biggest Loser! http://biggestloser.msn.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Fri Mar 7 08:57:03 2008 From: jayk123 at hotmail.com (Jay) Date: Fri, 7 Mar 2008 07:57:03 +0000 Subject: [M3devel] multi arch installs? Message-ID: background: Windows has $PATH and $PATHEXT.$PATHEXT is used by the usual command line -- cmd -- to help find "executables". On one of my systems: C:\>set pathPath=C:\Python25\;C:\Tcl\bin;C:\Perl\site\bin;C:\Perl\bin;C:\Perl\bin;c:\ruby\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\QuickTime\QTSystem\PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.RB;.RBW;.tcl;.py;.pyw (oops, I'll go clean it up..) "Executables" are not just .exes. The commands ftype and assoc combine to enable running "anything", based on extension. For example: C:\>assoc .exe.exe=exefile C:\>ftype exefileexefile="%1" %* C:\>assoc .js.js=JSFile C:\>assoc .vbs.vbs=VBSFile C:\>assoc .py.py=Python.File C:\>assoc .tcl.tcl=ActiveTclScript C:\>ftype activetclscriptactivetclscript="C:\Tcl\bin\wish85.exe" "%1" %* C:\>ftype python.filepython.file="C:\Python25\python.exe" "%1" %* C:\>ftype jsfilejsfile=%SystemRoot%\System32\WScript.exe "%1" %* C:\>ftype vbsfilevbsfile=%SystemRoot%\System32\WScript.exe "%1" %* If you ignore the need for executables to have extensions, this provides much ofthe feature of Unix where any file is executable, possibly via a specified interpreter.It is better and worse.It is worse in that extensions are needed.It is better in that files don't have to give the possibly system-specific pathto their interpreter.It is worse in that files cannot give the path to the interpreter. It is worse in that it is a feature of the command line interpreter, instead of lower layers (kernel) as I believe on Unix. Now, I have a small problem of convenience.And you all MIGHT have a "problem" of "What is Posix?" My c:\cm3 contains a mix of NT386 and NT386GNU files. c:\cm3\pkg\cm3\NT386\cm3.exe c:\cm3\pkg\cm3\NT386GNU\cm3.exe ok but then what is c:\cm3\bin\cm3.exe ? Currently it is the most recently shipped file. For the case of .dlls, I have enabled "colocation" by changing NT386GNU .dlls to end in .so.So I have: c:\cm3\bin\m3.dll c:\cm3\bin\m3.so Probably this should be libm3.so and probably will be some day. I would a system whereby somehow the two targets can coexist better. Perhaps I should just suck it up and have c:\cm3\nt386 and c:\cm3\nt386gnu.Perhaps. Quick experimentation shows that you can have an "empty" extension. set PATHEXT=.;%PATHEXT% cd \cygwin\bin copy ls.exe foo .\foo => It works. You don't actually have to do anything with assoc and ftype. Oops. I didn't have to bring that up. The order of %PATHEXT% varies. So, the proposal is, in the interest of killing two birds with one stone, neither of whichwas necessarily even alive at the start, is that NT386GNU use no extension for .exes. \cm3\bin\cm3 is NT386GNU \cm3\bin\cm3.exe is NT386 (or NT386MINGNU) I started thinking about this partly because "Services for Unix" makes .exes with no extension.And because I suspect the handling here in Cygwin is hacky.There is various code such that if someone looks for "foo", they might find "foo.exe" instead. Upsides: "It works". Downsides: "foo" used to resolve via %PATH% + %PATHEXT% search..and it still does, but putting . at the start will find foo where foo.exe was more common. It is not general. This scheme does not enable any other "multi arch" install. Alternate proposal with a question: How should "multi arch" installs be achieved? Probably via making \cm3\bin\foo on Posix and \cm3\bin\foo.cmd on NT that somehow sniffs its environment, perhaps with uname, and then runs \cm3\pkg\foo\?\foo (or foo.exe) Upsides: It works, at least for executables. It is more general than my previous suggestion. Downsides: Extra processes created. What about .dlls/.sos? Maybe nobody cares? I find the idea of one unified file system/archive that works on "everything" fairly compelling. Yeah, it's big. Yeah, it's slower to install. But it means one download can be had independently of platform. Maybe simply \cm3\platform instead of \cm3? upsides: it works is general no new mechisms invented! downsides: none? too boring? :) \cm3\platform\pkg\foo\platform is redundant. Less able to do "cross" scenarios that I'm using a lot lately, where NT386 and NT386GNU can build each other? Not clear. Probably just a matter of $PATH and $CM3_INSTALL not coinciding. cm3 -ship could see if CM3_INSTALL ends in "\platform" and in that case not put in the second instance, but this possibly asking for trouble. Probably a bunch of code would have to learn about this. Just live with the duplication of "platform", in the unusual situation of a "multi arch" file system? Maybe there is nothing to do here. The matter of what the file extension should be on NT386GNU perhaps is a different matter, left up to the hypothetical users of NT386GNU (not me). "What is Posix?" Executables with no extension? Or don't care? Cygwin usually uses ".exe", but of course Cygwin bash allows running stuff with "shebang". Microsoft/Interix "Services for Unix" puts no extension on executables. I found that a bit impressive, maybe worth striving for, and then surprisingly easy. I am almost surprised Cygwin doesn't do this. I don't know if putting "." in $PATHEXT is any worse than adding \cygwin\bin to $PATH. "If it is good enough for Cygwin, it is good enough for CM3/NT386GNU users?" I am not proposing extensionless executables for NT386. That would just be considered wierd. However the "wrapper .cmd" files, maybe. - Jay _________________________________________________________________ Shed those extra pounds with MSN and The Biggest Loser! http://biggestloser.msn.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Fri Mar 7 14:41:23 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 7 Mar 2008 08:41:23 -0500 Subject: [M3devel] thread.fork getting nil on nt386gnu? In-Reply-To: References: <20080224134642.48e7fowms8ooowo8@mail.elegosoft.com> <20080224143053.fdtxd48fkskggws0@mail.elegosoft.com> <20080224153541.59c105ro8e88480g@mail.elegosoft.com> <6346DE25-6495-49D0-A990-F48B989218AC@cs.purdue.edu> Message-ID: <5F9985B2-99AE-4C63-8624-3A33F9BD0569@cs.purdue.edu> It is always dangerous to remove assertions that have worked in the past. Better to understand why they no longer work. Anyway, what this suggests is something in alignment because what the assertion checks is that the method table for the Thread.Closure object passed to Thread.Fork contains a non-NIL function pointer in the "apply" slot. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 7, 2008, at 1:00 AM, Jay wrote: > This asssertion is absent from the two other thread implementations, > and is dependent on a loophole to underlying details that I didn't > track down to determine if they are corrrect. So I merely removed > this. > > Still can't connect to the X server for some reason.. > > - Jay > CC: m3devel at elegosoft.com > From: hosking at cs.purdue.edu > To: jayk123 at hotmail.com > Subject: Re: [M3devel] thread.fork getting nil on nt386gnu? > Date: Tue, 26 Feb 2008 13:29:28 -0500 > > Looks like a corrupted/uninitialized type (method table contains a > NIL pointer...). > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > On Feb 25, 2008, at 1:15 PM, Jay wrote: > > any obvious ideas? I'll have to pause for a bit again.. > > Jay at jay-win1 /cygdrive/c/dev2/cm3.2/m3-ui/juno-2/juno-app/NT386GNU/ > Juno.exe > > *** > *** runtime error: > *** Thread client error: NIL apply method passed to Thread.Fork! > *** file "ThreadWin32.m3", line 578 > *** > 14837 [sig] Juno 3908 c:\dev2\cm3.2\m3-ui\juno-2\juno-app\NT386GNU > \Juno.exe: * > ** fatal error - called with threadlist_ix -1 > Hangup > > - Jay > > Need to know the score, the latest news, or you need your Hotmail?- > get your "fix". Check it out. > > > Shed those extra pounds with MSN and The Biggest Loser! Learn more. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Fri Mar 7 20:21:38 2008 From: jayk123 at hotmail.com (Jay) Date: Fri, 7 Mar 2008 19:21:38 +0000 Subject: [M3devel] thread.fork getting nil on nt386gnu? In-Reply-To: <5F9985B2-99AE-4C63-8624-3A33F9BD0569@cs.purdue.edu> References: <20080224134642.48e7fowms8ooowo8@mail.elegosoft.com> <20080224143053.fdtxd48fkskggws0@mail.elegosoft.com> <20080224153541.59c105ro8e88480g@mail.elegosoft.com> <6346DE25-6495-49D0-A990-F48B989218AC@cs.purdue.edu> <5F9985B2-99AE-4C63-8624-3A33F9BD0569@cs.purdue.edu> Message-ID: assertions are also notoriously wrong. Two out of three thread implementations don't have it.. - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Fri, 7 Mar 2008 08:41:23 -0500 It is always dangerous to remove assertions that have worked in the past. Better to understand why they no longer work. Anyway, what this suggests is something in alignment because what the assertion checks is that the method table for the Thread.Closure object passed to Thread.Fork contains a non-NIL function pointer in the "apply" slot. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 7, 2008, at 1:00 AM, Jay wrote: This asssertion is absent from the two other thread implementations, and is dependent on a loophole to underlying details that I didn't track down to determine if they are corrrect. So I merely removed this. Still can't connect to the X server for some reason.. - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Tue, 26 Feb 2008 13:29:28 -0500Looks like a corrupted/uninitialized type (method table contains a NIL pointer...). Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 25, 2008, at 1:15 PM, Jay wrote: any obvious ideas? I'll have to pause for a bit again..Jay at jay-win1 /cygdrive/c/dev2/cm3.2/m3-ui/juno-2/juno-app/NT386GNU/Juno.exe****** runtime error:*** Thread client error: NIL apply method passed to Thread.Fork!*** file "ThreadWin32.m3", line 578*** 14837 [sig] Juno 3908 c:\dev2\cm3.2\m3-ui\juno-2\juno-app\NT386GNU\Juno.exe: *** fatal error - called with threadlist_ix -1Hangup - Jay Need to know the score, the latest news, or you need your Hotmail?-get your "fix". Check it out. Shed those extra pounds with MSN and The Biggest Loser! Learn more. _________________________________________________________________ Connect and share in new ways with Windows Live. http://www.windowslive.com/share.html?ocid=TXT_TAGHM_Wave2_sharelife_012008 -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Fri Mar 7 20:48:29 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 7 Mar 2008 14:48:29 -0500 Subject: [M3devel] thread.fork getting nil on nt386gnu? In-Reply-To: References: <20080224134642.48e7fowms8ooowo8@mail.elegosoft.com> <20080224143053.fdtxd48fkskggws0@mail.elegosoft.com> <20080224153541.59c105ro8e88480g@mail.elegosoft.com> <6346DE25-6495-49D0-A990-F48B989218AC@cs.purdue.edu> <5F9985B2-99AE-4C63-8624-3A33F9BD0569@cs.purdue.edu> Message-ID: <4CF63A9C-93DD-486D-825C-A07630AD13D6@cs.purdue.edu> Umm. If the assertion is wrong then how has this code ever run before... On Mar 7, 2008, at 2:21 PM, Jay wrote: > assertions are also notoriously wrong. > Two out of three thread implementations don't have it.. > > - Jay > > CC: m3devel at elegosoft.com > From: hosking at cs.purdue.edu > To: jayk123 at hotmail.com > Subject: Re: [M3devel] thread.fork getting nil on nt386gnu? > Date: Fri, 7 Mar 2008 08:41:23 -0500 > > It is always dangerous to remove assertions that have worked in the > past. Better to understand why they no longer work. Anyway, what > this suggests is something in alignment because what the assertion > checks is that the method table for the Thread.Closure object passed > to Thread.Fork contains a non-NIL function pointer in the "apply" > slot. > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > On Mar 7, 2008, at 1:00 AM, Jay wrote: > > This asssertion is absent from the two other thread implementations, > and is dependent on a loophole to underlying details that I didn't > track down to determine if they are corrrect. So I merely removed > this. > > Still can't connect to the X server for some reason.. > > - Jay > CC: m3devel at elegosoft.com > From: hosking at cs.purdue.edu > To: jayk123 at hotmail.com > Subject: Re: [M3devel] thread.fork getting nil on nt386gnu? > Date: Tue, 26 Feb 2008 13:29:28 -0500 > > Looks like a corrupted/uninitialized type (method table contains a > NIL pointer...). > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > On Feb 25, 2008, at 1:15 PM, Jay wrote: > > any obvious ideas? I'll have to pause for a bit again.. > > Jay at jay-win1 /cygdrive/c/dev2/cm3.2/m3-ui/juno-2/juno-app/NT386GNU/ > Juno.exe > > *** > *** runtime error: > *** Thread client error: NIL apply method passed to Thread.Fork! > *** file "ThreadWin32.m3", line 578 > *** > 14837 [sig] Juno 3908 c:\dev2\cm3.2\m3-ui\juno-2\juno-app\NT386GNU > \Juno.exe: * > ** fatal error - called with threadlist_ix -1 > Hangup > > - Jay > > Need to know the score, the latest news, or you need your Hotmail?- > get your "fix". Check it out. > > > Shed those extra pounds with MSN and The Biggest Loser! Learn more. > > > Connect and share in new ways with Windows Live. Get it now! -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Fri Mar 7 22:38:58 2008 From: jayk123 at hotmail.com (Jay) Date: Fri, 7 Mar 2008 21:38:58 +0000 Subject: [M3devel] thread.fork getting nil on nt386gnu? In-Reply-To: <4CF63A9C-93DD-486D-825C-A07630AD13D6@cs.purdue.edu> References: <20080224134642.48e7fowms8ooowo8@mail.elegosoft.com> <20080224143053.fdtxd48fkskggws0@mail.elegosoft.com> <20080224153541.59c105ro8e88480g@mail.elegosoft.com> <6346DE25-6495-49D0-A990-F48B989218AC@cs.purdue.edu> <5F9985B2-99AE-4C63-8624-3A33F9BD0569@cs.purdue.edu> <4CF63A9C-93DD-486D-825C-A07630AD13D6@cs.purdue.edu> Message-ID: Assertions, like a lot of code, often succeed by luck. There's little point in asserting a pointer isn't null right before dereferencing it. You are going to crash on one line or the next. - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Fri, 7 Mar 2008 14:48:29 -0500Umm. If the assertion is wrong then how has this code ever run before... On Mar 7, 2008, at 2:21 PM, Jay wrote: assertions are also notoriously wrong.Two out of three thread implementations don't have it.. - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Fri, 7 Mar 2008 08:41:23 -0500 It is always dangerous to remove assertions that have worked in the past. Better to understand why they no longer work. Anyway, what this suggests is something in alignment because what the assertion checks is that the method table for the Thread.Closure object passed to Thread.Fork contains a non-NIL function pointer in the "apply" slot. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 7, 2008, at 1:00 AM, Jay wrote: This asssertion is absent from the two other thread implementations, and is dependent on a loophole to underlying details that I didn't track down to determine if they are corrrect. So I merely removed this. Still can't connect to the X server for some reason.. - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Tue, 26 Feb 2008 13:29:28 -0500Looks like a corrupted/uninitialized type (method table contains a NIL pointer...). Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 25, 2008, at 1:15 PM, Jay wrote: any obvious ideas? I'll have to pause for a bit again..Jay at jay-win1 /cygdrive/c/dev2/cm3.2/m3-ui/juno-2/juno-app/NT386GNU/Juno.exe****** runtime error:*** Thread client error: NIL apply method passed to Thread.Fork!*** file "ThreadWin32.m3", line 578*** 14837 [sig] Juno 3908 c:\dev2\cm3.2\m3-ui\juno-2\juno-app\NT386GNU\Juno.exe: *** fatal error - called with threadlist_ix -1Hangup - Jay Need to know the score, the latest news, or you need your Hotmail?-get your "fix". Check it out. Shed those extra pounds with MSN and The Biggest Loser! Learn more. Connect and share in new ways with Windows Live. Get it now! _________________________________________________________________ Shed those extra pounds with MSN and The Biggest Loser! http://biggestloser.msn.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Sat Mar 8 00:17:55 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 7 Mar 2008 18:17:55 -0500 Subject: [M3devel] thread.fork getting nil on nt386gnu? In-Reply-To: References: <20080224134642.48e7fowms8ooowo8@mail.elegosoft.com> <20080224143053.fdtxd48fkskggws0@mail.elegosoft.com> <20080224153541.59c105ro8e88480g@mail.elegosoft.com> <6346DE25-6495-49D0-A990-F48B989218AC@cs.purdue.edu> <5F9985B2-99AE-4C63-8624-3A33F9BD0569@cs.purdue.edu> <4CF63A9C-93DD-486D-825C-A07630AD13D6@cs.purdue.edu> Message-ID: This wasn't an ASSERT assertion. It was an explicit check that the apply method was not null, along with a helpful message to the user. On Mar 7, 2008, at 4:38 PM, Jay wrote: > Assertions, like a lot of code, often succeed by luck. > There's little point in asserting a pointer isn't null right before > dereferencing it. You are going to crash on one line or the next. > > - Jay > > CC: m3devel at elegosoft.com > From: hosking at cs.purdue.edu > To: jayk123 at hotmail.com > Subject: Re: [M3devel] thread.fork getting nil on nt386gnu? > Date: Fri, 7 Mar 2008 14:48:29 -0500 > > Umm. If the assertion is wrong then how has this code ever run > before... > > On Mar 7, 2008, at 2:21 PM, Jay wrote: > > assertions are also notoriously wrong. > Two out of three thread implementations don't have it.. > > - Jay > > CC: m3devel at elegosoft.com > From: hosking at cs.purdue.edu > To: jayk123 at hotmail.com > Subject: Re: [M3devel] thread.fork getting nil on nt386gnu? > Date: Fri, 7 Mar 2008 08:41:23 -0500 > > It is always dangerous to remove assertions that have worked in the > past. Better to understand why they no longer work. Anyway, what > this suggests is something in alignment because what the assertion > checks is that the method table for the Thread.Closure object passed > to Thread.Fork contains a non-NIL function pointer in the "apply" > slot. > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > On Mar 7, 2008, at 1:00 AM, Jay wrote: > > This asssertion is absent from the two other thread implementations, > and is dependent on a loophole to underlying details that I didn't > track down to determine if they are corrrect. So I merely removed > this. > > Still can't connect to the X server for some reason.. > > - Jay > CC: m3devel at elegosoft.com > From: hosking at cs.purdue.edu > To: jayk123 at hotmail.com > Subject: Re: [M3devel] thread.fork getting nil on nt386gnu? > Date: Tue, 26 Feb 2008 13:29:28 -0500 > > Looks like a corrupted/uninitialized type (method table contains a > NIL pointer...). > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > On Feb 25, 2008, at 1:15 PM, Jay wrote: > > any obvious ideas? I'll have to pause for a bit again.. > > Jay at jay-win1 /cygdrive/c/dev2/cm3.2/m3-ui/juno-2/juno-app/NT386GNU/ > Juno.exe > > *** > *** runtime error: > *** Thread client error: NIL apply method passed to Thread.Fork! > *** file "ThreadWin32.m3", line 578 > *** > 14837 [sig] Juno 3908 c:\dev2\cm3.2\m3-ui\juno-2\juno-app\NT386GNU > \Juno.exe: * > ** fatal error - called with threadlist_ix -1 > Hangup > > - Jay > > Need to know the score, the latest news, or you need your Hotmail?- > get your "fix". Check it out. > > > Shed those extra pounds with MSN and The Biggest Loser! Learn more. > > > Connect and share in new ways with Windows Live. Get it now! > > > Shed those extra pounds with MSN and The Biggest Loser! Learn more. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Sat Mar 8 01:57:37 2008 From: jayk123 at hotmail.com (Jay) Date: Sat, 8 Mar 2008 00:57:37 +0000 Subject: [M3devel] thread.fork getting nil on nt386gnu? In-Reply-To: References: <20080224134642.48e7fowms8ooowo8@mail.elegosoft.com> <20080224143053.fdtxd48fkskggws0@mail.elegosoft.com> <20080224153541.59c105ro8e88480g@mail.elegosoft.com> <6346DE25-6495-49D0-A990-F48B989218AC@cs.purdue.edu> <5F9985B2-99AE-4C63-8624-3A33F9BD0569@cs.purdue.edu> <4CF63A9C-93DD-486D-825C-A07630AD13D6@cs.purdue.edu> Message-ID: Same thing.. I should look into what the underlying representation is and why it changed from "working" to non working. Note that this code runs about three times in the relevant scenario and only "fails" the third time, but yet, I'm pretty sure but will double check, the error message is plain wrong. I think I saw it run the thread function, it wasn't actually NULL. I'll have to double check that though. Imho, it could /easily/ be inspecting other nearby data that just happens to usually not be zero on NT386 but happens to sometimes be zero on NT386GNU. Without commiting the change, running the same code on pthreads/alarmthreads on other platforms might be "interesting". I wouldn't bet either way. I find there are bugs /everywhere/. Code that is trying to find bugs is no more immune from bugs than the code it is attempting to check. More code leads to more bugs. More asserts lead to more incorrect asserts. More runtime checks lead to more incorrect runtime checks. etc. Of course, checks are still good, and like everything else they must be written carefully and like everything else, when things go apparently wrong, pretty much all code should be assumed wrong until proven otherwise (read it, run it, read what it depends on, etc.). Code is guilty until proven innocent, many times. One rotten apple spoils the bunch.... - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Fri, 7 Mar 2008 18:17:55 -0500 This wasn't an ASSERT assertion. It was an explicit check that the apply method was not null, along with a helpful message to the user. On Mar 7, 2008, at 4:38 PM, Jay wrote: Assertions, like a lot of code, often succeed by luck.There's little point in asserting a pointer isn't null right before dereferencing it. You are going to crash on one line or the next. - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Fri, 7 Mar 2008 14:48:29 -0500Umm. If the assertion is wrong then how has this code ever run before... On Mar 7, 2008, at 2:21 PM, Jay wrote: assertions are also notoriously wrong.Two out of three thread implementations don't have it.. - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Fri, 7 Mar 2008 08:41:23 -0500 It is always dangerous to remove assertions that have worked in the past. Better to understand why they no longer work. Anyway, what this suggests is something in alignment because what the assertion checks is that the method table for the Thread.Closure object passed to Thread.Fork contains a non-NIL function pointer in the "apply" slot. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 7, 2008, at 1:00 AM, Jay wrote: This asssertion is absent from the two other thread implementations, and is dependent on a loophole to underlying details that I didn't track down to determine if they are corrrect. So I merely removed this. Still can't connect to the X server for some reason.. - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Tue, 26 Feb 2008 13:29:28 -0500Looks like a corrupted/uninitialized type (method table contains a NIL pointer...). Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 25, 2008, at 1:15 PM, Jay wrote: any obvious ideas? I'll have to pause for a bit again..Jay at jay-win1 /cygdrive/c/dev2/cm3.2/m3-ui/juno-2/juno-app/NT386GNU/Juno.exe****** runtime error:*** Thread client error: NIL apply method passed to Thread.Fork!*** file "ThreadWin32.m3", line 578*** 14837 [sig] Juno 3908 c:\dev2\cm3.2\m3-ui\juno-2\juno-app\NT386GNU\Juno.exe: *** fatal error - called with threadlist_ix -1Hangup - Jay Need to know the score, the latest news, or you need your Hotmail?-get your "fix". Check it out. Shed those extra pounds with MSN and The Biggest Loser! Learn more. Connect and share in new ways with Windows Live. Get it now! Shed those extra pounds with MSN and The Biggest Loser! Learn more. _________________________________________________________________ Helping your favorite cause is as easy as instant messaging.?You IM, we give. http://im.live.com/Messenger/IM/Home/?source=text_hotmail_join -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Sat Mar 8 02:23:18 2008 From: jayk123 at hotmail.com (Jay) Date: Sat, 8 Mar 2008 01:23:18 +0000 Subject: [M3devel] FW: thread.fork getting nil on nt386gnu? In-Reply-To: References: <20080224134642.48e7fowms8ooowo8@mail.elegosoft.com> <20080224143053.fdtxd48fkskggws0@mail.elegosoft.com> <20080224153541.59c105ro8e88480g@mail.elegosoft.com> <6346DE25-6495-49D0-A990-F48B989218AC@cs.purdue.edu> <5F9985B2-99AE-4C63-8624-3A33F9BD0569@cs.purdue.edu> <4CF63A9C-93DD-486D-825C-A07630AD13D6@cs.purdue.edu> Message-ID: My mail keeps getting truncated. I'll at least pay for hotmail to remove the annoying ads, maybe that'll help.. From: jayk123 at hotmail.com Same thing..I should look into what the underlying representation is and why it changed from "working" to non working.Note that this code runs about three times in the relevant scenario and only "fails" the third time, but yet, I'm pretty sure but will double check, the error message is plain wrong. I think I saw it run the thread function, it wasn't actually NULL. I'll have to double check that though. Imho, it could /easily/ be inspecting other nearby data that just happens to usually not be zero on NT386 but happens to sometimes be zero on NT386GNU. Without commiting the change, running the same code on pthreads/alarmthreads on other platforms might be "interesting". I wouldn't bet either way. I find there are bugs /everywhere/. Code that is trying to find bugs is no more immune from bugs than the code it is attempting to check. More code leads to more bugs. More asserts lead to more incorrect asserts. More runtime checks lead to more incorrect runtime checks. etc. Of course, checks are still good, and like everything else they must be written carefully and like everything else, when things go apparently wrong, pretty much all code should be assumed wrong until proven otherwise (read it, run it, read what it depends on, etc.). Code is guilty until proven innocent, many times. One rotten apple spoils the bunch.... - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Fri, 7 Mar 2008 18:17:55 -0500 This wasn't an ASSERT assertion. It was an explicit check that the apply method was not null, along with a helpful message to the user. On Mar 7, 2008, at 4:38 PM, Jay wrote: Assertions, like a lot of code, often succeed by luck.There's little point in asserting a pointer isn't null right before dereferencing it. You are going to crash on one line or the next. - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Fri, 7 Mar 2008 14:48:29 -0500Umm. If the assertion is wrong then how has this code ever run before... On Mar 7, 2008, at 2:21 PM, Jay wrote: assertions are also notoriously wrong.Two out of three thread implementations don't have it.. - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Fri, 7 Mar 2008 08:41:23 -0500 It is always dangerous to remove assertions that have worked in the past. Better to understand why they no longer work. Anyway, what this suggests is something in alignment because what the assertion checks is that the method table for the Thread.Closure object passed to Thread.Fork contains a non-NIL function pointer in the "apply" slot. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 7, 2008, at 1:00 AM, Jay wrote: This asssertion is absent from the two other thread implementations, and is dependent on a loophole to underlying details that I didn't track down to determine if they are corrrect. So I merely removed this. Still can't connect to the X server for some reason.. - Jay CC: m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] thread.fork getting nil on nt386gnu?Date: Tue, 26 Feb 2008 13:29:28 -0500Looks like a corrupted/uninitialized type (method table contains a NIL pointer...). Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 25, 2008, at 1:15 PM, Jay wrote: any obvious ideas? I'll have to pause for a bit again..Jay at jay-win1 /cygdrive/c/dev2/cm3.2/m3-ui/juno-2/juno-app/NT386GNU/Juno.exe****** runtime error:*** Thread client error: NIL apply method passed to Thread.Fork!*** file "ThreadWin32.m3", line 578*** 14837 [sig] Juno 3908 c:\dev2\cm3.2\m3-ui\juno-2\juno-app\NT386GNU\Juno.exe: *** fatal error - called with threadlist_ix -1Hangup - Jay _________________________________________________________________ Helping your favorite cause is as easy as instant messaging.?You IM, we give. http://im.live.com/Messenger/IM/Home/?source=text_hotmail_join -------------- next part -------------- An HTML attachment was scrubbed... URL: From rodney.bates at wichita.edu Sun Mar 9 01:19:32 2008 From: rodney.bates at wichita.edu (Rodney M. Bates) Date: Sat, 08 Mar 2008 18:19:32 -0600 Subject: [M3devel] Assertions (Was FW: thread.fork getting nil on nt386gnu?) In-Reply-To: References: <20080224134642.48e7fowms8ooowo8@mail.elegosoft.com> <20080224143053.fdtxd48fkskggws0@mail.elegosoft.com> <20080224153541.59c105ro8e88480g@mail.elegosoft.com> <6346DE25-6495-49D0-A990-F48B989218AC@cs.purdue.edu> <5F9985B2-99AE-4C63-8624-3A33F9BD0569@cs.purdue.edu> <4CF63A9C-93DD-486D-825C-A07630AD13D6@cs.purdue.edu> Message-ID: <47D32D14.8020501@wichita.edu> Jay wrote: > I find there are bugs /everywhere/. Code that is trying to find bugs is > no more immune from bugs than the code it is attempting to check. More > code leads to more bugs. More asserts lead to more incorrect asserts. > More runtime checks lead to more incorrect runtime checks. etc. Of > course, checks are still good, and like everything else they must be > written carefully and like everything else, when things go apparently > wrong, pretty much all code should be assumed wrong until proven > otherwise (read it, run it, read what it depends on, etc.). Code is > guilty until proven innocent, many times. One rotten apple spoils the > bunch.... > > Well, here is my documented experience. For more than 20 years, I have kept detailed logs of every bug in most programming projects of significant size. This is from a project in Modula-3 that I have worked on and off on for years, with around 105K SLOC, From about 15% of the work for which I kept compile-time error data, 75% of all programming errors were found at compile time. This is a count of required changes to the code, not of compile time messages, which, of course, usually greatly outnumber the number of actual errors. Where an equivalent change needed to be made systematically in many places, I count this as one change. No doubt records from earlier in the project would show a higher percentage of compile-time errors. I have kept logs for all of the errors discovered at runtime. From about 40% of the more recent of these logs that are already classified and tallied, the breakdown of how the errors manifested is; Type of error % of all % of errors found errors at runtime Compile time 75 % N/A Failures of correct assertions 10 % 40 % Runtime errors 6.5 % 26 % Observed bad output/behavior 6.5 % 26 % Runaway loops/recursion 1.03 % 4.10 % Bad internal values, observed in a .42 % 1.68 % debugger or debug output Overzealous assertions .37 % 1.47 % Uncaught exceptions .34 % 1.36 % Synchronization/deadlock, etc. .11 % .42 % By "overzealous assertions", I mean assertions that assert something that they should not, and thus crash the program unnecessarily. Instances of assertions finding real bugs outnumber overzealous assertions more than 25 to one. A very good tradeoff, I think. The valid assertion failures were more numerous earlier in the project. Finding bugs by assertion failure also has the big advantage that it does not require verifying correct output/behavior, or even deciding manually what is correct. Of course, such results depend on the programmer's style of deciding what to assert. The count of assertions coded in this program is 652. YMMV. -- ------------------------------------------------------------- Rodney M. Bates, retired assistant professor Dept. of Computer Science, Wichita State University Wichita, KS 67260-0083 316-978-3922 rodney.bates at wichita.edu From wagner at elegosoft.com Sun Mar 9 20:11:33 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Sun, 09 Mar 2008 20:11:33 +0100 Subject: [M3devel] RuntimeException catching broken? Re: assertion failure in RTCollector In-Reply-To: <81E268D8-B32A-45F0-A14D-61D7B833F4B2@cs.purdue.edu> References: <20080220170733.e5h5zpdlkw80s4ks@mail.elegosoft.com> <20080220185655.ll7dysf07swo4g04@mail.elegosoft.com> <3C31A48B-E365-41D3-956D-9069BDC028D9@cs.purdue.edu> <20080309120433.0vi02calq8sw0kc0@mail.elegosoft.com> <2501041B-CB64-40B3-8FCE-6485EB52C10F@cs.purdue.edu> <20080309190636.kl4fvqcqm84cgw8c@mail.elegosoft.com> <81E268D8-B32A-45F0-A14D-61D7B833F4B2@cs.purdue.edu> Message-ID: <20080309201133.ls6r99ads0c4sk0w@mail.elegosoft.com> Now for somethings completely different ;-) I've written this simple test program to verify Dragisa's report about not-working runtime exceptions: MODULE Main; IMPORT RuntimeError, IO, Process; CONST Tag = RuntimeError.Tag; PROCEDURE CatchAssert() = BEGIN TRY <*ASSERT FALSE*> EXCEPT RuntimeError.E( t ) => IO.Put( "OK: caught RuntimeError.Assert: " & Tag( t ) & "\n" ); END; END CatchAssert; PROCEDURE TestAll() = BEGIN CatchAssert(); END TestAll; BEGIN TRY TestAll(); EXCEPT ELSE IO.Put( "ERROR: caught unexpected exception\n" ); Process.Exit( 1 ); END; END Main. As he has reported, the exception is not caught. I've tried the debugger and set breakpoints at RTHooks.Raise() and RTHooks.AssertFailed(), but it did never stop there. Any ideas what's wrong there? You probably know better than me how exceptions are implemented. For today, I think I'll add that program as p208 to m3tests and then turn my attention to dinner :-) Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From wagner at elegosoft.com Mon Mar 10 00:33:05 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Mon, 10 Mar 2008 00:33:05 +0100 Subject: [M3devel] What is this??? How's it possible? In-Reply-To: <1204830152.8782.39.camel@hias.m3w.org> References: <1204818659.8782.16.camel@hias.m3w.org> <4AB9C403-1CF8-4621-8E90-9B5BB1F8B4B4@cs.purdue.edu> <20080306182319.u2flyt1uskg0owkg@mail.elegosoft.com> <1204830152.8782.39.camel@hias.m3w.org> Message-ID: <20080310003305.40ubkn1ke8s80scw@mail.elegosoft.com> Quoting Dragi?a Duri? : > Tried: > > TRY > RTLinker.AddUnit(LOOPHOLE(dl.sym(NARROW(r.get(i-1), TEXT) > & "_M3").addr, RT0.Binder)); > EXCEPT RuntimeError.E(expl) => > <* DEBUG "Error adding " & r.get(i-1) *> > END; > > It still fails with: > > *** > *** runtime error: > *** A compile-time type is missing. > *** file "../src/runtime/common/RTLinker.i3" > *** > > And RTType.Fail procedure, only one getting RuntimeError.MisingType > as argument, looks like this: > > PROCEDURE Fail (rte: RTE; m: RT0.ModulePtr; x, y: ADDRESS) = > <*FATAL ANY*> > ... > RTException.Raise (a); > END Fail; > > But, I am getting above mentioned error. Please retry with the latest m3core. I accidentally broke exception handling some weeks ago :-/ Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From wagner at elegosoft.com Mon Mar 10 00:38:29 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Mon, 10 Mar 2008 00:38:29 +0100 Subject: [M3devel] RuntimeException catching broken? Re: assertion failure in RTCollector In-Reply-To: <20080309201133.ls6r99ads0c4sk0w@mail.elegosoft.com> References: <20080220170733.e5h5zpdlkw80s4ks@mail.elegosoft.com> <20080220185655.ll7dysf07swo4g04@mail.elegosoft.com> <3C31A48B-E365-41D3-956D-9069BDC028D9@cs.purdue.edu> <20080309120433.0vi02calq8sw0kc0@mail.elegosoft.com> <2501041B-CB64-40B3-8FCE-6485EB52C10F@cs.purdue.edu> <20080309190636.kl4fvqcqm84cgw8c@mail.elegosoft.com> <81E268D8-B32A-45F0-A14D-61D7B833F4B2@cs.purdue.edu> <20080309201133.ls6r99ads0c4sk0w@mail.elegosoft.com> Message-ID: <20080310003829.ra46lb6k0soskwwk@mail.elegosoft.com> Please forget my last message. It seems I accidentally broke this myself some weeks ago while working on FreeBSD PTHREADS and committing too much (changes that were only temporary work-arounds). The test program should work again now. Quoting Olaf Wagner : > Now for somethings completely different ;-) > > I've written this simple test program to verify Dragisa's report > about not-working runtime exceptions: > > MODULE Main; > > IMPORT RuntimeError, IO, Process; > > CONST Tag = RuntimeError.Tag; > > PROCEDURE CatchAssert() = > BEGIN > TRY > <*ASSERT FALSE*> > EXCEPT > RuntimeError.E( t ) => > IO.Put( "OK: caught RuntimeError.Assert: " & Tag( t ) & "\n" ); > END; > END CatchAssert; > > PROCEDURE TestAll() = > BEGIN > CatchAssert(); > END TestAll; > > BEGIN > TRY > TestAll(); > EXCEPT > ELSE > IO.Put( "ERROR: caught unexpected exception\n" ); > Process.Exit( 1 ); > END; > END Main. > > As he has reported, the exception is not caught. I've tried the debugger > and set breakpoints at RTHooks.Raise() and RTHooks.AssertFailed(), but > it did never stop there. > > Any ideas what's wrong there? You probably know better than me how > exceptions are implemented. > > For today, I think I'll add that program as p208 to m3tests and then > turn my attention to dinner :-) > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From rcoleburn at scires.com Sat Mar 1 05:10:40 2008 From: rcoleburn at scires.com (Randy Coleburn) Date: Fri, 29 Feb 2008 23:10:40 -0500 Subject: [M3devel] Is the climate right for a new promotion? In-Reply-To: <47C45BB4.40009@wichita.edu> References: <47C45BB4.40009@wichita.edu> Message-ID: <47C890ED.1E75.00D7.1@scires.com> I agree! --Randy >>> "Rodney M. Bates" 2/26/2008 1:34 PM >>> I have been thinking lately that the climate might be right for a new attempt to promote Modula 3. I see two major trends and a small one that suggest this. Fist, probably nearly once week on the average, I see new articles that say that software developers are not prepared to utilize the now- proliferating multi-core processors. This actually greatly surprises me, because I had lots of exposure, decades ago, to all the concurrent programming stuff--threads, synchronization, scheduling, etc. But apparently, most programmers have been doing strictly sequential programming, and I have to admit, I myself only use concurrent programming techniques a minority of the time. In Modula-3, we have a language that already has better support than most, and certainly better than all the popular languages. Moreover, the implementation is out if front too, with Tony's recent thread and GC work for multi-processors. There are also a few articles out there that say, in effect, it is hopeless to implement threads and synchronization correctly as library code on top of existing C/C++. Second, the expansion of the internet and its vulnerability to automated attacks from anywhere in the world has greatly upped the ante on correctness of software. Before, it was enough if a program handled all the realistic cases that a reasonable user would present it with. Now, the whole theoretical space of inputs has to be handled with at least some level of correctness to avoid vulnerabilities. 1000-byte passwords are the standard example. All this means the limits of testing as a means of finding the bugs are a far greater obstacle than they once were. This makes the time right for another push toward static languages, which were somewhat of a popular flop the first time. Third, and perhaps rather more narrow, the developers of safety- critical software are reexamining the value of testing in the light of various close-calls, for example in airplanes. They are starting to look again at static methods. We have a language which, though little-known, is already designed and implemented and has a lot of convincing history. It has a lot of things that can be promoted. Aside from good thread support and the best safety properties around, it is much simpler than almost anything, yet more powerful than almost anything. A serious push would probably need the support of some large company or organization. But I think the climate is better than it has been in decades, and the technical arguments are very strong. -- ------------------------------------------------------------- Rodney M. Bates, retired assistant professor Dept. of Computer Science, Wichita State University Wichita, KS 67260-0083 316-978-3922 rodney.bates at wichita.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Sat Mar 8 04:32:49 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 7 Mar 2008 22:32:49 -0500 Subject: [M3devel] A start on Apple XCode 3.0 support for Modula-3 Message-ID: These go in ~/Library/Application Support/Developer/Shared/Xcode/ Specifications to get syntax highlighting and completion. No support for a lexer class to do proper indenting, etc., but at least this is a starting point. Building also still needs to be done with a Custom Build Tool Target set up to invoke cm3 in the package root directory. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: m3makefile.xclangspec Type: application/octet-stream Size: 3779 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: modula3.pbfilespec Type: application/octet-stream Size: 694 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: modula3.xclangspec Type: application/octet-stream Size: 4529 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 11 00:31:14 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 10 Mar 2008 19:31:14 -0400 Subject: [M3devel] FW: hand.c In-Reply-To: References: <995081.11151.qm@web27115.mail.ukl.yahoo.com> Message-ID: <25CBA31B-515C-480A-B51A-C4BDFED646EC@cs.purdue.edu> OK, weird. Now I'm getting e-mail from 6 months ago. Talk about deja vu. Any ideas what's going on? Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Sep 15, 2007, at 10:39 PM, Jay wrote: > I have since made some more of the changes. > > > > From: jayk123 at hotmail.com > To: m3devel at elegosoft.com > Subject: hand.c > Date: Mon, 3 Sep 2007 23:15:04 +0000 > > I'm not seeing commit mail. > Maybe my fault with the hotmail? > > I churned it relatively a lot. It's a small file. > It compiles with older Microsoft C compilers -- optionally using > __int64 instead of long long. (Visual C+ 8.0 worked as it was.) > I would have factored out the LL suffixes, but, really, the vast > majoriy, if not all, of the L and LL suffixes were unnecessary, right? > I removed most of them. > It is valid C++ now -- optional prototypes. > There was a race condition in initialization, fixed. > gcc -Wall complained about a safe signed/unsigned compare, fixed. > > Not changed: > all these longs should be unsigned long, right? > Or at least the bits, if not the input bit indices? > Negative bit indices should be either rejected or converted to 0, > and then treated as unsigned from there, right? > > _lowbits and _highbits should be > extern const ? > only on x86, right? I'm still checking if they are referenced > for all targets or only x86 > > HighBits and LowBits should be, like..ok switched to > #if LONG_MAX == 2147483647 > extern const .. hard coded 32 bit values ... > #elif LONG_MAX == 9223372036854775807 > extern const .. hard coded 64 bit values ... > #else > either compute them like currently or #error > #endif > > right? > > And if it is 32 bits, share the values of HighBits with _highbits, > LowBits with _lowbits? > They aren't /quite/ the same, but almost. _lowbits and _highbits > have one extra value, the zeros at the ends I think it was. > > - Jay > > Get news, entertainment and everything you care about at Live.com. > Check it out! > > Kick back and relax with hot games and cool activities at the > Messenger Caf?. Play now! -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Tue Mar 11 02:44:28 2008 From: jayk123 at hotmail.com (Jay) Date: Tue, 11 Mar 2008 01:44:28 +0000 Subject: [M3devel] FW: hand.c In-Reply-To: <25CBA31B-515C-480A-B51A-C4BDFED646EC@cs.purdue.edu> References: <995081.11151.qm@web27115.mail.ukl.yahoo.com> <25CBA31B-515C-480A-B51A-C4BDFED646EC@cs.purdue.edu> Message-ID: Invitation for an infinite stream of: "me too" also messages about "awaiting moderator approval". - Jay From: hosking at cs.purdue.eduTo: m3devel at elegosoft.comDate: Mon, 10 Mar 2008 19:31:14 -0400Subject: Re: [M3devel] FW: hand.c OK, weird. Now I'm getting e-mail from 6 months ago. Talk about deja vu. Any ideas what's going on? Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Sep 15, 2007, at 10:39 PM, Jay wrote: I have since made some more of the changes. From: jayk123 at hotmail.comTo: m3devel at elegosoft.comSubject: hand.cDate: Mon, 3 Sep 2007 23:15:04 +0000I'm not seeing commit mail.Maybe my fault with the hotmail? I churned it relatively a lot. It's a small file.It compiles with older Microsoft C compilers -- optionally using __int64 instead of long long. (Visual C+ 8.0 worked as it was.) I would have factored out the LL suffixes, but, really, the vast majoriy, if not all, of the L and LL suffixes were unnecessary, right? I removed most of them.It is valid C++ now -- optional prototypes.There was a race condition in initialization, fixed.gcc -Wall complained about a safe signed/unsigned compare, fixed. Not changed: all these longs should be unsigned long, right? Or at least the bits, if not the input bit indices? Negative bit indices should be either rejected or converted to 0, and then treated as unsigned from there, right? _lowbits and _highbits should be extern const ? only on x86, right? I'm still checking if they are referenced for all targets or only x86 HighBits and LowBits should be, like..ok switched to #if LONG_MAX == 2147483647 extern const .. hard coded 32 bit values ... #elif LONG_MAX == 9223372036854775807 extern const .. hard coded 64 bit values ... #else either compute them like currently or #error #endif right? And if it is 32 bits, share the values of HighBits with _highbits, LowBits with _lowbits?They aren't /quite/ the same, but almost. _lowbits and _highbits have one extra value, the zeros at the ends I think it was. - Jay Get news, entertainment and everything you care about at Live.com. Check it out! Kick back and relax with hot games and cool activities at the Messenger Caf?. Play now! _________________________________________________________________ Climb to the top of the charts!?Play the word scramble challenge with star power. http://club.live.com/star_shuffle.aspx?icid=starshuffle_wlmailtextlink_jan -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Tue Mar 11 09:03:09 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Tue, 11 Mar 2008 09:03:09 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead Message-ID: <1205222589.32152.27.camel@hias.m3w.org> I have single linked list that I use to send messages to many threads. It's end is locked for addition, and current end is given to each new client connecting. This way, all client are going towards end of the list so it's head becomes unreferenced and goes away by GC. Except it does not. I've added WeakRef cleanup and sequential id's so I can record maximum freed id for checking. No single cleanup happens. I'll cross check my code and count my references from application side, but maybe someone else has similar problems? dd -- Dragi?a Duri? From wagner at elegosoft.com Tue Mar 11 11:31:03 2008 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 11 Mar 2008 11:31:03 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <1205222589.32152.27.camel@hias.m3w.org> References: <1205222589.32152.27.camel@hias.m3w.org> Message-ID: <20080311113103.as92wmzolcws0csw@mail.elegosoft.com> Quoting Dragi?a Duri? : > I have single linked list that I use to send messages to many threads. > It's end is locked for addition, and current end is given to each new > client connecting. This way, all client are going towards end of the > list so it's head becomes unreferenced and goes away by GC. > > Except it does not. > > I've added WeakRef cleanup and sequential id's so I can record maximum > freed id for checking. No single cleanup happens. > > I'll cross check my code and count my references from application side, > but maybe someone else has similar problems? I'm not aware of any recent changes in this area. I could only assume that the behaviour of the garbage collector may have changed, as Antony Hosking has done considerable work on that, but I haven't noticed any general problems. (I think there's still a problem with generational collection and FreeBSD PTHREADs, but you're not using that, are you?) Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From hosking at cs.purdue.edu Tue Mar 11 14:50:27 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 11 Mar 2008 09:50:27 -0400 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <1205222589.32152.27.camel@hias.m3w.org> References: <1205222589.32152.27.camel@hias.m3w.org> Message-ID: Sounds like we need some sort of testcase. Would you be able to devise one? It will be hard to make it deterministic, but at least we should see a non-zero cleanup count. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: > I have single linked list that I use to send messages to many threads. > It's end is locked for addition, and current end is given to each new > client connecting. This way, all client are going towards end of the > list so it's head becomes unreferenced and goes away by GC. > > Except it does not. > > I've added WeakRef cleanup and sequential id's so I can record maximum > freed id for checking. No single cleanup happens. > > I'll cross check my code and count my references from application > side, > but maybe someone else has similar problems? > > dd > -- > Dragi?a Duri? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronny.forberger at elegosoft.com Tue Mar 11 16:49:03 2008 From: ronny.forberger at elegosoft.com (Ronny Forberger) Date: Tue, 11 Mar 2008 16:49:03 +0100 Subject: [M3devel] FW: hand.c In-Reply-To: <25CBA31B-515C-480A-B51A-C4BDFED646EC@cs.purdue.edu> References: <995081.11151.qm@web27115.mail.ukl.yahoo.com> <25CBA31B-515C-480A-B51A-C4BDFED646EC@cs.purdue.edu> Message-ID: <20080311164903.25piw31hus80s4sk@mail.elegosoft.com> The mailling list software had unfortunately held back these mails, as they could not be processed for some reason like not being able to contact the spam analyzer when sometimes that one gets restarted. It did not make any retry, so I had to send them out manually. :( Ronny -- Message from: Tony Hosking Date: Di 11 M?r 2008 00:31:14 CET Subject: Re: [M3devel] FW: hand.c > OK, weird. Now I'm getting e-mail from 6 months ago. Talk about deja > vu. Any ideas what's going on? > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > On Sep 15, 2007, at 10:39 PM, Jay wrote: > >> I have since made some more of the changes. >> >> >> >> From: jayk123 at hotmail.com >> To: m3devel at elegosoft.com >> Subject: hand.c >> Date: Mon, 3 Sep 2007 23:15:04 +0000 >> >> I'm not seeing commit mail. >> Maybe my fault with the hotmail? >> >> I churned it relatively a lot. It's a small file. >> It compiles with older Microsoft C compilers -- optionally using >> __int64 instead of long long. (Visual C+ 8.0 worked as it was.) >> I would have factored out the LL suffixes, but, really, the vast >> majoriy, if not all, of the L and LL suffixes were unnecessary, >> right? >> I removed most of them. >> It is valid C++ now -- optional prototypes. >> There was a race condition in initialization, fixed. >> gcc -Wall complained about a safe signed/unsigned compare, fixed. >> >> Not changed: >> all these longs should be unsigned long, right? >> Or at least the bits, if not the input bit indices? >> Negative bit indices should be either rejected or converted to 0, >> and then treated as unsigned from there, right? >> >> _lowbits and _highbits should be >> extern const ? >> only on x86, right? I'm still checking if they are referenced >> for all targets or only x86 >> >> HighBits and LowBits should be, like..ok switched to >> #if LONG_MAX == 2147483647 >> extern const .. hard coded 32 bit values ... >> #elif LONG_MAX == 9223372036854775807 >> extern const .. hard coded 64 bit values ... >> #else >> either compute them like currently or #error >> #endif >> >> right? >> >> And if it is 32 bits, share the values of HighBits with _highbits, >> LowBits with _lowbits? >> They aren't /quite/ the same, but almost. _lowbits and _highbits >> have one extra value, the zeros at the ends I think it was. >> >> - Jay >> >> Get news, entertainment and everything you care about at Live.com. >> Check it out! >> >> Kick back and relax with hot games and cool activities at the >> Messenger Caf?. Play now! -- Ronny Forberger Systemadministration & IT-Support elego Software Solutions GmbH Gustav-Meyer-Allee 25 Geb?ude 12, Raum 227 D-13355 Berlin Tel. +49 30 23 45 86 96 ronny.forberger at elegosoft.com Fax +49 30 23 45 86 95 http://www.elegosoft.com Gesch?ftsf?hrer: Olaf Wagner, Sitz Berlin Amtsgericht Berlin-Charlottenburg, HRB 77719, USt-IdNr: DE163214194 From dragisha at m3w.org Wed Mar 12 09:21:38 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Wed, 12 Mar 2008 09:21:38 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: References: <1205222589.32152.27.camel@hias.m3w.org> Message-ID: <1205310098.32152.43.camel@hias.m3w.org> Problem in my case can be thoroughness of GC... Because it's linked list of WeakRef-ed objects, it's always first element in list to become unreachable first - all others are reachable at least through previous (in list) WeakRef-ed object. So, behaviour I have observed can be because of some thousands of freeable objects, that one is missed too often. I have other object being cleaned and counted... There also cleanup invocation looks like it's lagging too much. I'll come with numbers after I have more tests. dd On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking wrote: > Sounds like we need some sort of testcase. Would you be able to > devise one? It will be hard to make it deterministic, but at least we > should see a non-zero cleanup count. > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: > > > I have single linked list that I use to send messages to many > > threads. > > It's end is locked for addition, and current end is given to each > > new > > client connecting. This way, all client are going towards end of the > > list so it's head becomes unreferenced and goes away by GC. > > > > Except it does not. > > > > I've added WeakRef cleanup and sequential id's so I can record > > maximum > > freed id for checking. No single cleanup happens. > > > > I'll cross check my code and count my references from application > > side, > > but maybe someone else has similar problems? > > > > dd > > -- > > Dragi?a Duri? > > > -- Dragi?a Duri? From hosking at cs.purdue.edu Wed Mar 12 16:12:57 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 12 Mar 2008 11:12:57 -0400 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <1205310098.32152.43.camel@hias.m3w.org> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> Message-ID: <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> This is probably a result of conservatism for references from thread stacks which results in spurious retention. It may be necessary to be more careful about what references are held on the stack in the threads implementation (and elsewhere in your code). Until we diagnose the place where objects are being retained it will be hard to pinpoint. You should realize that weak refs are a problem in many language implementations, not just Modula-3. I look forward to hearing more definitively from you. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: > Problem in my case can be thoroughness of GC... Because it's linked > list of WeakRef-ed objects, it's always first element in list to > become > unreachable first - all others are reachable at least through previous > (in list) WeakRef-ed object. So, behaviour I have observed can be > because of some thousands of freeable objects, that one is missed too > often. > > I have other object being cleaned and counted... There also cleanup > invocation looks like it's lagging too much. I'll come with numbers > after I have more tests. > > dd > > On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking wrote: >> Sounds like we need some sort of testcase. Would you be able to >> devise one? It will be hard to make it deterministic, but at least >> we >> should see a non-zero cleanup count. >> >> Antony Hosking | Associate Professor | Computer Science | Purdue >> University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> >> >> On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: >> >>> I have single linked list that I use to send messages to many >>> threads. >>> It's end is locked for addition, and current end is given to each >>> new >>> client connecting. This way, all client are going towards end of the >>> list so it's head becomes unreferenced and goes away by GC. >>> >>> Except it does not. >>> >>> I've added WeakRef cleanup and sequential id's so I can record >>> maximum >>> freed id for checking. No single cleanup happens. >>> >>> I'll cross check my code and count my references from application >>> side, >>> but maybe someone else has similar problems? >>> >>> dd >>> -- >>> Dragi?a Duri? >>> >> > -- > Dragi?a Duri? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Thu Mar 13 11:16:36 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 13 Mar 2008 11:16:36 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> Message-ID: <1205403396.32152.96.camel@hias.m3w.org> What happens to thread stacks once threads are Join-ed? dd On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking wrote: > This is probably a result of conservatism for references from thread > stacks which results in spurious retention. It may be necessary to be > more careful about what references are held on the stack in the > threads implementation (and elsewhere in your code). Until we > diagnose the place where objects are being retained it will be hard to > pinpoint. You should realize that weak refs are a problem in many > language implementations, not just Modula-3. I look forward to > hearing more definitively from you. > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: > > > Problem in my case can be thoroughness of GC... Because it's linked > > list of WeakRef-ed objects, it's always first element in list to > > become > > unreachable first - all others are reachable at least through > > previous > > (in list) WeakRef-ed object. So, behaviour I have observed can be > > because of some thousands of freeable objects, that one is missed > > too > > often. > > > > I have other object being cleaned and counted... There also cleanup > > invocation looks like it's lagging too much. I'll come with numbers > > after I have more tests. > > > > dd > > > > On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking wrote: > > > Sounds like we need some sort of testcase. Would you be able to > > > devise one? It will be hard to make it deterministic, but at > > > least we > > > should see a non-zero cleanup count. > > > > > > Antony Hosking | Associate Professor | Computer Science | Purdue > > > University > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: > > > > > > > I have single linked list that I use to send messages to many > > > > threads. > > > > It's end is locked for addition, and current end is given to > > > > each > > > > new > > > > client connecting. This way, all client are going towards end of > > > > the > > > > list so it's head becomes unreferenced and goes away by GC. > > > > > > > > Except it does not. > > > > > > > > I've added WeakRef cleanup and sequential id's so I can record > > > > maximum > > > > freed id for checking. No single cleanup happens. > > > > > > > > I'll cross check my code and count my references from > > > > application > > > > side, > > > > but maybe someone else has similar problems? > > > > > > > > dd > > > > -- > > > > Dragi?a Duri? > > > > > > > > > -- > > Dragi?a Duri? > > > > -- Dragi?a Duri? From dragisha at m3w.org Thu Mar 13 13:41:40 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 13 Mar 2008 13:41:40 +0100 Subject: [M3devel] argh Message-ID: <1205412100.32152.100.camel@hias.m3w.org> I've added Convert.FromLongInt to both Convert.i3 and Convert.m3, shipped m3core to my current install, build cm3-min and started making system, and it always fails here. Hints? dd === package m3-sys/cm3 === +++ cm3 -build -override -DROOT='/usr/src/redhat/BUILD/cm3-base-5.6.0/cm3' -DCM3_VERSION_TEXT='d5.6.0' -DCM3_VERSION_NUMBER='050600' -DCM3_LAST_CHANGED='2008-01-31' +++ --- building in LINUXLIBC6 --- new source -> compiling Version.m3 new "Version.mo" -> linking cm3 Fatal Error: incomplete program Convert.FromLongInt: is exported, but not implemented: Convert.i3 *** execution of cm3 -build -override -DROOT='/usr/src/redhat/BUILD/cm3-base-5.6.0/cm3' -DCM3_VERSION_TEXT='d5.6.0' -DCM3_VERSION_NUMBER='050600' -DCM3_LAST_CHANGED='2008-01-31' failed *** -- Dragi?a Duri? From dragisha at m3w.org Thu Mar 13 14:21:59 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 13 Mar 2008 14:21:59 +0100 Subject: [M3devel] argh In-Reply-To: <1205412100.32152.100.camel@hias.m3w.org> References: <1205412100.32152.100.camel@hias.m3w.org> Message-ID: <1205414519.32152.102.camel@hias.m3w.org> Of course... disregard this :) Anyone interested in RPM building? I can submit a patch for it, nothing spectacular but works nicely. dd On Thu, 2008-03-13 at 13:41 +0100, Dragi?a Duri? wrote: > I've added Convert.FromLongInt to both Convert.i3 and Convert.m3, > shipped m3core to my current install, build cm3-min and started making > system, and it always fails here. > > Hints? > > dd > > === package m3-sys/cm3 === > +++ cm3 -build -override > -DROOT='/usr/src/redhat/BUILD/cm3-base-5.6.0/cm3' > -DCM3_VERSION_TEXT='d5.6.0' -DCM3_VERSION_NUMBER='050600' > -DCM3_LAST_CHANGED='2008-01-31' +++ > --- building in LINUXLIBC6 --- > > new source -> compiling Version.m3 > new "Version.mo" -> linking cm3 > > Fatal Error: incomplete program > > Convert.FromLongInt: is exported, but not implemented: Convert.i3 > *** execution of cm3 -build -override > -DROOT='/usr/src/redhat/BUILD/cm3-base-5.6.0/cm3' > -DCM3_VERSION_TEXT='d5.6.0' -DCM3_VERSION_NUMBER='050600' > -DCM3_LAST_CHANGED='2008-01-31' failed *** > -- Dragi?a Duri? From hosking at cs.purdue.edu Thu Mar 13 14:46:27 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 13 Mar 2008 09:46:27 -0400 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <1205403396.32152.96.camel@hias.m3w.org> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> Message-ID: <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> They should get unlinked from the global ring. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: > What happens to thread stacks once threads are Join-ed? > > dd > > On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking wrote: >> This is probably a result of conservatism for references from thread >> stacks which results in spurious retention. It may be necessary to >> be >> more careful about what references are held on the stack in the >> threads implementation (and elsewhere in your code). Until we >> diagnose the place where objects are being retained it will be hard >> to >> pinpoint. You should realize that weak refs are a problem in many >> language implementations, not just Modula-3. I look forward to >> hearing more definitively from you. >> >> Antony Hosking | Associate Professor | Computer Science | Purdue >> University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> >> >> On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: >> >>> Problem in my case can be thoroughness of GC... Because it's linked >>> list of WeakRef-ed objects, it's always first element in list to >>> become >>> unreachable first - all others are reachable at least through >>> previous >>> (in list) WeakRef-ed object. So, behaviour I have observed can be >>> because of some thousands of freeable objects, that one is missed >>> too >>> often. >>> >>> I have other object being cleaned and counted... There also cleanup >>> invocation looks like it's lagging too much. I'll come with numbers >>> after I have more tests. >>> >>> dd >>> >>> On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking wrote: >>>> Sounds like we need some sort of testcase. Would you be able to >>>> devise one? It will be hard to make it deterministic, but at >>>> least we >>>> should see a non-zero cleanup count. >>>> >>>> Antony Hosking | Associate Professor | Computer Science | Purdue >>>> University >>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>> >>>> >>>> >>>> >>>> >>>> On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: >>>> >>>>> I have single linked list that I use to send messages to many >>>>> threads. >>>>> It's end is locked for addition, and current end is given to >>>>> each >>>>> new >>>>> client connecting. This way, all client are going towards end of >>>>> the >>>>> list so it's head becomes unreferenced and goes away by GC. >>>>> >>>>> Except it does not. >>>>> >>>>> I've added WeakRef cleanup and sequential id's so I can record >>>>> maximum >>>>> freed id for checking. No single cleanup happens. >>>>> >>>>> I'll cross check my code and count my references from >>>>> application >>>>> side, >>>>> but maybe someone else has similar problems? >>>>> >>>>> dd >>>>> -- >>>>> Dragi?a Duri? >>>>> >>>> >>> -- >>> Dragi?a Duri? >>> >> >> > -- > Dragi?a Duri? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Thu Mar 13 14:56:28 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 13 Mar 2008 14:56:28 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> Message-ID: <1205416588.32152.127.camel@hias.m3w.org> And not only that, their stackbase is also set to NIL, meaning their stack is not regarded in any way during future collections - meaning all local variables are forgotten once apply method returns. That way, "spuriousness" is not an issue, once thread returns? dd On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: > They should get unlinked from the global ring. > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: > > > What happens to thread stacks once threads are Join-ed? > > > > dd > > > > On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking wrote: > > > This is probably a result of conservatism for references from > > > thread > > > stacks which results in spurious retention. It may be necessary > > > to be > > > more careful about what references are held on the stack in the > > > threads implementation (and elsewhere in your code). Until we > > > diagnose the place where objects are being retained it will be > > > hard to > > > pinpoint. You should realize that weak refs are a problem in many > > > language implementations, not just Modula-3. I look forward to > > > hearing more definitively from you. > > > > > > Antony Hosking | Associate Professor | Computer Science | Purdue > > > University > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: > > > > > > > Problem in my case can be thoroughness of GC... Because it's > > > > linked > > > > list of WeakRef-ed objects, it's always first element in list to > > > > become > > > > unreachable first - all others are reachable at least through > > > > previous > > > > (in list) WeakRef-ed object. So, behaviour I have observed can > > > > be > > > > because of some thousands of freeable objects, that one is > > > > missed > > > > too > > > > often. > > > > > > > > I have other object being cleaned and counted... There also > > > > cleanup > > > > invocation looks like it's lagging too much. I'll come with > > > > numbers > > > > after I have more tests. > > > > > > > > dd > > > > > > > > On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking wrote: > > > > > Sounds like we need some sort of testcase. Would you be able > > > > > to > > > > > devise one? It will be hard to make it deterministic, but at > > > > > least we > > > > > should see a non-zero cleanup count. > > > > > > > > > > Antony Hosking | Associate Professor | Computer Science | > > > > > Purdue > > > > > University > > > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: > > > > > > > > > > > I have single linked list that I use to send messages to > > > > > > many > > > > > > threads. > > > > > > It's end is locked for addition, and current end is given to > > > > > > each > > > > > > new > > > > > > client connecting. This way, all client are going towards > > > > > > end of > > > > > > the > > > > > > list so it's head becomes unreferenced and goes away by GC. > > > > > > > > > > > > Except it does not. > > > > > > > > > > > > I've added WeakRef cleanup and sequential id's so I can > > > > > > record > > > > > > maximum > > > > > > freed id for checking. No single cleanup happens. > > > > > > > > > > > > I'll cross check my code and count my references from > > > > > > application > > > > > > side, > > > > > > but maybe someone else has similar problems? > > > > > > > > > > > > dd > > > > > > -- > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > -- > > > > Dragi?a Duri? > > > > > > > > > > > > -- > > Dragi?a Duri? > > > > -- Dragi?a Duri? From hosking at cs.purdue.edu Thu Mar 13 15:08:53 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 13 Mar 2008 10:08:53 -0400 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <1205416588.32152.127.camel@hias.m3w.org> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> Message-ID: Indeed! Can you devise a testcase? Note that the generational collector will retain old objects for some time until a major collection occurs, Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: > And not only that, their stackbase is also set to NIL, meaning their > stack is not regarded in any way during future collections - meaning > all > local variables are forgotten once apply method returns. > > That way, "spuriousness" is not an issue, once thread returns? > > dd > > On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: >> They should get unlinked from the global ring. >> >> Antony Hosking | Associate Professor | Computer Science | Purdue >> University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> >> >> On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: >> >>> What happens to thread stacks once threads are Join-ed? >>> >>> dd >>> >>> On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking wrote: >>>> This is probably a result of conservatism for references from >>>> thread >>>> stacks which results in spurious retention. It may be necessary >>>> to be >>>> more careful about what references are held on the stack in the >>>> threads implementation (and elsewhere in your code). Until we >>>> diagnose the place where objects are being retained it will be >>>> hard to >>>> pinpoint. You should realize that weak refs are a problem in many >>>> language implementations, not just Modula-3. I look forward to >>>> hearing more definitively from you. >>>> >>>> Antony Hosking | Associate Professor | Computer Science | Purdue >>>> University >>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>> >>>> >>>> >>>> >>>> >>>> On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: >>>> >>>>> Problem in my case can be thoroughness of GC... Because it's >>>>> linked >>>>> list of WeakRef-ed objects, it's always first element in list to >>>>> become >>>>> unreachable first - all others are reachable at least through >>>>> previous >>>>> (in list) WeakRef-ed object. So, behaviour I have observed can >>>>> be >>>>> because of some thousands of freeable objects, that one is >>>>> missed >>>>> too >>>>> often. >>>>> >>>>> I have other object being cleaned and counted... There also >>>>> cleanup >>>>> invocation looks like it's lagging too much. I'll come with >>>>> numbers >>>>> after I have more tests. >>>>> >>>>> dd >>>>> >>>>> On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking wrote: >>>>>> Sounds like we need some sort of testcase. Would you be able >>>>>> to >>>>>> devise one? It will be hard to make it deterministic, but at >>>>>> least we >>>>>> should see a non-zero cleanup count. >>>>>> >>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>> Purdue >>>>>> University >>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: >>>>>> >>>>>>> I have single linked list that I use to send messages to >>>>>>> many >>>>>>> threads. >>>>>>> It's end is locked for addition, and current end is given to >>>>>>> each >>>>>>> new >>>>>>> client connecting. This way, all client are going towards >>>>>>> end of >>>>>>> the >>>>>>> list so it's head becomes unreferenced and goes away by GC. >>>>>>> >>>>>>> Except it does not. >>>>>>> >>>>>>> I've added WeakRef cleanup and sequential id's so I can >>>>>>> record >>>>>>> maximum >>>>>>> freed id for checking. No single cleanup happens. >>>>>>> >>>>>>> I'll cross check my code and count my references from >>>>>>> application >>>>>>> side, >>>>>>> but maybe someone else has similar problems? >>>>>>> >>>>>>> dd >>>>>>> -- >>>>>>> Dragi?a Duri? >>>>>>> >>>>>> >>>>> -- >>>>> Dragi?a Duri? >>>>> >>>> >>>> >>> -- >>> Dragi?a Duri? >>> >> >> > -- > Dragi?a Duri? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Thu Mar 13 15:20:04 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 13 Mar 2008 15:20:04 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> Message-ID: <1205418004.32152.137.camel@hias.m3w.org> Garbage collector is one piece of runtime I am groking only now. First, I'll try to analyze whole weakref structure and flow and see where these numbers mismatch so much: [15:13:58] 96 users, 1 ops and 0 unresolved; minId = 38348; thrs = 102 [15:13:58] started = 1991; cleaned = 1579; nilled = 1889; joined = 1889 [15:13:58] Dangling thrs-a-b-n = 5 [15:13:58] Live queue end is 38396 (head = 38340), maxClean = 37821, cleaned = 35309 joined is number of finished threads, cleaned is number of timer their Client.T's weakref was cleaned. Same is head = 38340 = number of head elements from my message queue (all are unlinked) and cleaned = 35309 = number of elements RTHeapRep's FinalCleanup happened. This process' uptime is around 5 hours. Allocations are as evenly distributed as it's important to get picture. dd On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: > Indeed! Can you devise a testcase? Note that the generational > collector will retain old objects for some time until a major > collection occurs, > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: > > > And not only that, their stackbase is also set to NIL, meaning their > > stack is not regarded in any way during future collections - meaning > > all > > local variables are forgotten once apply method returns. > > > > That way, "spuriousness" is not an issue, once thread returns? > > > > dd > > > > On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: > > > They should get unlinked from the global ring. > > > > > > Antony Hosking | Associate Professor | Computer Science | Purdue > > > University > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: > > > > > > > What happens to thread stacks once threads are Join-ed? > > > > > > > > dd > > > > > > > > On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking wrote: > > > > > This is probably a result of conservatism for references from > > > > > thread > > > > > stacks which results in spurious retention. It may be > > > > > necessary > > > > > to be > > > > > more careful about what references are held on the stack in > > > > > the > > > > > threads implementation (and elsewhere in your code). Until we > > > > > diagnose the place where objects are being retained it will be > > > > > hard to > > > > > pinpoint. You should realize that weak refs are a problem in > > > > > many > > > > > language implementations, not just Modula-3. I look forward > > > > > to > > > > > hearing more definitively from you. > > > > > > > > > > Antony Hosking | Associate Professor | Computer Science | > > > > > Purdue > > > > > University > > > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: > > > > > > > > > > > Problem in my case can be thoroughness of GC... Because it's > > > > > > linked > > > > > > list of WeakRef-ed objects, it's always first element in > > > > > > list to > > > > > > become > > > > > > unreachable first - all others are reachable at least > > > > > > through > > > > > > previous > > > > > > (in list) WeakRef-ed object. So, behaviour I have observed > > > > > > can > > > > > > be > > > > > > because of some thousands of freeable objects, that one is > > > > > > missed > > > > > > too > > > > > > often. > > > > > > > > > > > > I have other object being cleaned and counted... There also > > > > > > cleanup > > > > > > invocation looks like it's lagging too much. I'll come with > > > > > > numbers > > > > > > after I have more tests. > > > > > > > > > > > > dd > > > > > > > > > > > > On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking wrote: > > > > > > > Sounds like we need some sort of testcase. Would you be > > > > > > > able > > > > > > > to > > > > > > > devise one? It will be hard to make it deterministic, but > > > > > > > at > > > > > > > least we > > > > > > > should see a non-zero cleanup count. > > > > > > > > > > > > > > Antony Hosking | Associate Professor | Computer Science | > > > > > > > Purdue > > > > > > > University > > > > > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: > > > > > > > > > > > > > > > I have single linked list that I use to send messages to > > > > > > > > many > > > > > > > > threads. > > > > > > > > It's end is locked for addition, and current end is > > > > > > > > given to > > > > > > > > each > > > > > > > > new > > > > > > > > client connecting. This way, all client are going > > > > > > > > towards > > > > > > > > end of > > > > > > > > the > > > > > > > > list so it's head becomes unreferenced and goes away by > > > > > > > > GC. > > > > > > > > > > > > > > > > Except it does not. > > > > > > > > > > > > > > > > I've added WeakRef cleanup and sequential id's so I can > > > > > > > > record > > > > > > > > maximum > > > > > > > > freed id for checking. No single cleanup happens. > > > > > > > > > > > > > > > > I'll cross check my code and count my references from > > > > > > > > application > > > > > > > > side, > > > > > > > > but maybe someone else has similar problems? > > > > > > > > > > > > > > > > dd > > > > > > > > -- > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > -- > > > > Dragi?a Duri? > > > > > > > > > > > > -- > > Dragi?a Duri? > > > -- Dragi?a Duri? From dragisha at m3w.org Thu Mar 13 15:22:17 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 13 Mar 2008 15:22:17 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> Message-ID: <1205418137.32152.139.camel@hias.m3w.org> Can I force "major collection" ? dd On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: > Indeed! Can you devise a testcase? Note that the generational > collector will retain old objects for some time until a major > collection occurs, > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: > > > And not only that, their stackbase is also set to NIL, meaning their > > stack is not regarded in any way during future collections - meaning > > all > > local variables are forgotten once apply method returns. > > > > That way, "spuriousness" is not an issue, once thread returns? > > > > dd > > > > On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: > > > They should get unlinked from the global ring. > > > > > > Antony Hosking | Associate Professor | Computer Science | Purdue > > > University > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: > > > > > > > What happens to thread stacks once threads are Join-ed? > > > > > > > > dd > > > > > > > > On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking wrote: > > > > > This is probably a result of conservatism for references from > > > > > thread > > > > > stacks which results in spurious retention. It may be > > > > > necessary > > > > > to be > > > > > more careful about what references are held on the stack in > > > > > the > > > > > threads implementation (and elsewhere in your code). Until we > > > > > diagnose the place where objects are being retained it will be > > > > > hard to > > > > > pinpoint. You should realize that weak refs are a problem in > > > > > many > > > > > language implementations, not just Modula-3. I look forward > > > > > to > > > > > hearing more definitively from you. > > > > > > > > > > Antony Hosking | Associate Professor | Computer Science | > > > > > Purdue > > > > > University > > > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: > > > > > > > > > > > Problem in my case can be thoroughness of GC... Because it's > > > > > > linked > > > > > > list of WeakRef-ed objects, it's always first element in > > > > > > list to > > > > > > become > > > > > > unreachable first - all others are reachable at least > > > > > > through > > > > > > previous > > > > > > (in list) WeakRef-ed object. So, behaviour I have observed > > > > > > can > > > > > > be > > > > > > because of some thousands of freeable objects, that one is > > > > > > missed > > > > > > too > > > > > > often. > > > > > > > > > > > > I have other object being cleaned and counted... There also > > > > > > cleanup > > > > > > invocation looks like it's lagging too much. I'll come with > > > > > > numbers > > > > > > after I have more tests. > > > > > > > > > > > > dd > > > > > > > > > > > > On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking wrote: > > > > > > > Sounds like we need some sort of testcase. Would you be > > > > > > > able > > > > > > > to > > > > > > > devise one? It will be hard to make it deterministic, but > > > > > > > at > > > > > > > least we > > > > > > > should see a non-zero cleanup count. > > > > > > > > > > > > > > Antony Hosking | Associate Professor | Computer Science | > > > > > > > Purdue > > > > > > > University > > > > > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: > > > > > > > > > > > > > > > I have single linked list that I use to send messages to > > > > > > > > many > > > > > > > > threads. > > > > > > > > It's end is locked for addition, and current end is > > > > > > > > given to > > > > > > > > each > > > > > > > > new > > > > > > > > client connecting. This way, all client are going > > > > > > > > towards > > > > > > > > end of > > > > > > > > the > > > > > > > > list so it's head becomes unreferenced and goes away by > > > > > > > > GC. > > > > > > > > > > > > > > > > Except it does not. > > > > > > > > > > > > > > > > I've added WeakRef cleanup and sequential id's so I can > > > > > > > > record > > > > > > > > maximum > > > > > > > > freed id for checking. No single cleanup happens. > > > > > > > > > > > > > > > > I'll cross check my code and count my references from > > > > > > > > application > > > > > > > > side, > > > > > > > > but maybe someone else has similar problems? > > > > > > > > > > > > > > > > dd > > > > > > > > -- > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > -- > > > > Dragi?a Duri? > > > > > > > > > > > > -- > > Dragi?a Duri? > > > -- Dragi?a Duri? From hosking at cs.purdue.edu Thu Mar 13 15:54:23 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 13 Mar 2008 10:54:23 -0400 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <1205418137.32152.139.camel@hias.m3w.org> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> Message-ID: <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> Yes. RTCollectorSRC.FinishCollection; RTCollectorSRC.StartCollection. This will finish the current (possibly minor) collection, and start a major collection. The major collection must finish before you can guarantee garbage has been freed, so you may want to RTCollectorSRC.FinishCollection. This sequence is packaged as RTCollector.Collect. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? wrote: > Can I force "major collection" ? > > dd > > On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: >> Indeed! Can you devise a testcase? Note that the generational >> collector will retain old objects for some time until a major >> collection occurs, >> >> Antony Hosking | Associate Professor | Computer Science | Purdue >> University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> >> >> On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: >> >>> And not only that, their stackbase is also set to NIL, meaning their >>> stack is not regarded in any way during future collections - meaning >>> all >>> local variables are forgotten once apply method returns. >>> >>> That way, "spuriousness" is not an issue, once thread returns? >>> >>> dd >>> >>> On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: >>>> They should get unlinked from the global ring. >>>> >>>> Antony Hosking | Associate Professor | Computer Science | Purdue >>>> University >>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>> >>>> >>>> >>>> >>>> >>>> On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: >>>> >>>>> What happens to thread stacks once threads are Join-ed? >>>>> >>>>> dd >>>>> >>>>> On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking wrote: >>>>>> This is probably a result of conservatism for references from >>>>>> thread >>>>>> stacks which results in spurious retention. It may be >>>>>> necessary >>>>>> to be >>>>>> more careful about what references are held on the stack in >>>>>> the >>>>>> threads implementation (and elsewhere in your code). Until we >>>>>> diagnose the place where objects are being retained it will be >>>>>> hard to >>>>>> pinpoint. You should realize that weak refs are a problem in >>>>>> many >>>>>> language implementations, not just Modula-3. I look forward >>>>>> to >>>>>> hearing more definitively from you. >>>>>> >>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>> Purdue >>>>>> University >>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: >>>>>> >>>>>>> Problem in my case can be thoroughness of GC... Because it's >>>>>>> linked >>>>>>> list of WeakRef-ed objects, it's always first element in >>>>>>> list to >>>>>>> become >>>>>>> unreachable first - all others are reachable at least >>>>>>> through >>>>>>> previous >>>>>>> (in list) WeakRef-ed object. So, behaviour I have observed >>>>>>> can >>>>>>> be >>>>>>> because of some thousands of freeable objects, that one is >>>>>>> missed >>>>>>> too >>>>>>> often. >>>>>>> >>>>>>> I have other object being cleaned and counted... There also >>>>>>> cleanup >>>>>>> invocation looks like it's lagging too much. I'll come with >>>>>>> numbers >>>>>>> after I have more tests. >>>>>>> >>>>>>> dd >>>>>>> >>>>>>> On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking wrote: >>>>>>>> Sounds like we need some sort of testcase. Would you be >>>>>>>> able >>>>>>>> to >>>>>>>> devise one? It will be hard to make it deterministic, but >>>>>>>> at >>>>>>>> least we >>>>>>>> should see a non-zero cleanup count. >>>>>>>> >>>>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>>>> Purdue >>>>>>>> University >>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: >>>>>>>> >>>>>>>>> I have single linked list that I use to send messages to >>>>>>>>> many >>>>>>>>> threads. >>>>>>>>> It's end is locked for addition, and current end is >>>>>>>>> given to >>>>>>>>> each >>>>>>>>> new >>>>>>>>> client connecting. This way, all client are going >>>>>>>>> towards >>>>>>>>> end of >>>>>>>>> the >>>>>>>>> list so it's head becomes unreferenced and goes away by >>>>>>>>> GC. >>>>>>>>> >>>>>>>>> Except it does not. >>>>>>>>> >>>>>>>>> I've added WeakRef cleanup and sequential id's so I can >>>>>>>>> record >>>>>>>>> maximum >>>>>>>>> freed id for checking. No single cleanup happens. >>>>>>>>> >>>>>>>>> I'll cross check my code and count my references from >>>>>>>>> application >>>>>>>>> side, >>>>>>>>> but maybe someone else has similar problems? >>>>>>>>> >>>>>>>>> dd >>>>>>>>> -- >>>>>>>>> Dragi?a Duri? >>>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> Dragi?a Duri? >>>>>>> >>>>>> >>>>>> >>>>> -- >>>>> Dragi?a Duri? >>>>> >>>> >>>> >>> -- >>> Dragi?a Duri? >>> >> > -- > Dragi?a Duri? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Thu Mar 13 17:28:54 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 13 Mar 2008 17:28:54 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> Message-ID: <1205425734.32152.144.camel@hias.m3w.org> [17:25:19] 117 users, 1 ops and 1 unresolved; minId = 18617; thrs = 120 [17:25:19] started = 663; cleaned = 452; nilled = 543; joined = 543 [17:25:19] Dangling thrs-a-b-n = 1 [17:25:19] Live queue end is 18621 (head = 18616), maxClean = 18611, cleaned = 16583 About one hour uptime, after I added RTCollector.Collect() call every 20 seconds. dd On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking wrote: > Yes. RTCollectorSRC.FinishCollection; > RTCollectorSRC.StartCollection. This will finish the current > (possibly minor) collection, and start a major collection. The major > collection must finish before you can guarantee garbage has been > freed, so you may want to RTCollectorSRC.FinishCollection. This > sequence is packaged as RTCollector.Collect. > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? wrote: > > > Can I force "major collection" ? > > > > dd > > > > On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: > > > Indeed! Can you devise a testcase? Note that the generational > > > collector will retain old objects for some time until a major > > > collection occurs, > > > > > > Antony Hosking | Associate Professor | Computer Science | Purdue > > > University > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: > > > > > > > And not only that, their stackbase is also set to NIL, meaning > > > > their > > > > stack is not regarded in any way during future collections - > > > > meaning > > > > all > > > > local variables are forgotten once apply method returns. > > > > > > > > That way, "spuriousness" is not an issue, once thread returns? > > > > > > > > dd > > > > > > > > On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: > > > > > They should get unlinked from the global ring. > > > > > > > > > > Antony Hosking | Associate Professor | Computer Science | > > > > > Purdue > > > > > University > > > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: > > > > > > > > > > > What happens to thread stacks once threads are Join-ed? > > > > > > > > > > > > dd > > > > > > > > > > > > On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking wrote: > > > > > > > This is probably a result of conservatism for references > > > > > > > from > > > > > > > thread > > > > > > > stacks which results in spurious retention. It may be > > > > > > > necessary > > > > > > > to be > > > > > > > more careful about what references are held on the stack > > > > > > > in > > > > > > > the > > > > > > > threads implementation (and elsewhere in your code). > > > > > > > Until we > > > > > > > diagnose the place where objects are being retained it > > > > > > > will be > > > > > > > hard to > > > > > > > pinpoint. You should realize that weak refs are a problem > > > > > > > in > > > > > > > many > > > > > > > language implementations, not just Modula-3. I look > > > > > > > forward > > > > > > > to > > > > > > > hearing more definitively from you. > > > > > > > > > > > > > > Antony Hosking | Associate Professor | Computer Science | > > > > > > > Purdue > > > > > > > University > > > > > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: > > > > > > > > > > > > > > > Problem in my case can be thoroughness of GC... Because > > > > > > > > it's > > > > > > > > linked > > > > > > > > list of WeakRef-ed objects, it's always first element in > > > > > > > > list to > > > > > > > > become > > > > > > > > unreachable first - all others are reachable at least > > > > > > > > through > > > > > > > > previous > > > > > > > > (in list) WeakRef-ed object. So, behaviour I have > > > > > > > > observed > > > > > > > > can > > > > > > > > be > > > > > > > > because of some thousands of freeable objects, that one > > > > > > > > is > > > > > > > > missed > > > > > > > > too > > > > > > > > often. > > > > > > > > > > > > > > > > I have other object being cleaned and counted... There > > > > > > > > also > > > > > > > > cleanup > > > > > > > > invocation looks like it's lagging too much. I'll come > > > > > > > > with > > > > > > > > numbers > > > > > > > > after I have more tests. > > > > > > > > > > > > > > > > dd > > > > > > > > > > > > > > > > On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking wrote: > > > > > > > > > Sounds like we need some sort of testcase. Would you > > > > > > > > > be > > > > > > > > > able > > > > > > > > > to > > > > > > > > > devise one? It will be hard to make it deterministic, > > > > > > > > > but > > > > > > > > > at > > > > > > > > > least we > > > > > > > > > should see a non-zero cleanup count. > > > > > > > > > > > > > > > > > > Antony Hosking | Associate Professor | Computer > > > > > > > > > Science | > > > > > > > > > Purdue > > > > > > > > > University > > > > > > > > > 305 N. University Street | West Lafayette | IN 47907 | > > > > > > > > > USA > > > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: > > > > > > > > > > > > > > > > > > > I have single linked list that I use to send > > > > > > > > > > messages to > > > > > > > > > > many > > > > > > > > > > threads. > > > > > > > > > > It's end is locked for addition, and current end is > > > > > > > > > > given to > > > > > > > > > > each > > > > > > > > > > new > > > > > > > > > > client connecting. This way, all client are going > > > > > > > > > > towards > > > > > > > > > > end of > > > > > > > > > > the > > > > > > > > > > list so it's head becomes unreferenced and goes away > > > > > > > > > > by > > > > > > > > > > GC. > > > > > > > > > > > > > > > > > > > > Except it does not. > > > > > > > > > > > > > > > > > > > > I've added WeakRef cleanup and sequential id's so I > > > > > > > > > > can > > > > > > > > > > record > > > > > > > > > > maximum > > > > > > > > > > freed id for checking. No single cleanup happens. > > > > > > > > > > > > > > > > > > > > I'll cross check my code and count my references > > > > > > > > > > from > > > > > > > > > > application > > > > > > > > > > side, > > > > > > > > > > but maybe someone else has similar problems? > > > > > > > > > > > > > > > > > > > > dd > > > > > > > > > > -- > > > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > -- > > > > Dragi?a Duri? > > > > > > > > > -- > > Dragi?a Duri? > > > -- Dragi?a Duri? From hosking at cs.purdue.edu Thu Mar 13 17:45:26 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 13 Mar 2008 12:45:26 -0400 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <1205425734.32152.144.camel@hias.m3w.org> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> <1205425734.32152.144.camel@hias.m3w.org> Message-ID: <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> Realizing that conservative roots collectors like ours can retain data unnecessarily. If a thread stack refers to a page that holds a reference that (transitively) refers to some otherwise dead object then that object will be retained. We say that the page is pinned. Smartening up the collector to ignore dead objects in pinned pages is possible, but not currently implemented. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 13, 2008, at 12:28 PM, Dragi?a Duri? wrote: > [17:25:19] 117 users, 1 ops and 1 unresolved; minId = > 18617; thrs = 120 > [17:25:19] started = 663; cleaned = 452; nilled = 543; > joined = 543 > [17:25:19] Dangling thrs-a-b-n = 1 > [17:25:19] Live queue end is 18621 (head = 18616), > maxClean = 18611, cleaned = 16583 > > About one hour uptime, after I added RTCollector.Collect() call > every 20 > seconds. > > dd > > On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking wrote: >> Yes. RTCollectorSRC.FinishCollection; >> RTCollectorSRC.StartCollection. This will finish the current >> (possibly minor) collection, and start a major collection. The major >> collection must finish before you can guarantee garbage has been >> freed, so you may want to RTCollectorSRC.FinishCollection. This >> sequence is packaged as RTCollector.Collect. >> >> Antony Hosking | Associate Professor | Computer Science | Purdue >> University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> >> >> On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? wrote: >> >>> Can I force "major collection" ? >>> >>> dd >>> >>> On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: >>>> Indeed! Can you devise a testcase? Note that the generational >>>> collector will retain old objects for some time until a major >>>> collection occurs, >>>> >>>> Antony Hosking | Associate Professor | Computer Science | Purdue >>>> University >>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>> >>>> >>>> >>>> >>>> >>>> On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: >>>> >>>>> And not only that, their stackbase is also set to NIL, meaning >>>>> their >>>>> stack is not regarded in any way during future collections - >>>>> meaning >>>>> all >>>>> local variables are forgotten once apply method returns. >>>>> >>>>> That way, "spuriousness" is not an issue, once thread returns? >>>>> >>>>> dd >>>>> >>>>> On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: >>>>>> They should get unlinked from the global ring. >>>>>> >>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>> Purdue >>>>>> University >>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: >>>>>> >>>>>>> What happens to thread stacks once threads are Join-ed? >>>>>>> >>>>>>> dd >>>>>>> >>>>>>> On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking wrote: >>>>>>>> This is probably a result of conservatism for references >>>>>>>> from >>>>>>>> thread >>>>>>>> stacks which results in spurious retention. It may be >>>>>>>> necessary >>>>>>>> to be >>>>>>>> more careful about what references are held on the stack >>>>>>>> in >>>>>>>> the >>>>>>>> threads implementation (and elsewhere in your code). >>>>>>>> Until we >>>>>>>> diagnose the place where objects are being retained it >>>>>>>> will be >>>>>>>> hard to >>>>>>>> pinpoint. You should realize that weak refs are a problem >>>>>>>> in >>>>>>>> many >>>>>>>> language implementations, not just Modula-3. I look >>>>>>>> forward >>>>>>>> to >>>>>>>> hearing more definitively from you. >>>>>>>> >>>>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>>>> Purdue >>>>>>>> University >>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: >>>>>>>> >>>>>>>>> Problem in my case can be thoroughness of GC... Because >>>>>>>>> it's >>>>>>>>> linked >>>>>>>>> list of WeakRef-ed objects, it's always first element in >>>>>>>>> list to >>>>>>>>> become >>>>>>>>> unreachable first - all others are reachable at least >>>>>>>>> through >>>>>>>>> previous >>>>>>>>> (in list) WeakRef-ed object. So, behaviour I have >>>>>>>>> observed >>>>>>>>> can >>>>>>>>> be >>>>>>>>> because of some thousands of freeable objects, that one >>>>>>>>> is >>>>>>>>> missed >>>>>>>>> too >>>>>>>>> often. >>>>>>>>> >>>>>>>>> I have other object being cleaned and counted... There >>>>>>>>> also >>>>>>>>> cleanup >>>>>>>>> invocation looks like it's lagging too much. I'll come >>>>>>>>> with >>>>>>>>> numbers >>>>>>>>> after I have more tests. >>>>>>>>> >>>>>>>>> dd >>>>>>>>> >>>>>>>>> On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking wrote: >>>>>>>>>> Sounds like we need some sort of testcase. Would you >>>>>>>>>> be >>>>>>>>>> able >>>>>>>>>> to >>>>>>>>>> devise one? It will be hard to make it deterministic, >>>>>>>>>> but >>>>>>>>>> at >>>>>>>>>> least we >>>>>>>>>> should see a non-zero cleanup count. >>>>>>>>>> >>>>>>>>>> Antony Hosking | Associate Professor | Computer >>>>>>>>>> Science | >>>>>>>>>> Purdue >>>>>>>>>> University >>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | >>>>>>>>>> USA >>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: >>>>>>>>>> >>>>>>>>>>> I have single linked list that I use to send >>>>>>>>>>> messages to >>>>>>>>>>> many >>>>>>>>>>> threads. >>>>>>>>>>> It's end is locked for addition, and current end is >>>>>>>>>>> given to >>>>>>>>>>> each >>>>>>>>>>> new >>>>>>>>>>> client connecting. This way, all client are going >>>>>>>>>>> towards >>>>>>>>>>> end of >>>>>>>>>>> the >>>>>>>>>>> list so it's head becomes unreferenced and goes away >>>>>>>>>>> by >>>>>>>>>>> GC. >>>>>>>>>>> >>>>>>>>>>> Except it does not. >>>>>>>>>>> >>>>>>>>>>> I've added WeakRef cleanup and sequential id's so I >>>>>>>>>>> can >>>>>>>>>>> record >>>>>>>>>>> maximum >>>>>>>>>>> freed id for checking. No single cleanup happens. >>>>>>>>>>> >>>>>>>>>>> I'll cross check my code and count my references >>>>>>>>>>> from >>>>>>>>>>> application >>>>>>>>>>> side, >>>>>>>>>>> but maybe someone else has similar problems? >>>>>>>>>>> >>>>>>>>>>> dd >>>>>>>>>>> -- >>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Dragi?a Duri? >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> Dragi?a Duri? >>>>>>> >>>>>> >>>>>> >>>>> -- >>>>> Dragi?a Duri? >>>>> >>>> >>> -- >>> Dragi?a Duri? >>> >> > -- > Dragi?a Duri? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Thu Mar 13 18:43:19 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 13 Mar 2008 18:43:19 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> <1205425734.32152.144.camel@hias.m3w.org> <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> Message-ID: <1205430199.32152.152.camel@hias.m3w.org> I understand transitiveness :), but I've made big effort to escape that. Lifetime of thread equals lifetime of Client.T instance, and that instance is kept in thread's closure, and also in local variable. Both are set to NIL before thread termination - all terminations and joins happen and are counted and shown. Pointers to Message objects are kept in two places. Hub.T keeps pointer to list tail, and each Client.T contains pointer to it's current element. That pointer is forwarded as Message's are added to end. No temporary variable, not even WITH substitution is used. And these threads are ones dying with their Client.T's. "head" is new mechanism used to unlink all elements with id less than minimal id used in active Client.T instances. Unlinked totally and all as one free for garbage collector to finish them. I see no transition. dd On Thu, 2008-03-13 at 12:45 -0400, Tony Hosking wrote: > Realizing that conservative roots collectors like ours can retain data > unnecessarily. If a thread stack refers to a page that holds a > reference that (transitively) refers to some otherwise dead object > then that object will be retained. We say that the page is pinned. > Smartening up the collector to ignore dead objects in pinned pages is > possible, but not currently implemented. > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > On Mar 13, 2008, at 12:28 PM, Dragi?a Duri? wrote: > > > [17:25:19] 117 users, 1 ops and 1 unresolved; minId = > > 18617; thrs = 120 > > [17:25:19] started = 663; cleaned = 452; nilled = 543; > > joined = 543 > > [17:25:19] Dangling thrs-a-b-n = 1 > > [17:25:19] Live queue end is 18621 (head = 18616), > > maxClean = 18611, cleaned = 16583 > > > > About one hour uptime, after I added RTCollector.Collect() call > > every 20 > > seconds. > > > > dd > > > > On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking wrote: > > > Yes. RTCollectorSRC.FinishCollection; > > > RTCollectorSRC.StartCollection. This will finish the current > > > (possibly minor) collection, and start a major collection. The > > > major > > > collection must finish before you can guarantee garbage has been > > > freed, so you may want to RTCollectorSRC.FinishCollection. This > > > sequence is packaged as RTCollector.Collect. > > > > > > Antony Hosking | Associate Professor | Computer Science | Purdue > > > University > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? wrote: > > > > > > > Can I force "major collection" ? > > > > > > > > dd > > > > > > > > On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: > > > > > Indeed! Can you devise a testcase? Note that the generational > > > > > collector will retain old objects for some time until a major > > > > > collection occurs, > > > > > > > > > > Antony Hosking | Associate Professor | Computer Science | > > > > > Purdue > > > > > University > > > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: > > > > > > > > > > > And not only that, their stackbase is also set to NIL, > > > > > > meaning > > > > > > their > > > > > > stack is not regarded in any way during future collections - > > > > > > meaning > > > > > > all > > > > > > local variables are forgotten once apply method returns. > > > > > > > > > > > > That way, "spuriousness" is not an issue, once thread > > > > > > returns? > > > > > > > > > > > > dd > > > > > > > > > > > > On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: > > > > > > > They should get unlinked from the global ring. > > > > > > > > > > > > > > Antony Hosking | Associate Professor | Computer Science | > > > > > > > Purdue > > > > > > > University > > > > > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: > > > > > > > > > > > > > > > What happens to thread stacks once threads are Join-ed? > > > > > > > > > > > > > > > > dd > > > > > > > > > > > > > > > > On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking wrote: > > > > > > > > > This is probably a result of conservatism for > > > > > > > > > references > > > > > > > > > from > > > > > > > > > thread > > > > > > > > > stacks which results in spurious retention. It may be > > > > > > > > > necessary > > > > > > > > > to be > > > > > > > > > more careful about what references are held on the > > > > > > > > > stack > > > > > > > > > in > > > > > > > > > the > > > > > > > > > threads implementation (and elsewhere in your code). > > > > > > > > > Until we > > > > > > > > > diagnose the place where objects are being retained it > > > > > > > > > will be > > > > > > > > > hard to > > > > > > > > > pinpoint. You should realize that weak refs are a > > > > > > > > > problem > > > > > > > > > in > > > > > > > > > many > > > > > > > > > language implementations, not just Modula-3. I look > > > > > > > > > forward > > > > > > > > > to > > > > > > > > > hearing more definitively from you. > > > > > > > > > > > > > > > > > > Antony Hosking | Associate Professor | Computer > > > > > > > > > Science | > > > > > > > > > Purdue > > > > > > > > > University > > > > > > > > > 305 N. University Street | West Lafayette | IN 47907 | > > > > > > > > > USA > > > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: > > > > > > > > > > > > > > > > > > > Problem in my case can be thoroughness of GC... > > > > > > > > > > Because > > > > > > > > > > it's > > > > > > > > > > linked > > > > > > > > > > list of WeakRef-ed objects, it's always first > > > > > > > > > > element in > > > > > > > > > > list to > > > > > > > > > > become > > > > > > > > > > unreachable first - all others are reachable at > > > > > > > > > > least > > > > > > > > > > through > > > > > > > > > > previous > > > > > > > > > > (in list) WeakRef-ed object. So, behaviour I have > > > > > > > > > > observed > > > > > > > > > > can > > > > > > > > > > be > > > > > > > > > > because of some thousands of freeable objects, that > > > > > > > > > > one > > > > > > > > > > is > > > > > > > > > > missed > > > > > > > > > > too > > > > > > > > > > often. > > > > > > > > > > > > > > > > > > > > I have other object being cleaned and counted... > > > > > > > > > > There > > > > > > > > > > also > > > > > > > > > > cleanup > > > > > > > > > > invocation looks like it's lagging too much. I'll > > > > > > > > > > come > > > > > > > > > > with > > > > > > > > > > numbers > > > > > > > > > > after I have more tests. > > > > > > > > > > > > > > > > > > > > dd > > > > > > > > > > > > > > > > > > > > On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking > > > > > > > > > > wrote: > > > > > > > > > > > Sounds like we need some sort of testcase. Would > > > > > > > > > > > you > > > > > > > > > > > be > > > > > > > > > > > able > > > > > > > > > > > to > > > > > > > > > > > devise one? It will be hard to make it > > > > > > > > > > > deterministic, > > > > > > > > > > > but > > > > > > > > > > > at > > > > > > > > > > > least we > > > > > > > > > > > should see a non-zero cleanup count. > > > > > > > > > > > > > > > > > > > > > > Antony Hosking | Associate Professor | Computer > > > > > > > > > > > Science | > > > > > > > > > > > Purdue > > > > > > > > > > > University > > > > > > > > > > > 305 N. University Street | West Lafayette | IN > > > > > > > > > > > 47907 | > > > > > > > > > > > USA > > > > > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: > > > > > > > > > > > > > > > > > > > > > > > I have single linked list that I use to send > > > > > > > > > > > > messages to > > > > > > > > > > > > many > > > > > > > > > > > > threads. > > > > > > > > > > > > It's end is locked for addition, and current end > > > > > > > > > > > > is > > > > > > > > > > > > given to > > > > > > > > > > > > each > > > > > > > > > > > > new > > > > > > > > > > > > client connecting. This way, all client are > > > > > > > > > > > > going > > > > > > > > > > > > towards > > > > > > > > > > > > end of > > > > > > > > > > > > the > > > > > > > > > > > > list so it's head becomes unreferenced and goes > > > > > > > > > > > > away > > > > > > > > > > > > by > > > > > > > > > > > > GC. > > > > > > > > > > > > > > > > > > > > > > > > Except it does not. > > > > > > > > > > > > > > > > > > > > > > > > I've added WeakRef cleanup and sequential id's > > > > > > > > > > > > so I > > > > > > > > > > > > can > > > > > > > > > > > > record > > > > > > > > > > > > maximum > > > > > > > > > > > > freed id for checking. No single cleanup > > > > > > > > > > > > happens. > > > > > > > > > > > > > > > > > > > > > > > > I'll cross check my code and count my references > > > > > > > > > > > > from > > > > > > > > > > > > application > > > > > > > > > > > > side, > > > > > > > > > > > > but maybe someone else has similar problems? > > > > > > > > > > > > > > > > > > > > > > > > dd > > > > > > > > > > > > -- > > > > > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > -- > > > > Dragi?a Duri? > > > > > > > > > -- > > Dragi?a Duri? > > > -- Dragi?a Duri? From hosking at cs.purdue.edu Thu Mar 13 22:41:56 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 13 Mar 2008 17:41:56 -0400 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <1205430199.32152.152.camel@hias.m3w.org> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> <1205425734.32152.144.camel@hias.m3w.org> <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> <1205430199.32152.152.camel@hias.m3w.org> Message-ID: The thing about conservatism is that even if you NIL *all* references to an object that just happens to lie on a pinned page, that object (and objects reachable from it) will be retained until the page is no longer pinned. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Mar 13, 2008, at 1:43 PM, Dragi?a Duri? wrote: > I understand transitiveness :), but I've made big effort to escape > that. > > Lifetime of thread equals lifetime of Client.T instance, and that > instance is kept in thread's closure, and also in local variable. Both > are set to NIL before thread termination - all terminations and joins > happen and are counted and shown. > > Pointers to Message objects are kept in two places. Hub.T keeps > pointer > to list tail, and each Client.T contains pointer to it's current > element. That pointer is forwarded as Message's are added to end. No > temporary variable, not even WITH substitution is used. And these > threads are ones dying with their Client.T's. > > "head" is new mechanism used to unlink all elements with id less than > minimal id used in active Client.T instances. Unlinked totally and all > as one free for garbage collector to finish them. > > I see no transition. > > dd > > On Thu, 2008-03-13 at 12:45 -0400, Tony Hosking wrote: >> Realizing that conservative roots collectors like ours can retain >> data >> unnecessarily. If a thread stack refers to a page that holds a >> reference that (transitively) refers to some otherwise dead object >> then that object will be retained. We say that the page is pinned. >> Smartening up the collector to ignore dead objects in pinned pages is >> possible, but not currently implemented. >> >> Antony Hosking | Associate Professor | Computer Science | Purdue >> University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> >> >> On Mar 13, 2008, at 12:28 PM, Dragi?a Duri? wrote: >> >>> [17:25:19] 117 users, 1 ops and 1 unresolved; minId = >>> 18617; thrs = 120 >>> [17:25:19] started = 663; cleaned = 452; nilled = 543; >>> joined = 543 >>> [17:25:19] Dangling thrs-a-b-n = 1 >>> [17:25:19] Live queue end is 18621 (head = 18616), >>> maxClean = 18611, cleaned = 16583 >>> >>> About one hour uptime, after I added RTCollector.Collect() call >>> every 20 >>> seconds. >>> >>> dd >>> >>> On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking wrote: >>>> Yes. RTCollectorSRC.FinishCollection; >>>> RTCollectorSRC.StartCollection. This will finish the current >>>> (possibly minor) collection, and start a major collection. The >>>> major >>>> collection must finish before you can guarantee garbage has been >>>> freed, so you may want to RTCollectorSRC.FinishCollection. This >>>> sequence is packaged as RTCollector.Collect. >>>> >>>> Antony Hosking | Associate Professor | Computer Science | Purdue >>>> University >>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>> >>>> >>>> >>>> >>>> >>>> On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? wrote: >>>> >>>>> Can I force "major collection" ? >>>>> >>>>> dd >>>>> >>>>> On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: >>>>>> Indeed! Can you devise a testcase? Note that the generational >>>>>> collector will retain old objects for some time until a major >>>>>> collection occurs, >>>>>> >>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>> Purdue >>>>>> University >>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: >>>>>> >>>>>>> And not only that, their stackbase is also set to NIL, >>>>>>> meaning >>>>>>> their >>>>>>> stack is not regarded in any way during future collections - >>>>>>> meaning >>>>>>> all >>>>>>> local variables are forgotten once apply method returns. >>>>>>> >>>>>>> That way, "spuriousness" is not an issue, once thread >>>>>>> returns? >>>>>>> >>>>>>> dd >>>>>>> >>>>>>> On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: >>>>>>>> They should get unlinked from the global ring. >>>>>>>> >>>>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>>>> Purdue >>>>>>>> University >>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: >>>>>>>> >>>>>>>>> What happens to thread stacks once threads are Join-ed? >>>>>>>>> >>>>>>>>> dd >>>>>>>>> >>>>>>>>> On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking wrote: >>>>>>>>>> This is probably a result of conservatism for >>>>>>>>>> references >>>>>>>>>> from >>>>>>>>>> thread >>>>>>>>>> stacks which results in spurious retention. It may be >>>>>>>>>> necessary >>>>>>>>>> to be >>>>>>>>>> more careful about what references are held on the >>>>>>>>>> stack >>>>>>>>>> in >>>>>>>>>> the >>>>>>>>>> threads implementation (and elsewhere in your code). >>>>>>>>>> Until we >>>>>>>>>> diagnose the place where objects are being retained it >>>>>>>>>> will be >>>>>>>>>> hard to >>>>>>>>>> pinpoint. You should realize that weak refs are a >>>>>>>>>> problem >>>>>>>>>> in >>>>>>>>>> many >>>>>>>>>> language implementations, not just Modula-3. I look >>>>>>>>>> forward >>>>>>>>>> to >>>>>>>>>> hearing more definitively from you. >>>>>>>>>> >>>>>>>>>> Antony Hosking | Associate Professor | Computer >>>>>>>>>> Science | >>>>>>>>>> Purdue >>>>>>>>>> University >>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | >>>>>>>>>> USA >>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: >>>>>>>>>> >>>>>>>>>>> Problem in my case can be thoroughness of GC... >>>>>>>>>>> Because >>>>>>>>>>> it's >>>>>>>>>>> linked >>>>>>>>>>> list of WeakRef-ed objects, it's always first >>>>>>>>>>> element in >>>>>>>>>>> list to >>>>>>>>>>> become >>>>>>>>>>> unreachable first - all others are reachable at >>>>>>>>>>> least >>>>>>>>>>> through >>>>>>>>>>> previous >>>>>>>>>>> (in list) WeakRef-ed object. So, behaviour I have >>>>>>>>>>> observed >>>>>>>>>>> can >>>>>>>>>>> be >>>>>>>>>>> because of some thousands of freeable objects, that >>>>>>>>>>> one >>>>>>>>>>> is >>>>>>>>>>> missed >>>>>>>>>>> too >>>>>>>>>>> often. >>>>>>>>>>> >>>>>>>>>>> I have other object being cleaned and counted... >>>>>>>>>>> There >>>>>>>>>>> also >>>>>>>>>>> cleanup >>>>>>>>>>> invocation looks like it's lagging too much. I'll >>>>>>>>>>> come >>>>>>>>>>> with >>>>>>>>>>> numbers >>>>>>>>>>> after I have more tests. >>>>>>>>>>> >>>>>>>>>>> dd >>>>>>>>>>> >>>>>>>>>>> On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking >>>>>>>>>>> wrote: >>>>>>>>>>>> Sounds like we need some sort of testcase. Would >>>>>>>>>>>> you >>>>>>>>>>>> be >>>>>>>>>>>> able >>>>>>>>>>>> to >>>>>>>>>>>> devise one? It will be hard to make it >>>>>>>>>>>> deterministic, >>>>>>>>>>>> but >>>>>>>>>>>> at >>>>>>>>>>>> least we >>>>>>>>>>>> should see a non-zero cleanup count. >>>>>>>>>>>> >>>>>>>>>>>> Antony Hosking | Associate Professor | Computer >>>>>>>>>>>> Science | >>>>>>>>>>>> Purdue >>>>>>>>>>>> University >>>>>>>>>>>> 305 N. University Street | West Lafayette | IN >>>>>>>>>>>> 47907 | >>>>>>>>>>>> USA >>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I have single linked list that I use to send >>>>>>>>>>>>> messages to >>>>>>>>>>>>> many >>>>>>>>>>>>> threads. >>>>>>>>>>>>> It's end is locked for addition, and current end >>>>>>>>>>>>> is >>>>>>>>>>>>> given to >>>>>>>>>>>>> each >>>>>>>>>>>>> new >>>>>>>>>>>>> client connecting. This way, all client are >>>>>>>>>>>>> going >>>>>>>>>>>>> towards >>>>>>>>>>>>> end of >>>>>>>>>>>>> the >>>>>>>>>>>>> list so it's head becomes unreferenced and goes >>>>>>>>>>>>> away >>>>>>>>>>>>> by >>>>>>>>>>>>> GC. >>>>>>>>>>>>> >>>>>>>>>>>>> Except it does not. >>>>>>>>>>>>> >>>>>>>>>>>>> I've added WeakRef cleanup and sequential id's >>>>>>>>>>>>> so I >>>>>>>>>>>>> can >>>>>>>>>>>>> record >>>>>>>>>>>>> maximum >>>>>>>>>>>>> freed id for checking. No single cleanup >>>>>>>>>>>>> happens. >>>>>>>>>>>>> >>>>>>>>>>>>> I'll cross check my code and count my references >>>>>>>>>>>>> from >>>>>>>>>>>>> application >>>>>>>>>>>>> side, >>>>>>>>>>>>> but maybe someone else has similar problems? >>>>>>>>>>>>> >>>>>>>>>>>>> dd >>>>>>>>>>>>> -- >>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Dragi?a Duri? >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> Dragi?a Duri? >>>>>>> >>>>>> >>>>> -- >>>>> Dragi?a Duri? >>>>> >>>> >>> -- >>> Dragi?a Duri? >>> >> > -- > Dragi?a Duri? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Thu Mar 13 22:46:14 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 13 Mar 2008 22:46:14 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> <1205425734.32152.144.camel@hias.m3w.org> <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> <1205430199.32152.152.camel@hias.m3w.org> Message-ID: <1205444774.32152.158.camel@hias.m3w.org> Are objects on pages prepacked/moved? On Thu, 2008-03-13 at 17:41 -0400, Tony Hosking wrote: > The thing about conservatism is that even if you NIL *all* references > to an object that just happens to lie on a pinned page, that object > (and objects reachable from it) will be retained until the page is no > longer pinned. > > Antony Hosking | Associate Professor | Computer Science | Purdue > University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > On Mar 13, 2008, at 1:43 PM, Dragi?a Duri? wrote: > > > I understand transitiveness :), but I've made big effort to escape > > that. > > > > Lifetime of thread equals lifetime of Client.T instance, and that > > instance is kept in thread's closure, and also in local variable. > > Both > > are set to NIL before thread termination - all terminations and > > joins > > happen and are counted and shown. > > > > Pointers to Message objects are kept in two places. Hub.T keeps > > pointer > > to list tail, and each Client.T contains pointer to it's current > > element. That pointer is forwarded as Message's are added to end. No > > temporary variable, not even WITH substitution is used. And these > > threads are ones dying with their Client.T's. > > > > "head" is new mechanism used to unlink all elements with id less > > than > > minimal id used in active Client.T instances. Unlinked totally and > > all > > as one free for garbage collector to finish them. > > > > I see no transition. > > > > dd > > > > On Thu, 2008-03-13 at 12:45 -0400, Tony Hosking wrote: > > > Realizing that conservative roots collectors like ours can retain > > > data > > > unnecessarily. If a thread stack refers to a page that holds a > > > reference that (transitively) refers to some otherwise dead object > > > then that object will be retained. We say that the page is > > > pinned. > > > Smartening up the collector to ignore dead objects in pinned pages > > > is > > > possible, but not currently implemented. > > > > > > Antony Hosking | Associate Professor | Computer Science | Purdue > > > University > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 12:28 PM, Dragi?a Duri? wrote: > > > > > > > [17:25:19] 117 users, 1 ops and 1 unresolved; minId = > > > > 18617; thrs = 120 > > > > [17:25:19] started = 663; cleaned = 452; nilled = 543; > > > > joined = 543 > > > > [17:25:19] Dangling thrs-a-b-n = 1 > > > > [17:25:19] Live queue end is 18621 (head = 18616), > > > > maxClean = 18611, cleaned = 16583 > > > > > > > > About one hour uptime, after I added RTCollector.Collect() call > > > > every 20 > > > > seconds. > > > > > > > > dd > > > > > > > > On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking wrote: > > > > > Yes. RTCollectorSRC.FinishCollection; > > > > > RTCollectorSRC.StartCollection. This will finish the current > > > > > (possibly minor) collection, and start a major collection. > > > > > The > > > > > major > > > > > collection must finish before you can guarantee garbage has > > > > > been > > > > > freed, so you may want to RTCollectorSRC.FinishCollection. > > > > > This > > > > > sequence is packaged as RTCollector.Collect. > > > > > > > > > > Antony Hosking | Associate Professor | Computer Science | > > > > > Purdue > > > > > University > > > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? wrote: > > > > > > > > > > > Can I force "major collection" ? > > > > > > > > > > > > dd > > > > > > > > > > > > On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: > > > > > > > Indeed! Can you devise a testcase? Note that the > > > > > > > generational > > > > > > > collector will retain old objects for some time until a > > > > > > > major > > > > > > > collection occurs, > > > > > > > > > > > > > > Antony Hosking | Associate Professor | Computer Science | > > > > > > > Purdue > > > > > > > University > > > > > > > 305 N. University Street | West Lafayette | IN 47907 | USA > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: > > > > > > > > > > > > > > > And not only that, their stackbase is also set to NIL, > > > > > > > > meaning > > > > > > > > their > > > > > > > > stack is not regarded in any way during future > > > > > > > > collections - > > > > > > > > meaning > > > > > > > > all > > > > > > > > local variables are forgotten once apply method returns. > > > > > > > > > > > > > > > > That way, "spuriousness" is not an issue, once thread > > > > > > > > returns? > > > > > > > > > > > > > > > > dd > > > > > > > > > > > > > > > > On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: > > > > > > > > > They should get unlinked from the global ring. > > > > > > > > > > > > > > > > > > Antony Hosking | Associate Professor | Computer > > > > > > > > > Science | > > > > > > > > > Purdue > > > > > > > > > University > > > > > > > > > 305 N. University Street | West Lafayette | IN 47907 | > > > > > > > > > USA > > > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: > > > > > > > > > > > > > > > > > > > What happens to thread stacks once threads are > > > > > > > > > > Join-ed? > > > > > > > > > > > > > > > > > > > > dd > > > > > > > > > > > > > > > > > > > > On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking > > > > > > > > > > wrote: > > > > > > > > > > > This is probably a result of conservatism for > > > > > > > > > > > references > > > > > > > > > > > from > > > > > > > > > > > thread > > > > > > > > > > > stacks which results in spurious retention. It > > > > > > > > > > > may be > > > > > > > > > > > necessary > > > > > > > > > > > to be > > > > > > > > > > > more careful about what references are held on the > > > > > > > > > > > stack > > > > > > > > > > > in > > > > > > > > > > > the > > > > > > > > > > > threads implementation (and elsewhere in your > > > > > > > > > > > code). > > > > > > > > > > > Until we > > > > > > > > > > > diagnose the place where objects are being > > > > > > > > > > > retained it > > > > > > > > > > > will be > > > > > > > > > > > hard to > > > > > > > > > > > pinpoint. You should realize that weak refs are a > > > > > > > > > > > problem > > > > > > > > > > > in > > > > > > > > > > > many > > > > > > > > > > > language implementations, not just Modula-3. I > > > > > > > > > > > look > > > > > > > > > > > forward > > > > > > > > > > > to > > > > > > > > > > > hearing more definitively from you. > > > > > > > > > > > > > > > > > > > > > > Antony Hosking | Associate Professor | Computer > > > > > > > > > > > Science | > > > > > > > > > > > Purdue > > > > > > > > > > > University > > > > > > > > > > > 305 N. University Street | West Lafayette | IN > > > > > > > > > > > 47907 | > > > > > > > > > > > USA > > > > > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: > > > > > > > > > > > > > > > > > > > > > > > Problem in my case can be thoroughness of GC... > > > > > > > > > > > > Because > > > > > > > > > > > > it's > > > > > > > > > > > > linked > > > > > > > > > > > > list of WeakRef-ed objects, it's always first > > > > > > > > > > > > element in > > > > > > > > > > > > list to > > > > > > > > > > > > become > > > > > > > > > > > > unreachable first - all others are reachable at > > > > > > > > > > > > least > > > > > > > > > > > > through > > > > > > > > > > > > previous > > > > > > > > > > > > (in list) WeakRef-ed object. So, behaviour I > > > > > > > > > > > > have > > > > > > > > > > > > observed > > > > > > > > > > > > can > > > > > > > > > > > > be > > > > > > > > > > > > because of some thousands of freeable objects, > > > > > > > > > > > > that > > > > > > > > > > > > one > > > > > > > > > > > > is > > > > > > > > > > > > missed > > > > > > > > > > > > too > > > > > > > > > > > > often. > > > > > > > > > > > > > > > > > > > > > > > > I have other object being cleaned and counted... > > > > > > > > > > > > There > > > > > > > > > > > > also > > > > > > > > > > > > cleanup > > > > > > > > > > > > invocation looks like it's lagging too much. > > > > > > > > > > > > I'll > > > > > > > > > > > > come > > > > > > > > > > > > with > > > > > > > > > > > > numbers > > > > > > > > > > > > after I have more tests. > > > > > > > > > > > > > > > > > > > > > > > > dd > > > > > > > > > > > > > > > > > > > > > > > > On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Sounds like we need some sort of testcase. > > > > > > > > > > > > > Would > > > > > > > > > > > > > you > > > > > > > > > > > > > be > > > > > > > > > > > > > able > > > > > > > > > > > > > to > > > > > > > > > > > > > devise one? It will be hard to make it > > > > > > > > > > > > > deterministic, > > > > > > > > > > > > > but > > > > > > > > > > > > > at > > > > > > > > > > > > > least we > > > > > > > > > > > > > should see a non-zero cleanup count. > > > > > > > > > > > > > > > > > > > > > > > > > > Antony Hosking | Associate Professor | > > > > > > > > > > > > > Computer > > > > > > > > > > > > > Science | > > > > > > > > > > > > > Purdue > > > > > > > > > > > > > University > > > > > > > > > > > > > 305 N. University Street | West Lafayette | IN > > > > > > > > > > > > > 47907 | > > > > > > > > > > > > > USA > > > > > > > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 > > > > > > > > > > > > > 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > I have single linked list that I use to send > > > > > > > > > > > > > > messages to > > > > > > > > > > > > > > many > > > > > > > > > > > > > > threads. > > > > > > > > > > > > > > It's end is locked for addition, and current > > > > > > > > > > > > > > end > > > > > > > > > > > > > > is > > > > > > > > > > > > > > given to > > > > > > > > > > > > > > each > > > > > > > > > > > > > > new > > > > > > > > > > > > > > client connecting. This way, all client are > > > > > > > > > > > > > > going > > > > > > > > > > > > > > towards > > > > > > > > > > > > > > end of > > > > > > > > > > > > > > the > > > > > > > > > > > > > > list so it's head becomes unreferenced and > > > > > > > > > > > > > > goes > > > > > > > > > > > > > > away > > > > > > > > > > > > > > by > > > > > > > > > > > > > > GC. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Except it does not. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I've added WeakRef cleanup and sequential > > > > > > > > > > > > > > id's > > > > > > > > > > > > > > so I > > > > > > > > > > > > > > can > > > > > > > > > > > > > > record > > > > > > > > > > > > > > maximum > > > > > > > > > > > > > > freed id for checking. No single cleanup > > > > > > > > > > > > > > happens. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'll cross check my code and count my > > > > > > > > > > > > > > references > > > > > > > > > > > > > > from > > > > > > > > > > > > > > application > > > > > > > > > > > > > > side, > > > > > > > > > > > > > > but maybe someone else has similar problems? > > > > > > > > > > > > > > > > > > > > > > > > > > > > dd > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Dragi?a Duri? > > > > > > > > > > > > > > > -- > > > > Dragi?a Duri? > > > > > > > > > -- > > Dragi?a Duri? > > > -- Dragi?a Duri? From hosking at cs.purdue.edu Thu Mar 13 23:03:36 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 13 Mar 2008 18:03:36 -0400 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <1205444774.32152.158.camel@hias.m3w.org> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> <1205425734.32152.144.camel@hias.m3w.org> <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> <1205430199.32152.152.camel@hias.m3w.org> <1205444774.32152.158.camel@hias.m3w.org> Message-ID: <5649BA54-5B2F-423C-AEF7-65FF09CF3763@cs.purdue.edu> I'm not sure I understand what you mean? On Mar 13, 2008, at 5:46 PM, Dragi?a Duri? wrote: > Are objects on pages prepacked/moved? > > On Thu, 2008-03-13 at 17:41 -0400, Tony Hosking wrote: >> The thing about conservatism is that even if you NIL *all* references >> to an object that just happens to lie on a pinned page, that object >> (and objects reachable from it) will be retained until the page is no >> longer pinned. >> >> Antony Hosking | Associate Professor | Computer Science | Purdue >> University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> >> >> On Mar 13, 2008, at 1:43 PM, Dragi?a Duri? wrote: >> >>> I understand transitiveness :), but I've made big effort to escape >>> that. >>> >>> Lifetime of thread equals lifetime of Client.T instance, and that >>> instance is kept in thread's closure, and also in local variable. >>> Both >>> are set to NIL before thread termination - all terminations and >>> joins >>> happen and are counted and shown. >>> >>> Pointers to Message objects are kept in two places. Hub.T keeps >>> pointer >>> to list tail, and each Client.T contains pointer to it's current >>> element. That pointer is forwarded as Message's are added to end. No >>> temporary variable, not even WITH substitution is used. And these >>> threads are ones dying with their Client.T's. >>> >>> "head" is new mechanism used to unlink all elements with id less >>> than >>> minimal id used in active Client.T instances. Unlinked totally and >>> all >>> as one free for garbage collector to finish them. >>> >>> I see no transition. >>> >>> dd >>> >>> On Thu, 2008-03-13 at 12:45 -0400, Tony Hosking wrote: >>>> Realizing that conservative roots collectors like ours can retain >>>> data >>>> unnecessarily. If a thread stack refers to a page that holds a >>>> reference that (transitively) refers to some otherwise dead object >>>> then that object will be retained. We say that the page is >>>> pinned. >>>> Smartening up the collector to ignore dead objects in pinned pages >>>> is >>>> possible, but not currently implemented. >>>> >>>> Antony Hosking | Associate Professor | Computer Science | Purdue >>>> University >>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>> >>>> >>>> >>>> >>>> >>>> On Mar 13, 2008, at 12:28 PM, Dragi?a Duri? wrote: >>>> >>>>> [17:25:19] 117 users, 1 ops and 1 unresolved; minId = >>>>> 18617; thrs = 120 >>>>> [17:25:19] started = 663; cleaned = 452; nilled = 543; >>>>> joined = 543 >>>>> [17:25:19] Dangling thrs-a-b-n = 1 >>>>> [17:25:19] Live queue end is 18621 (head = 18616), >>>>> maxClean = 18611, cleaned = 16583 >>>>> >>>>> About one hour uptime, after I added RTCollector.Collect() call >>>>> every 20 >>>>> seconds. >>>>> >>>>> dd >>>>> >>>>> On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking wrote: >>>>>> Yes. RTCollectorSRC.FinishCollection; >>>>>> RTCollectorSRC.StartCollection. This will finish the current >>>>>> (possibly minor) collection, and start a major collection. >>>>>> The >>>>>> major >>>>>> collection must finish before you can guarantee garbage has >>>>>> been >>>>>> freed, so you may want to RTCollectorSRC.FinishCollection. >>>>>> This >>>>>> sequence is packaged as RTCollector.Collect. >>>>>> >>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>> Purdue >>>>>> University >>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? wrote: >>>>>> >>>>>>> Can I force "major collection" ? >>>>>>> >>>>>>> dd >>>>>>> >>>>>>> On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: >>>>>>>> Indeed! Can you devise a testcase? Note that the >>>>>>>> generational >>>>>>>> collector will retain old objects for some time until a >>>>>>>> major >>>>>>>> collection occurs, >>>>>>>> >>>>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>>>> Purdue >>>>>>>> University >>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: >>>>>>>> >>>>>>>>> And not only that, their stackbase is also set to NIL, >>>>>>>>> meaning >>>>>>>>> their >>>>>>>>> stack is not regarded in any way during future >>>>>>>>> collections - >>>>>>>>> meaning >>>>>>>>> all >>>>>>>>> local variables are forgotten once apply method returns. >>>>>>>>> >>>>>>>>> That way, "spuriousness" is not an issue, once thread >>>>>>>>> returns? >>>>>>>>> >>>>>>>>> dd >>>>>>>>> >>>>>>>>> On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: >>>>>>>>>> They should get unlinked from the global ring. >>>>>>>>>> >>>>>>>>>> Antony Hosking | Associate Professor | Computer >>>>>>>>>> Science | >>>>>>>>>> Purdue >>>>>>>>>> University >>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | >>>>>>>>>> USA >>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: >>>>>>>>>> >>>>>>>>>>> What happens to thread stacks once threads are >>>>>>>>>>> Join-ed? >>>>>>>>>>> >>>>>>>>>>> dd >>>>>>>>>>> >>>>>>>>>>> On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking >>>>>>>>>>> wrote: >>>>>>>>>>>> This is probably a result of conservatism for >>>>>>>>>>>> references >>>>>>>>>>>> from >>>>>>>>>>>> thread >>>>>>>>>>>> stacks which results in spurious retention. It >>>>>>>>>>>> may be >>>>>>>>>>>> necessary >>>>>>>>>>>> to be >>>>>>>>>>>> more careful about what references are held on the >>>>>>>>>>>> stack >>>>>>>>>>>> in >>>>>>>>>>>> the >>>>>>>>>>>> threads implementation (and elsewhere in your >>>>>>>>>>>> code). >>>>>>>>>>>> Until we >>>>>>>>>>>> diagnose the place where objects are being >>>>>>>>>>>> retained it >>>>>>>>>>>> will be >>>>>>>>>>>> hard to >>>>>>>>>>>> pinpoint. You should realize that weak refs are a >>>>>>>>>>>> problem >>>>>>>>>>>> in >>>>>>>>>>>> many >>>>>>>>>>>> language implementations, not just Modula-3. I >>>>>>>>>>>> look >>>>>>>>>>>> forward >>>>>>>>>>>> to >>>>>>>>>>>> hearing more definitively from you. >>>>>>>>>>>> >>>>>>>>>>>> Antony Hosking | Associate Professor | Computer >>>>>>>>>>>> Science | >>>>>>>>>>>> Purdue >>>>>>>>>>>> University >>>>>>>>>>>> 305 N. University Street | West Lafayette | IN >>>>>>>>>>>> 47907 | >>>>>>>>>>>> USA >>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Problem in my case can be thoroughness of GC... >>>>>>>>>>>>> Because >>>>>>>>>>>>> it's >>>>>>>>>>>>> linked >>>>>>>>>>>>> list of WeakRef-ed objects, it's always first >>>>>>>>>>>>> element in >>>>>>>>>>>>> list to >>>>>>>>>>>>> become >>>>>>>>>>>>> unreachable first - all others are reachable at >>>>>>>>>>>>> least >>>>>>>>>>>>> through >>>>>>>>>>>>> previous >>>>>>>>>>>>> (in list) WeakRef-ed object. So, behaviour I >>>>>>>>>>>>> have >>>>>>>>>>>>> observed >>>>>>>>>>>>> can >>>>>>>>>>>>> be >>>>>>>>>>>>> because of some thousands of freeable objects, >>>>>>>>>>>>> that >>>>>>>>>>>>> one >>>>>>>>>>>>> is >>>>>>>>>>>>> missed >>>>>>>>>>>>> too >>>>>>>>>>>>> often. >>>>>>>>>>>>> >>>>>>>>>>>>> I have other object being cleaned and counted... >>>>>>>>>>>>> There >>>>>>>>>>>>> also >>>>>>>>>>>>> cleanup >>>>>>>>>>>>> invocation looks like it's lagging too much. >>>>>>>>>>>>> I'll >>>>>>>>>>>>> come >>>>>>>>>>>>> with >>>>>>>>>>>>> numbers >>>>>>>>>>>>> after I have more tests. >>>>>>>>>>>>> >>>>>>>>>>>>> dd >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> Sounds like we need some sort of testcase. >>>>>>>>>>>>>> Would >>>>>>>>>>>>>> you >>>>>>>>>>>>>> be >>>>>>>>>>>>>> able >>>>>>>>>>>>>> to >>>>>>>>>>>>>> devise one? It will be hard to make it >>>>>>>>>>>>>> deterministic, >>>>>>>>>>>>>> but >>>>>>>>>>>>>> at >>>>>>>>>>>>>> least we >>>>>>>>>>>>>> should see a non-zero cleanup count. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Antony Hosking | Associate Professor | >>>>>>>>>>>>>> Computer >>>>>>>>>>>>>> Science | >>>>>>>>>>>>>> Purdue >>>>>>>>>>>>>> University >>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN >>>>>>>>>>>>>> 47907 | >>>>>>>>>>>>>> USA >>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 >>>>>>>>>>>>>> 5484 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I have single linked list that I use to send >>>>>>>>>>>>>>> messages to >>>>>>>>>>>>>>> many >>>>>>>>>>>>>>> threads. >>>>>>>>>>>>>>> It's end is locked for addition, and current >>>>>>>>>>>>>>> end >>>>>>>>>>>>>>> is >>>>>>>>>>>>>>> given to >>>>>>>>>>>>>>> each >>>>>>>>>>>>>>> new >>>>>>>>>>>>>>> client connecting. This way, all client are >>>>>>>>>>>>>>> going >>>>>>>>>>>>>>> towards >>>>>>>>>>>>>>> end of >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> list so it's head becomes unreferenced and >>>>>>>>>>>>>>> goes >>>>>>>>>>>>>>> away >>>>>>>>>>>>>>> by >>>>>>>>>>>>>>> GC. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Except it does not. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I've added WeakRef cleanup and sequential >>>>>>>>>>>>>>> id's >>>>>>>>>>>>>>> so I >>>>>>>>>>>>>>> can >>>>>>>>>>>>>>> record >>>>>>>>>>>>>>> maximum >>>>>>>>>>>>>>> freed id for checking. No single cleanup >>>>>>>>>>>>>>> happens. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'll cross check my code and count my >>>>>>>>>>>>>>> references >>>>>>>>>>>>>>> from >>>>>>>>>>>>>>> application >>>>>>>>>>>>>>> side, >>>>>>>>>>>>>>> but maybe someone else has similar problems? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> dd >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Dragi?a Duri? >>>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> Dragi?a Duri? >>>>>>> >>>>>> >>>>> -- >>>>> Dragi?a Duri? >>>>> >>>> >>> -- >>> Dragi?a Duri? >>> >> > -- > Dragi?a Duri? From dragisha at m3w.org Thu Mar 13 23:09:38 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 13 Mar 2008 23:09:38 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <5649BA54-5B2F-423C-AEF7-65FF09CF3763@cs.purdue.edu> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> <1205425734.32152.144.camel@hias.m3w.org> <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> <1205430199.32152.152.camel@hias.m3w.org> <1205444774.32152.158.camel@hias.m3w.org> <5649BA54-5B2F-423C-AEF7-65FF09CF3763@cs.purdue.edu> Message-ID: <1205446178.32152.162.camel@hias.m3w.org> When object becomes unreachable, and it sits on page with other reachable objects, it's pinned? What I asked is... As garbage collector moves objects around, would it pack reachable objects over unreachable ones in these pages? Thus kicking them out of pages where they're pinned? dd On Thu, 2008-03-13 at 18:03 -0400, Tony Hosking wrote: > I'm not sure I understand what you mean? > > On Mar 13, 2008, at 5:46 PM, Dragi?a Duri? wrote: > > > Are objects on pages prepacked/moved? > > > > On Thu, 2008-03-13 at 17:41 -0400, Tony Hosking wrote: > >> The thing about conservatism is that even if you NIL *all* references > >> to an object that just happens to lie on a pinned page, that object > >> (and objects reachable from it) will be retained until the page is no > >> longer pinned. > >> > >> Antony Hosking | Associate Professor | Computer Science | Purdue > >> University > >> 305 N. University Street | West Lafayette | IN 47907 | USA > >> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >> > >> > >> > >> > >> > >> On Mar 13, 2008, at 1:43 PM, Dragi?a Duri? wrote: > >> > >>> I understand transitiveness :), but I've made big effort to escape > >>> that. > >>> > >>> Lifetime of thread equals lifetime of Client.T instance, and that > >>> instance is kept in thread's closure, and also in local variable. > >>> Both > >>> are set to NIL before thread termination - all terminations and > >>> joins > >>> happen and are counted and shown. > >>> > >>> Pointers to Message objects are kept in two places. Hub.T keeps > >>> pointer > >>> to list tail, and each Client.T contains pointer to it's current > >>> element. That pointer is forwarded as Message's are added to end. No > >>> temporary variable, not even WITH substitution is used. And these > >>> threads are ones dying with their Client.T's. > >>> > >>> "head" is new mechanism used to unlink all elements with id less > >>> than > >>> minimal id used in active Client.T instances. Unlinked totally and > >>> all > >>> as one free for garbage collector to finish them. > >>> > >>> I see no transition. > >>> > >>> dd > >>> > >>> On Thu, 2008-03-13 at 12:45 -0400, Tony Hosking wrote: > >>>> Realizing that conservative roots collectors like ours can retain > >>>> data > >>>> unnecessarily. If a thread stack refers to a page that holds a > >>>> reference that (transitively) refers to some otherwise dead object > >>>> then that object will be retained. We say that the page is > >>>> pinned. > >>>> Smartening up the collector to ignore dead objects in pinned pages > >>>> is > >>>> possible, but not currently implemented. > >>>> > >>>> Antony Hosking | Associate Professor | Computer Science | Purdue > >>>> University > >>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On Mar 13, 2008, at 12:28 PM, Dragi?a Duri? wrote: > >>>> > >>>>> [17:25:19] 117 users, 1 ops and 1 unresolved; minId = > >>>>> 18617; thrs = 120 > >>>>> [17:25:19] started = 663; cleaned = 452; nilled = 543; > >>>>> joined = 543 > >>>>> [17:25:19] Dangling thrs-a-b-n = 1 > >>>>> [17:25:19] Live queue end is 18621 (head = 18616), > >>>>> maxClean = 18611, cleaned = 16583 > >>>>> > >>>>> About one hour uptime, after I added RTCollector.Collect() call > >>>>> every 20 > >>>>> seconds. > >>>>> > >>>>> dd > >>>>> > >>>>> On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking wrote: > >>>>>> Yes. RTCollectorSRC.FinishCollection; > >>>>>> RTCollectorSRC.StartCollection. This will finish the current > >>>>>> (possibly minor) collection, and start a major collection. > >>>>>> The > >>>>>> major > >>>>>> collection must finish before you can guarantee garbage has > >>>>>> been > >>>>>> freed, so you may want to RTCollectorSRC.FinishCollection. > >>>>>> This > >>>>>> sequence is packaged as RTCollector.Collect. > >>>>>> > >>>>>> Antony Hosking | Associate Professor | Computer Science | > >>>>>> Purdue > >>>>>> University > >>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? wrote: > >>>>>> > >>>>>>> Can I force "major collection" ? > >>>>>>> > >>>>>>> dd > >>>>>>> > >>>>>>> On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: > >>>>>>>> Indeed! Can you devise a testcase? Note that the > >>>>>>>> generational > >>>>>>>> collector will retain old objects for some time until a > >>>>>>>> major > >>>>>>>> collection occurs, > >>>>>>>> > >>>>>>>> Antony Hosking | Associate Professor | Computer Science | > >>>>>>>> Purdue > >>>>>>>> University > >>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: > >>>>>>>> > >>>>>>>>> And not only that, their stackbase is also set to NIL, > >>>>>>>>> meaning > >>>>>>>>> their > >>>>>>>>> stack is not regarded in any way during future > >>>>>>>>> collections - > >>>>>>>>> meaning > >>>>>>>>> all > >>>>>>>>> local variables are forgotten once apply method returns. > >>>>>>>>> > >>>>>>>>> That way, "spuriousness" is not an issue, once thread > >>>>>>>>> returns? > >>>>>>>>> > >>>>>>>>> dd > >>>>>>>>> > >>>>>>>>> On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: > >>>>>>>>>> They should get unlinked from the global ring. > >>>>>>>>>> > >>>>>>>>>> Antony Hosking | Associate Professor | Computer > >>>>>>>>>> Science | > >>>>>>>>>> Purdue > >>>>>>>>>> University > >>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | > >>>>>>>>>> USA > >>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: > >>>>>>>>>> > >>>>>>>>>>> What happens to thread stacks once threads are > >>>>>>>>>>> Join-ed? > >>>>>>>>>>> > >>>>>>>>>>> dd > >>>>>>>>>>> > >>>>>>>>>>> On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking > >>>>>>>>>>> wrote: > >>>>>>>>>>>> This is probably a result of conservatism for > >>>>>>>>>>>> references > >>>>>>>>>>>> from > >>>>>>>>>>>> thread > >>>>>>>>>>>> stacks which results in spurious retention. It > >>>>>>>>>>>> may be > >>>>>>>>>>>> necessary > >>>>>>>>>>>> to be > >>>>>>>>>>>> more careful about what references are held on the > >>>>>>>>>>>> stack > >>>>>>>>>>>> in > >>>>>>>>>>>> the > >>>>>>>>>>>> threads implementation (and elsewhere in your > >>>>>>>>>>>> code). > >>>>>>>>>>>> Until we > >>>>>>>>>>>> diagnose the place where objects are being > >>>>>>>>>>>> retained it > >>>>>>>>>>>> will be > >>>>>>>>>>>> hard to > >>>>>>>>>>>> pinpoint. You should realize that weak refs are a > >>>>>>>>>>>> problem > >>>>>>>>>>>> in > >>>>>>>>>>>> many > >>>>>>>>>>>> language implementations, not just Modula-3. I > >>>>>>>>>>>> look > >>>>>>>>>>>> forward > >>>>>>>>>>>> to > >>>>>>>>>>>> hearing more definitively from you. > >>>>>>>>>>>> > >>>>>>>>>>>> Antony Hosking | Associate Professor | Computer > >>>>>>>>>>>> Science | > >>>>>>>>>>>> Purdue > >>>>>>>>>>>> University > >>>>>>>>>>>> 305 N. University Street | West Lafayette | IN > >>>>>>>>>>>> 47907 | > >>>>>>>>>>>> USA > >>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> Problem in my case can be thoroughness of GC... > >>>>>>>>>>>>> Because > >>>>>>>>>>>>> it's > >>>>>>>>>>>>> linked > >>>>>>>>>>>>> list of WeakRef-ed objects, it's always first > >>>>>>>>>>>>> element in > >>>>>>>>>>>>> list to > >>>>>>>>>>>>> become > >>>>>>>>>>>>> unreachable first - all others are reachable at > >>>>>>>>>>>>> least > >>>>>>>>>>>>> through > >>>>>>>>>>>>> previous > >>>>>>>>>>>>> (in list) WeakRef-ed object. So, behaviour I > >>>>>>>>>>>>> have > >>>>>>>>>>>>> observed > >>>>>>>>>>>>> can > >>>>>>>>>>>>> be > >>>>>>>>>>>>> because of some thousands of freeable objects, > >>>>>>>>>>>>> that > >>>>>>>>>>>>> one > >>>>>>>>>>>>> is > >>>>>>>>>>>>> missed > >>>>>>>>>>>>> too > >>>>>>>>>>>>> often. > >>>>>>>>>>>>> > >>>>>>>>>>>>> I have other object being cleaned and counted... > >>>>>>>>>>>>> There > >>>>>>>>>>>>> also > >>>>>>>>>>>>> cleanup > >>>>>>>>>>>>> invocation looks like it's lagging too much. > >>>>>>>>>>>>> I'll > >>>>>>>>>>>>> come > >>>>>>>>>>>>> with > >>>>>>>>>>>>> numbers > >>>>>>>>>>>>> after I have more tests. > >>>>>>>>>>>>> > >>>>>>>>>>>>> dd > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> Sounds like we need some sort of testcase. > >>>>>>>>>>>>>> Would > >>>>>>>>>>>>>> you > >>>>>>>>>>>>>> be > >>>>>>>>>>>>>> able > >>>>>>>>>>>>>> to > >>>>>>>>>>>>>> devise one? It will be hard to make it > >>>>>>>>>>>>>> deterministic, > >>>>>>>>>>>>>> but > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> least we > >>>>>>>>>>>>>> should see a non-zero cleanup count. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Antony Hosking | Associate Professor | > >>>>>>>>>>>>>> Computer > >>>>>>>>>>>>>> Science | > >>>>>>>>>>>>>> Purdue > >>>>>>>>>>>>>> University > >>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN > >>>>>>>>>>>>>> 47907 | > >>>>>>>>>>>>>> USA > >>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 > >>>>>>>>>>>>>> 5484 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? > >>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I have single linked list that I use to send > >>>>>>>>>>>>>>> messages to > >>>>>>>>>>>>>>> many > >>>>>>>>>>>>>>> threads. > >>>>>>>>>>>>>>> It's end is locked for addition, and current > >>>>>>>>>>>>>>> end > >>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>> given to > >>>>>>>>>>>>>>> each > >>>>>>>>>>>>>>> new > >>>>>>>>>>>>>>> client connecting. This way, all client are > >>>>>>>>>>>>>>> going > >>>>>>>>>>>>>>> towards > >>>>>>>>>>>>>>> end of > >>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>> list so it's head becomes unreferenced and > >>>>>>>>>>>>>>> goes > >>>>>>>>>>>>>>> away > >>>>>>>>>>>>>>> by > >>>>>>>>>>>>>>> GC. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Except it does not. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I've added WeakRef cleanup and sequential > >>>>>>>>>>>>>>> id's > >>>>>>>>>>>>>>> so I > >>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>> record > >>>>>>>>>>>>>>> maximum > >>>>>>>>>>>>>>> freed id for checking. No single cleanup > >>>>>>>>>>>>>>> happens. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I'll cross check my code and count my > >>>>>>>>>>>>>>> references > >>>>>>>>>>>>>>> from > >>>>>>>>>>>>>>> application > >>>>>>>>>>>>>>> side, > >>>>>>>>>>>>>>> but maybe someone else has similar problems? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> dd > >>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>> Dragi?a Duri? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> -- > >>>>>>>>>>>>> Dragi?a Duri? > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> Dragi?a Duri? > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> Dragi?a Duri? > >>>>>>>>> > >>>>>>>> > >>>>>>> -- > >>>>>>> Dragi?a Duri? > >>>>>>> > >>>>>> > >>>>> -- > >>>>> Dragi?a Duri? > >>>>> > >>>> > >>> -- > >>> Dragi?a Duri? > >>> > >> > > -- > > Dragi?a Duri? > -- Dragi?a Duri? From hosking at cs.purdue.edu Thu Mar 13 23:15:59 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 13 Mar 2008 18:15:59 -0400 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <1205446178.32152.162.camel@hias.m3w.org> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> <1205425734.32152.144.camel@hias.m3w.org> <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> <1205430199.32152.152.camel@hias.m3w.org> <1205444774.32152.158.camel@hias.m3w.org> <5649BA54-5B2F-423C-AEF7-65FF09CF3763@cs.purdue.edu> <1205446178.32152.162.camel@hias.m3w.org> Message-ID: <4DD89DC7-FA3C-4B61-A2A1-5C8488CF3A3B@cs.purdue.edu> No, when a page is pinned because one of its objects is directly referenced from a thread stack then all objects on that page, both live and dead, are retained. The collector does not move objects to or from pinned pages. The objects on the page can only be freed when the page is no longer pinned by a direct reference from any thread stack. Moral of the story is to NIL references aggressively in local variables. On Mar 13, 2008, at 6:09 PM, Dragi?a Duri? wrote: > When object becomes unreachable, and it sits on page with other > reachable objects, it's pinned? > > What I asked is... As garbage collector moves objects around, would > it > pack reachable objects over unreachable ones in these pages? Thus > kicking them out of pages where they're pinned? > > dd > > On Thu, 2008-03-13 at 18:03 -0400, Tony Hosking wrote: >> I'm not sure I understand what you mean? >> >> On Mar 13, 2008, at 5:46 PM, Dragi?a Duri? wrote: >> >>> Are objects on pages prepacked/moved? >>> >>> On Thu, 2008-03-13 at 17:41 -0400, Tony Hosking wrote: >>>> The thing about conservatism is that even if you NIL *all* >>>> references >>>> to an object that just happens to lie on a pinned page, that object >>>> (and objects reachable from it) will be retained until the page >>>> is no >>>> longer pinned. >>>> >>>> Antony Hosking | Associate Professor | Computer Science | Purdue >>>> University >>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>> >>>> >>>> >>>> >>>> >>>> On Mar 13, 2008, at 1:43 PM, Dragi?a Duri? wrote: >>>> >>>>> I understand transitiveness :), but I've made big effort to escape >>>>> that. >>>>> >>>>> Lifetime of thread equals lifetime of Client.T instance, and that >>>>> instance is kept in thread's closure, and also in local variable. >>>>> Both >>>>> are set to NIL before thread termination - all terminations and >>>>> joins >>>>> happen and are counted and shown. >>>>> >>>>> Pointers to Message objects are kept in two places. Hub.T keeps >>>>> pointer >>>>> to list tail, and each Client.T contains pointer to it's current >>>>> element. That pointer is forwarded as Message's are added to >>>>> end. No >>>>> temporary variable, not even WITH substitution is used. And these >>>>> threads are ones dying with their Client.T's. >>>>> >>>>> "head" is new mechanism used to unlink all elements with id less >>>>> than >>>>> minimal id used in active Client.T instances. Unlinked totally and >>>>> all >>>>> as one free for garbage collector to finish them. >>>>> >>>>> I see no transition. >>>>> >>>>> dd >>>>> >>>>> On Thu, 2008-03-13 at 12:45 -0400, Tony Hosking wrote: >>>>>> Realizing that conservative roots collectors like ours can retain >>>>>> data >>>>>> unnecessarily. If a thread stack refers to a page that holds a >>>>>> reference that (transitively) refers to some otherwise dead >>>>>> object >>>>>> then that object will be retained. We say that the page is >>>>>> pinned. >>>>>> Smartening up the collector to ignore dead objects in pinned >>>>>> pages >>>>>> is >>>>>> possible, but not currently implemented. >>>>>> >>>>>> Antony Hosking | Associate Professor | Computer Science | Purdue >>>>>> University >>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mar 13, 2008, at 12:28 PM, Dragi?a Duri? wrote: >>>>>> >>>>>>> [17:25:19] 117 users, 1 ops and 1 unresolved; minId = >>>>>>> 18617; thrs = 120 >>>>>>> [17:25:19] started = 663; cleaned = 452; nilled = 543; >>>>>>> joined = 543 >>>>>>> [17:25:19] Dangling thrs-a-b-n = 1 >>>>>>> [17:25:19] Live queue end is 18621 (head = 18616), >>>>>>> maxClean = 18611, cleaned = 16583 >>>>>>> >>>>>>> About one hour uptime, after I added RTCollector.Collect() call >>>>>>> every 20 >>>>>>> seconds. >>>>>>> >>>>>>> dd >>>>>>> >>>>>>> On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking wrote: >>>>>>>> Yes. RTCollectorSRC.FinishCollection; >>>>>>>> RTCollectorSRC.StartCollection. This will finish the current >>>>>>>> (possibly minor) collection, and start a major collection. >>>>>>>> The >>>>>>>> major >>>>>>>> collection must finish before you can guarantee garbage has >>>>>>>> been >>>>>>>> freed, so you may want to RTCollectorSRC.FinishCollection. >>>>>>>> This >>>>>>>> sequence is packaged as RTCollector.Collect. >>>>>>>> >>>>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>>>> Purdue >>>>>>>> University >>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? wrote: >>>>>>>> >>>>>>>>> Can I force "major collection" ? >>>>>>>>> >>>>>>>>> dd >>>>>>>>> >>>>>>>>> On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: >>>>>>>>>> Indeed! Can you devise a testcase? Note that the >>>>>>>>>> generational >>>>>>>>>> collector will retain old objects for some time until a >>>>>>>>>> major >>>>>>>>>> collection occurs, >>>>>>>>>> >>>>>>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>>>>>> Purdue >>>>>>>>>> University >>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: >>>>>>>>>> >>>>>>>>>>> And not only that, their stackbase is also set to NIL, >>>>>>>>>>> meaning >>>>>>>>>>> their >>>>>>>>>>> stack is not regarded in any way during future >>>>>>>>>>> collections - >>>>>>>>>>> meaning >>>>>>>>>>> all >>>>>>>>>>> local variables are forgotten once apply method returns. >>>>>>>>>>> >>>>>>>>>>> That way, "spuriousness" is not an issue, once thread >>>>>>>>>>> returns? >>>>>>>>>>> >>>>>>>>>>> dd >>>>>>>>>>> >>>>>>>>>>> On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: >>>>>>>>>>>> They should get unlinked from the global ring. >>>>>>>>>>>> >>>>>>>>>>>> Antony Hosking | Associate Professor | Computer >>>>>>>>>>>> Science | >>>>>>>>>>>> Purdue >>>>>>>>>>>> University >>>>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | >>>>>>>>>>>> USA >>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: >>>>>>>>>>>> >>>>>>>>>>>>> What happens to thread stacks once threads are >>>>>>>>>>>>> Join-ed? >>>>>>>>>>>>> >>>>>>>>>>>>> dd >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> This is probably a result of conservatism for >>>>>>>>>>>>>> references >>>>>>>>>>>>>> from >>>>>>>>>>>>>> thread >>>>>>>>>>>>>> stacks which results in spurious retention. It >>>>>>>>>>>>>> may be >>>>>>>>>>>>>> necessary >>>>>>>>>>>>>> to be >>>>>>>>>>>>>> more careful about what references are held on the >>>>>>>>>>>>>> stack >>>>>>>>>>>>>> in >>>>>>>>>>>>>> the >>>>>>>>>>>>>> threads implementation (and elsewhere in your >>>>>>>>>>>>>> code). >>>>>>>>>>>>>> Until we >>>>>>>>>>>>>> diagnose the place where objects are being >>>>>>>>>>>>>> retained it >>>>>>>>>>>>>> will be >>>>>>>>>>>>>> hard to >>>>>>>>>>>>>> pinpoint. You should realize that weak refs are a >>>>>>>>>>>>>> problem >>>>>>>>>>>>>> in >>>>>>>>>>>>>> many >>>>>>>>>>>>>> language implementations, not just Modula-3. I >>>>>>>>>>>>>> look >>>>>>>>>>>>>> forward >>>>>>>>>>>>>> to >>>>>>>>>>>>>> hearing more definitively from you. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Antony Hosking | Associate Professor | Computer >>>>>>>>>>>>>> Science | >>>>>>>>>>>>>> Purdue >>>>>>>>>>>>>> University >>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN >>>>>>>>>>>>>> 47907 | >>>>>>>>>>>>>> USA >>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Problem in my case can be thoroughness of GC... >>>>>>>>>>>>>>> Because >>>>>>>>>>>>>>> it's >>>>>>>>>>>>>>> linked >>>>>>>>>>>>>>> list of WeakRef-ed objects, it's always first >>>>>>>>>>>>>>> element in >>>>>>>>>>>>>>> list to >>>>>>>>>>>>>>> become >>>>>>>>>>>>>>> unreachable first - all others are reachable at >>>>>>>>>>>>>>> least >>>>>>>>>>>>>>> through >>>>>>>>>>>>>>> previous >>>>>>>>>>>>>>> (in list) WeakRef-ed object. So, behaviour I >>>>>>>>>>>>>>> have >>>>>>>>>>>>>>> observed >>>>>>>>>>>>>>> can >>>>>>>>>>>>>>> be >>>>>>>>>>>>>>> because of some thousands of freeable objects, >>>>>>>>>>>>>>> that >>>>>>>>>>>>>>> one >>>>>>>>>>>>>>> is >>>>>>>>>>>>>>> missed >>>>>>>>>>>>>>> too >>>>>>>>>>>>>>> often. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I have other object being cleaned and counted... >>>>>>>>>>>>>>> There >>>>>>>>>>>>>>> also >>>>>>>>>>>>>>> cleanup >>>>>>>>>>>>>>> invocation looks like it's lagging too much. >>>>>>>>>>>>>>> I'll >>>>>>>>>>>>>>> come >>>>>>>>>>>>>>> with >>>>>>>>>>>>>>> numbers >>>>>>>>>>>>>>> after I have more tests. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> dd >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> Sounds like we need some sort of testcase. >>>>>>>>>>>>>>>> Would >>>>>>>>>>>>>>>> you >>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>> able >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>> devise one? It will be hard to make it >>>>>>>>>>>>>>>> deterministic, >>>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> least we >>>>>>>>>>>>>>>> should see a non-zero cleanup count. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Antony Hosking | Associate Professor | >>>>>>>>>>>>>>>> Computer >>>>>>>>>>>>>>>> Science | >>>>>>>>>>>>>>>> Purdue >>>>>>>>>>>>>>>> University >>>>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN >>>>>>>>>>>>>>>> 47907 | >>>>>>>>>>>>>>>> USA >>>>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 >>>>>>>>>>>>>>>> 5484 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I have single linked list that I use to send >>>>>>>>>>>>>>>>> messages to >>>>>>>>>>>>>>>>> many >>>>>>>>>>>>>>>>> threads. >>>>>>>>>>>>>>>>> It's end is locked for addition, and current >>>>>>>>>>>>>>>>> end >>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>> given to >>>>>>>>>>>>>>>>> each >>>>>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>> client connecting. This way, all client are >>>>>>>>>>>>>>>>> going >>>>>>>>>>>>>>>>> towards >>>>>>>>>>>>>>>>> end of >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> list so it's head becomes unreferenced and >>>>>>>>>>>>>>>>> goes >>>>>>>>>>>>>>>>> away >>>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>> GC. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Except it does not. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I've added WeakRef cleanup and sequential >>>>>>>>>>>>>>>>> id's >>>>>>>>>>>>>>>>> so I >>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>> record >>>>>>>>>>>>>>>>> maximum >>>>>>>>>>>>>>>>> freed id for checking. No single cleanup >>>>>>>>>>>>>>>>> happens. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I'll cross check my code and count my >>>>>>>>>>>>>>>>> references >>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>> application >>>>>>>>>>>>>>>>> side, >>>>>>>>>>>>>>>>> but maybe someone else has similar problems? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> dd >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Dragi?a Duri? >>>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> Dragi?a Duri? >>>>>>> >>>>>> >>>>> -- >>>>> Dragi?a Duri? >>>>> >>>> >>> -- >>> Dragi?a Duri? >> > -- > Dragi?a Duri? From dragisha at m3w.org Thu Mar 13 23:21:42 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 13 Mar 2008 23:21:42 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <4DD89DC7-FA3C-4B61-A2A1-5C8488CF3A3B@cs.purdue.edu> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> <1205425734.32152.144.camel@hias.m3w.org> <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> <1205430199.32152.152.camel@hias.m3w.org> <1205444774.32152.158.camel@hias.m3w.org> <5649BA54-5B2F-423C-AEF7-65FF09CF3763@cs.purdue.edu> <1205446178.32152.162.camel@hias.m3w.org> <4DD89DC7-FA3C-4B61-A2A1-5C8488CF3A3B@cs.purdue.edu> Message-ID: <1205446902.32152.168.camel@hias.m3w.org> I see... What bugs me now is a fact about my flow... I am already doing this, and there is always few thousand objects uncleaned from of type Message... I'll follow this through, to crosscheck everything again. What is nice, after I made this agressive RTCollector.Collect "policy", my memory footprint remains same for many hours... It was swelling before. Thanks, dd On Thu, 2008-03-13 at 18:15 -0400, Tony Hosking wrote: > No, when a page is pinned because one of its objects is directly > referenced from a thread stack then all objects on that page, both > live and dead, are retained. The collector does not move objects to > or from pinned pages. The objects on the page can only be freed when > the page is no longer pinned by a direct reference from any thread > stack. Moral of the story is to NIL references aggressively in local > variables. > > On Mar 13, 2008, at 6:09 PM, Dragi?a Duri? wrote: > > > When object becomes unreachable, and it sits on page with other > > reachable objects, it's pinned? > > > > What I asked is... As garbage collector moves objects around, would > > it > > pack reachable objects over unreachable ones in these pages? Thus > > kicking them out of pages where they're pinned? > > > > dd > > > > On Thu, 2008-03-13 at 18:03 -0400, Tony Hosking wrote: > >> I'm not sure I understand what you mean? > >> > >> On Mar 13, 2008, at 5:46 PM, Dragi?a Duri? wrote: > >> > >>> Are objects on pages prepacked/moved? > >>> > >>> On Thu, 2008-03-13 at 17:41 -0400, Tony Hosking wrote: > >>>> The thing about conservatism is that even if you NIL *all* > >>>> references > >>>> to an object that just happens to lie on a pinned page, that object > >>>> (and objects reachable from it) will be retained until the page > >>>> is no > >>>> longer pinned. > >>>> > >>>> Antony Hosking | Associate Professor | Computer Science | Purdue > >>>> University > >>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On Mar 13, 2008, at 1:43 PM, Dragi?a Duri? wrote: > >>>> > >>>>> I understand transitiveness :), but I've made big effort to escape > >>>>> that. > >>>>> > >>>>> Lifetime of thread equals lifetime of Client.T instance, and that > >>>>> instance is kept in thread's closure, and also in local variable. > >>>>> Both > >>>>> are set to NIL before thread termination - all terminations and > >>>>> joins > >>>>> happen and are counted and shown. > >>>>> > >>>>> Pointers to Message objects are kept in two places. Hub.T keeps > >>>>> pointer > >>>>> to list tail, and each Client.T contains pointer to it's current > >>>>> element. That pointer is forwarded as Message's are added to > >>>>> end. No > >>>>> temporary variable, not even WITH substitution is used. And these > >>>>> threads are ones dying with their Client.T's. > >>>>> > >>>>> "head" is new mechanism used to unlink all elements with id less > >>>>> than > >>>>> minimal id used in active Client.T instances. Unlinked totally and > >>>>> all > >>>>> as one free for garbage collector to finish them. > >>>>> > >>>>> I see no transition. > >>>>> > >>>>> dd > >>>>> > >>>>> On Thu, 2008-03-13 at 12:45 -0400, Tony Hosking wrote: > >>>>>> Realizing that conservative roots collectors like ours can retain > >>>>>> data > >>>>>> unnecessarily. If a thread stack refers to a page that holds a > >>>>>> reference that (transitively) refers to some otherwise dead > >>>>>> object > >>>>>> then that object will be retained. We say that the page is > >>>>>> pinned. > >>>>>> Smartening up the collector to ignore dead objects in pinned > >>>>>> pages > >>>>>> is > >>>>>> possible, but not currently implemented. > >>>>>> > >>>>>> Antony Hosking | Associate Professor | Computer Science | Purdue > >>>>>> University > >>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> On Mar 13, 2008, at 12:28 PM, Dragi?a Duri? wrote: > >>>>>> > >>>>>>> [17:25:19] 117 users, 1 ops and 1 unresolved; minId = > >>>>>>> 18617; thrs = 120 > >>>>>>> [17:25:19] started = 663; cleaned = 452; nilled = 543; > >>>>>>> joined = 543 > >>>>>>> [17:25:19] Dangling thrs-a-b-n = 1 > >>>>>>> [17:25:19] Live queue end is 18621 (head = 18616), > >>>>>>> maxClean = 18611, cleaned = 16583 > >>>>>>> > >>>>>>> About one hour uptime, after I added RTCollector.Collect() call > >>>>>>> every 20 > >>>>>>> seconds. > >>>>>>> > >>>>>>> dd > >>>>>>> > >>>>>>> On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking wrote: > >>>>>>>> Yes. RTCollectorSRC.FinishCollection; > >>>>>>>> RTCollectorSRC.StartCollection. This will finish the current > >>>>>>>> (possibly minor) collection, and start a major collection. > >>>>>>>> The > >>>>>>>> major > >>>>>>>> collection must finish before you can guarantee garbage has > >>>>>>>> been > >>>>>>>> freed, so you may want to RTCollectorSRC.FinishCollection. > >>>>>>>> This > >>>>>>>> sequence is packaged as RTCollector.Collect. > >>>>>>>> > >>>>>>>> Antony Hosking | Associate Professor | Computer Science | > >>>>>>>> Purdue > >>>>>>>> University > >>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? wrote: > >>>>>>>> > >>>>>>>>> Can I force "major collection" ? > >>>>>>>>> > >>>>>>>>> dd > >>>>>>>>> > >>>>>>>>> On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: > >>>>>>>>>> Indeed! Can you devise a testcase? Note that the > >>>>>>>>>> generational > >>>>>>>>>> collector will retain old objects for some time until a > >>>>>>>>>> major > >>>>>>>>>> collection occurs, > >>>>>>>>>> > >>>>>>>>>> Antony Hosking | Associate Professor | Computer Science | > >>>>>>>>>> Purdue > >>>>>>>>>> University > >>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: > >>>>>>>>>> > >>>>>>>>>>> And not only that, their stackbase is also set to NIL, > >>>>>>>>>>> meaning > >>>>>>>>>>> their > >>>>>>>>>>> stack is not regarded in any way during future > >>>>>>>>>>> collections - > >>>>>>>>>>> meaning > >>>>>>>>>>> all > >>>>>>>>>>> local variables are forgotten once apply method returns. > >>>>>>>>>>> > >>>>>>>>>>> That way, "spuriousness" is not an issue, once thread > >>>>>>>>>>> returns? > >>>>>>>>>>> > >>>>>>>>>>> dd > >>>>>>>>>>> > >>>>>>>>>>> On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: > >>>>>>>>>>>> They should get unlinked from the global ring. > >>>>>>>>>>>> > >>>>>>>>>>>> Antony Hosking | Associate Professor | Computer > >>>>>>>>>>>> Science | > >>>>>>>>>>>> Purdue > >>>>>>>>>>>> University > >>>>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | > >>>>>>>>>>>> USA > >>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> What happens to thread stacks once threads are > >>>>>>>>>>>>> Join-ed? > >>>>>>>>>>>>> > >>>>>>>>>>>>> dd > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> This is probably a result of conservatism for > >>>>>>>>>>>>>> references > >>>>>>>>>>>>>> from > >>>>>>>>>>>>>> thread > >>>>>>>>>>>>>> stacks which results in spurious retention. It > >>>>>>>>>>>>>> may be > >>>>>>>>>>>>>> necessary > >>>>>>>>>>>>>> to be > >>>>>>>>>>>>>> more careful about what references are held on the > >>>>>>>>>>>>>> stack > >>>>>>>>>>>>>> in > >>>>>>>>>>>>>> the > >>>>>>>>>>>>>> threads implementation (and elsewhere in your > >>>>>>>>>>>>>> code). > >>>>>>>>>>>>>> Until we > >>>>>>>>>>>>>> diagnose the place where objects are being > >>>>>>>>>>>>>> retained it > >>>>>>>>>>>>>> will be > >>>>>>>>>>>>>> hard to > >>>>>>>>>>>>>> pinpoint. You should realize that weak refs are a > >>>>>>>>>>>>>> problem > >>>>>>>>>>>>>> in > >>>>>>>>>>>>>> many > >>>>>>>>>>>>>> language implementations, not just Modula-3. I > >>>>>>>>>>>>>> look > >>>>>>>>>>>>>> forward > >>>>>>>>>>>>>> to > >>>>>>>>>>>>>> hearing more definitively from you. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Antony Hosking | Associate Professor | Computer > >>>>>>>>>>>>>> Science | > >>>>>>>>>>>>>> Purdue > >>>>>>>>>>>>>> University > >>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN > >>>>>>>>>>>>>> 47907 | > >>>>>>>>>>>>>> USA > >>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Problem in my case can be thoroughness of GC... > >>>>>>>>>>>>>>> Because > >>>>>>>>>>>>>>> it's > >>>>>>>>>>>>>>> linked > >>>>>>>>>>>>>>> list of WeakRef-ed objects, it's always first > >>>>>>>>>>>>>>> element in > >>>>>>>>>>>>>>> list to > >>>>>>>>>>>>>>> become > >>>>>>>>>>>>>>> unreachable first - all others are reachable at > >>>>>>>>>>>>>>> least > >>>>>>>>>>>>>>> through > >>>>>>>>>>>>>>> previous > >>>>>>>>>>>>>>> (in list) WeakRef-ed object. So, behaviour I > >>>>>>>>>>>>>>> have > >>>>>>>>>>>>>>> observed > >>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>> because of some thousands of freeable objects, > >>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>> one > >>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>> missed > >>>>>>>>>>>>>>> too > >>>>>>>>>>>>>>> often. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I have other object being cleaned and counted... > >>>>>>>>>>>>>>> There > >>>>>>>>>>>>>>> also > >>>>>>>>>>>>>>> cleanup > >>>>>>>>>>>>>>> invocation looks like it's lagging too much. > >>>>>>>>>>>>>>> I'll > >>>>>>>>>>>>>>> come > >>>>>>>>>>>>>>> with > >>>>>>>>>>>>>>> numbers > >>>>>>>>>>>>>>> after I have more tests. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> dd > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> Sounds like we need some sort of testcase. > >>>>>>>>>>>>>>>> Would > >>>>>>>>>>>>>>>> you > >>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>> able > >>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>> devise one? It will be hard to make it > >>>>>>>>>>>>>>>> deterministic, > >>>>>>>>>>>>>>>> but > >>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>> least we > >>>>>>>>>>>>>>>> should see a non-zero cleanup count. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Antony Hosking | Associate Professor | > >>>>>>>>>>>>>>>> Computer > >>>>>>>>>>>>>>>> Science | > >>>>>>>>>>>>>>>> Purdue > >>>>>>>>>>>>>>>> University > >>>>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN > >>>>>>>>>>>>>>>> 47907 | > >>>>>>>>>>>>>>>> USA > >>>>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 > >>>>>>>>>>>>>>>> 5484 > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? > >>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I have single linked list that I use to send > >>>>>>>>>>>>>>>>> messages to > >>>>>>>>>>>>>>>>> many > >>>>>>>>>>>>>>>>> threads. > >>>>>>>>>>>>>>>>> It's end is locked for addition, and current > >>>>>>>>>>>>>>>>> end > >>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>> given to > >>>>>>>>>>>>>>>>> each > >>>>>>>>>>>>>>>>> new > >>>>>>>>>>>>>>>>> client connecting. This way, all client are > >>>>>>>>>>>>>>>>> going > >>>>>>>>>>>>>>>>> towards > >>>>>>>>>>>>>>>>> end of > >>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>> list so it's head becomes unreferenced and > >>>>>>>>>>>>>>>>> goes > >>>>>>>>>>>>>>>>> away > >>>>>>>>>>>>>>>>> by > >>>>>>>>>>>>>>>>> GC. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Except it does not. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I've added WeakRef cleanup and sequential > >>>>>>>>>>>>>>>>> id's > >>>>>>>>>>>>>>>>> so I > >>>>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>> record > >>>>>>>>>>>>>>>>> maximum > >>>>>>>>>>>>>>>>> freed id for checking. No single cleanup > >>>>>>>>>>>>>>>>> happens. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I'll cross check my code and count my > >>>>>>>>>>>>>>>>> references > >>>>>>>>>>>>>>>>> from > >>>>>>>>>>>>>>>>> application > >>>>>>>>>>>>>>>>> side, > >>>>>>>>>>>>>>>>> but maybe someone else has similar problems? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> dd > >>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>> Dragi?a Duri? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>> Dragi?a Duri? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> -- > >>>>>>>>>>>>> Dragi?a Duri? > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> Dragi?a Duri? > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> Dragi?a Duri? > >>>>>>>>> > >>>>>>>> > >>>>>>> -- > >>>>>>> Dragi?a Duri? > >>>>>>> > >>>>>> > >>>>> -- > >>>>> Dragi?a Duri? > >>>>> > >>>> > >>> -- > >>> Dragi?a Duri? > >> > > -- > > Dragi?a Duri? > -- Dragi?a Duri? From hosking at cs.purdue.edu Thu Mar 13 23:35:00 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 13 Mar 2008 18:35:00 -0400 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <1205446902.32152.168.camel@hias.m3w.org> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> <1205425734.32152.144.camel@hias.m3w.org> <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> <1205430199.32152.152.camel@hias.m3w.org> <1205444774.32152.158.camel@hias.m3w.org> <5649BA54-5B2F-423C-AEF7-65FF09CF3763@cs.purdue.edu> <1205446178.32152.162.camel@hias.m3w.org> <4DD89DC7-FA3C-4B61-A2A1-5C8488CF3A3B@cs.purdue.edu> <1205446902.32152.168.camel@hias.m3w.org> Message-ID: <2EA2C202-F053-4AE8-BA60-E51A1A339C6B@cs.purdue.edu> Ah, the fact that you needed to be aggressive suggests that conservatism may be the culprit. I keep meaning one day to make sure that we do not can objects on pinned pages unless we know that object is definitely alive. On Mar 13, 2008, at 6:21 PM, Dragi?a Duri? wrote: > I see... What bugs me now is a fact about my flow... I am already > doing > this, and there is always few thousand objects uncleaned from of type > Message... I'll follow this through, to crosscheck everything again. > > What is nice, after I made this agressive RTCollector.Collect > "policy", > my memory footprint remains same for many hours... It was swelling > before. > > Thanks, > dd > > On Thu, 2008-03-13 at 18:15 -0400, Tony Hosking wrote: >> No, when a page is pinned because one of its objects is directly >> referenced from a thread stack then all objects on that page, both >> live and dead, are retained. The collector does not move objects to >> or from pinned pages. The objects on the page can only be freed when >> the page is no longer pinned by a direct reference from any thread >> stack. Moral of the story is to NIL references aggressively in local >> variables. >> >> On Mar 13, 2008, at 6:09 PM, Dragi?a Duri? wrote: >> >>> When object becomes unreachable, and it sits on page with other >>> reachable objects, it's pinned? >>> >>> What I asked is... As garbage collector moves objects around, would >>> it >>> pack reachable objects over unreachable ones in these pages? Thus >>> kicking them out of pages where they're pinned? >>> >>> dd >>> >>> On Thu, 2008-03-13 at 18:03 -0400, Tony Hosking wrote: >>>> I'm not sure I understand what you mean? >>>> >>>> On Mar 13, 2008, at 5:46 PM, Dragi?a Duri? wrote: >>>> >>>>> Are objects on pages prepacked/moved? >>>>> >>>>> On Thu, 2008-03-13 at 17:41 -0400, Tony Hosking wrote: >>>>>> The thing about conservatism is that even if you NIL *all* >>>>>> references >>>>>> to an object that just happens to lie on a pinned page, that >>>>>> object >>>>>> (and objects reachable from it) will be retained until the page >>>>>> is no >>>>>> longer pinned. >>>>>> >>>>>> Antony Hosking | Associate Professor | Computer Science | Purdue >>>>>> University >>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mar 13, 2008, at 1:43 PM, Dragi?a Duri? wrote: >>>>>> >>>>>>> I understand transitiveness :), but I've made big effort to >>>>>>> escape >>>>>>> that. >>>>>>> >>>>>>> Lifetime of thread equals lifetime of Client.T instance, and >>>>>>> that >>>>>>> instance is kept in thread's closure, and also in local >>>>>>> variable. >>>>>>> Both >>>>>>> are set to NIL before thread termination - all terminations and >>>>>>> joins >>>>>>> happen and are counted and shown. >>>>>>> >>>>>>> Pointers to Message objects are kept in two places. Hub.T keeps >>>>>>> pointer >>>>>>> to list tail, and each Client.T contains pointer to it's current >>>>>>> element. That pointer is forwarded as Message's are added to >>>>>>> end. No >>>>>>> temporary variable, not even WITH substitution is used. And >>>>>>> these >>>>>>> threads are ones dying with their Client.T's. >>>>>>> >>>>>>> "head" is new mechanism used to unlink all elements with id less >>>>>>> than >>>>>>> minimal id used in active Client.T instances. Unlinked totally >>>>>>> and >>>>>>> all >>>>>>> as one free for garbage collector to finish them. >>>>>>> >>>>>>> I see no transition. >>>>>>> >>>>>>> dd >>>>>>> >>>>>>> On Thu, 2008-03-13 at 12:45 -0400, Tony Hosking wrote: >>>>>>>> Realizing that conservative roots collectors like ours can >>>>>>>> retain >>>>>>>> data >>>>>>>> unnecessarily. If a thread stack refers to a page that holds a >>>>>>>> reference that (transitively) refers to some otherwise dead >>>>>>>> object >>>>>>>> then that object will be retained. We say that the page is >>>>>>>> pinned. >>>>>>>> Smartening up the collector to ignore dead objects in pinned >>>>>>>> pages >>>>>>>> is >>>>>>>> possible, but not currently implemented. >>>>>>>> >>>>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>>>> Purdue >>>>>>>> University >>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mar 13, 2008, at 12:28 PM, Dragi?a Duri? wrote: >>>>>>>> >>>>>>>>> [17:25:19] 117 users, 1 ops and 1 unresolved; >>>>>>>>> minId = >>>>>>>>> 18617; thrs = 120 >>>>>>>>> [17:25:19] started = 663; cleaned = 452; nilled = >>>>>>>>> 543; >>>>>>>>> joined = 543 >>>>>>>>> [17:25:19] Dangling thrs-a-b-n = 1 >>>>>>>>> [17:25:19] Live queue end is 18621 (head = 18616), >>>>>>>>> maxClean = 18611, cleaned = 16583 >>>>>>>>> >>>>>>>>> About one hour uptime, after I added RTCollector.Collect() >>>>>>>>> call >>>>>>>>> every 20 >>>>>>>>> seconds. >>>>>>>>> >>>>>>>>> dd >>>>>>>>> >>>>>>>>> On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking wrote: >>>>>>>>>> Yes. RTCollectorSRC.FinishCollection; >>>>>>>>>> RTCollectorSRC.StartCollection. This will finish the current >>>>>>>>>> (possibly minor) collection, and start a major collection. >>>>>>>>>> The >>>>>>>>>> major >>>>>>>>>> collection must finish before you can guarantee garbage has >>>>>>>>>> been >>>>>>>>>> freed, so you may want to RTCollectorSRC.FinishCollection. >>>>>>>>>> This >>>>>>>>>> sequence is packaged as RTCollector.Collect. >>>>>>>>>> >>>>>>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>>>>>> Purdue >>>>>>>>>> University >>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? wrote: >>>>>>>>>> >>>>>>>>>>> Can I force "major collection" ? >>>>>>>>>>> >>>>>>>>>>> dd >>>>>>>>>>> >>>>>>>>>>> On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: >>>>>>>>>>>> Indeed! Can you devise a testcase? Note that the >>>>>>>>>>>> generational >>>>>>>>>>>> collector will retain old objects for some time until a >>>>>>>>>>>> major >>>>>>>>>>>> collection occurs, >>>>>>>>>>>> >>>>>>>>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>>>>>>>> Purdue >>>>>>>>>>>> University >>>>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: >>>>>>>>>>>> >>>>>>>>>>>>> And not only that, their stackbase is also set to NIL, >>>>>>>>>>>>> meaning >>>>>>>>>>>>> their >>>>>>>>>>>>> stack is not regarded in any way during future >>>>>>>>>>>>> collections - >>>>>>>>>>>>> meaning >>>>>>>>>>>>> all >>>>>>>>>>>>> local variables are forgotten once apply method returns. >>>>>>>>>>>>> >>>>>>>>>>>>> That way, "spuriousness" is not an issue, once thread >>>>>>>>>>>>> returns? >>>>>>>>>>>>> >>>>>>>>>>>>> dd >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: >>>>>>>>>>>>>> They should get unlinked from the global ring. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Antony Hosking | Associate Professor | Computer >>>>>>>>>>>>>> Science | >>>>>>>>>>>>>> Purdue >>>>>>>>>>>>>> University >>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | >>>>>>>>>>>>>> USA >>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> What happens to thread stacks once threads are >>>>>>>>>>>>>>> Join-ed? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> dd >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> This is probably a result of conservatism for >>>>>>>>>>>>>>>> references >>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>> thread >>>>>>>>>>>>>>>> stacks which results in spurious retention. It >>>>>>>>>>>>>>>> may be >>>>>>>>>>>>>>>> necessary >>>>>>>>>>>>>>>> to be >>>>>>>>>>>>>>>> more careful about what references are held on the >>>>>>>>>>>>>>>> stack >>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> threads implementation (and elsewhere in your >>>>>>>>>>>>>>>> code). >>>>>>>>>>>>>>>> Until we >>>>>>>>>>>>>>>> diagnose the place where objects are being >>>>>>>>>>>>>>>> retained it >>>>>>>>>>>>>>>> will be >>>>>>>>>>>>>>>> hard to >>>>>>>>>>>>>>>> pinpoint. You should realize that weak refs are a >>>>>>>>>>>>>>>> problem >>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>> many >>>>>>>>>>>>>>>> language implementations, not just Modula-3. I >>>>>>>>>>>>>>>> look >>>>>>>>>>>>>>>> forward >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>> hearing more definitively from you. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Antony Hosking | Associate Professor | Computer >>>>>>>>>>>>>>>> Science | >>>>>>>>>>>>>>>> Purdue >>>>>>>>>>>>>>>> University >>>>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN >>>>>>>>>>>>>>>> 47907 | >>>>>>>>>>>>>>>> USA >>>>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Problem in my case can be thoroughness of GC... >>>>>>>>>>>>>>>>> Because >>>>>>>>>>>>>>>>> it's >>>>>>>>>>>>>>>>> linked >>>>>>>>>>>>>>>>> list of WeakRef-ed objects, it's always first >>>>>>>>>>>>>>>>> element in >>>>>>>>>>>>>>>>> list to >>>>>>>>>>>>>>>>> become >>>>>>>>>>>>>>>>> unreachable first - all others are reachable at >>>>>>>>>>>>>>>>> least >>>>>>>>>>>>>>>>> through >>>>>>>>>>>>>>>>> previous >>>>>>>>>>>>>>>>> (in list) WeakRef-ed object. So, behaviour I >>>>>>>>>>>>>>>>> have >>>>>>>>>>>>>>>>> observed >>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>> because of some thousands of freeable objects, >>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>> one >>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>> missed >>>>>>>>>>>>>>>>> too >>>>>>>>>>>>>>>>> often. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I have other object being cleaned and counted... >>>>>>>>>>>>>>>>> There >>>>>>>>>>>>>>>>> also >>>>>>>>>>>>>>>>> cleanup >>>>>>>>>>>>>>>>> invocation looks like it's lagging too much. >>>>>>>>>>>>>>>>> I'll >>>>>>>>>>>>>>>>> come >>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>> numbers >>>>>>>>>>>>>>>>> after I have more tests. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> dd >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> Sounds like we need some sort of testcase. >>>>>>>>>>>>>>>>>> Would >>>>>>>>>>>>>>>>>> you >>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>> able >>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>> devise one? It will be hard to make it >>>>>>>>>>>>>>>>>> deterministic, >>>>>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>> least we >>>>>>>>>>>>>>>>>> should see a non-zero cleanup count. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Antony Hosking | Associate Professor | >>>>>>>>>>>>>>>>>> Computer >>>>>>>>>>>>>>>>>> Science | >>>>>>>>>>>>>>>>>> Purdue >>>>>>>>>>>>>>>>>> University >>>>>>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN >>>>>>>>>>>>>>>>>> 47907 | >>>>>>>>>>>>>>>>>> USA >>>>>>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 >>>>>>>>>>>>>>>>>> 5484 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? >>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I have single linked list that I use to send >>>>>>>>>>>>>>>>>>> messages to >>>>>>>>>>>>>>>>>>> many >>>>>>>>>>>>>>>>>>> threads. >>>>>>>>>>>>>>>>>>> It's end is locked for addition, and current >>>>>>>>>>>>>>>>>>> end >>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>> given to >>>>>>>>>>>>>>>>>>> each >>>>>>>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>>>> client connecting. This way, all client are >>>>>>>>>>>>>>>>>>> going >>>>>>>>>>>>>>>>>>> towards >>>>>>>>>>>>>>>>>>> end of >>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> list so it's head becomes unreferenced and >>>>>>>>>>>>>>>>>>> goes >>>>>>>>>>>>>>>>>>> away >>>>>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>> GC. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Except it does not. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I've added WeakRef cleanup and sequential >>>>>>>>>>>>>>>>>>> id's >>>>>>>>>>>>>>>>>>> so I >>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>> record >>>>>>>>>>>>>>>>>>> maximum >>>>>>>>>>>>>>>>>>> freed id for checking. No single cleanup >>>>>>>>>>>>>>>>>>> happens. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I'll cross check my code and count my >>>>>>>>>>>>>>>>>>> references >>>>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>>>> application >>>>>>>>>>>>>>>>>>> side, >>>>>>>>>>>>>>>>>>> but maybe someone else has similar problems? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> dd >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Dragi?a Duri? >>>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> Dragi?a Duri? >>>>>>> >>>>>> >>>>> -- >>>>> Dragi?a Duri? >>>> >>> -- >>> Dragi?a Duri? >> > -- > Dragi?a Duri? From dragisha at m3w.org Fri Mar 14 00:05:10 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Fri, 14 Mar 2008 00:05:10 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <2EA2C202-F053-4AE8-BA60-E51A1A339C6B@cs.purdue.edu> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> <1205425734.32152.144.camel@hias.m3w.org> <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> <1205430199.32152.152.camel@hias.m3w.org> <1205444774.32152.158.camel@hias.m3w.org> <5649BA54-5B2F-423C-AEF7-65FF09CF3763@cs.purdue.edu> <1205446178.32152.162.camel@hias.m3w.org> <4DD89DC7-FA3C-4B61-A2A1-5C8488CF3A3B@cs.purdue.edu> <1205446902.32152.168.camel@hias.m3w.org> <2EA2C202-F053-4AE8-BA60-E51A1A339C6B@cs.purdue.edu> Message-ID: <1205449510.32152.171.camel@hias.m3w.org> Is your work on GC covered somewhere else except source, comments and this list? dd On Thu, 2008-03-13 at 18:35 -0400, Tony Hosking wrote: > Ah, the fact that you needed to be aggressive suggests that > conservatism may be the culprit. I keep meaning one day to make sure > that we do not can objects on pinned pages unless we know that object > is definitely alive. > > On Mar 13, 2008, at 6:21 PM, Dragi?a Duri? wrote: > > > I see... What bugs me now is a fact about my flow... I am already > > doing > > this, and there is always few thousand objects uncleaned from of type > > Message... I'll follow this through, to crosscheck everything again. > > > > What is nice, after I made this agressive RTCollector.Collect > > "policy", > > my memory footprint remains same for many hours... It was swelling > > before. > > > > Thanks, > > dd > > > > On Thu, 2008-03-13 at 18:15 -0400, Tony Hosking wrote: > >> No, when a page is pinned because one of its objects is directly > >> referenced from a thread stack then all objects on that page, both > >> live and dead, are retained. The collector does not move objects to > >> or from pinned pages. The objects on the page can only be freed when > >> the page is no longer pinned by a direct reference from any thread > >> stack. Moral of the story is to NIL references aggressively in local > >> variables. > >> > >> On Mar 13, 2008, at 6:09 PM, Dragi?a Duri? wrote: > >> > >>> When object becomes unreachable, and it sits on page with other > >>> reachable objects, it's pinned? > >>> > >>> What I asked is... As garbage collector moves objects around, would > >>> it > >>> pack reachable objects over unreachable ones in these pages? Thus > >>> kicking them out of pages where they're pinned? > >>> > >>> dd > >>> > >>> On Thu, 2008-03-13 at 18:03 -0400, Tony Hosking wrote: > >>>> I'm not sure I understand what you mean? > >>>> > >>>> On Mar 13, 2008, at 5:46 PM, Dragi?a Duri? wrote: > >>>> > >>>>> Are objects on pages prepacked/moved? > >>>>> > >>>>> On Thu, 2008-03-13 at 17:41 -0400, Tony Hosking wrote: > >>>>>> The thing about conservatism is that even if you NIL *all* > >>>>>> references > >>>>>> to an object that just happens to lie on a pinned page, that > >>>>>> object > >>>>>> (and objects reachable from it) will be retained until the page > >>>>>> is no > >>>>>> longer pinned. > >>>>>> > >>>>>> Antony Hosking | Associate Professor | Computer Science | Purdue > >>>>>> University > >>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> On Mar 13, 2008, at 1:43 PM, Dragi?a Duri? wrote: > >>>>>> > >>>>>>> I understand transitiveness :), but I've made big effort to > >>>>>>> escape > >>>>>>> that. > >>>>>>> > >>>>>>> Lifetime of thread equals lifetime of Client.T instance, and > >>>>>>> that > >>>>>>> instance is kept in thread's closure, and also in local > >>>>>>> variable. > >>>>>>> Both > >>>>>>> are set to NIL before thread termination - all terminations and > >>>>>>> joins > >>>>>>> happen and are counted and shown. > >>>>>>> > >>>>>>> Pointers to Message objects are kept in two places. Hub.T keeps > >>>>>>> pointer > >>>>>>> to list tail, and each Client.T contains pointer to it's current > >>>>>>> element. That pointer is forwarded as Message's are added to > >>>>>>> end. No > >>>>>>> temporary variable, not even WITH substitution is used. And > >>>>>>> these > >>>>>>> threads are ones dying with their Client.T's. > >>>>>>> > >>>>>>> "head" is new mechanism used to unlink all elements with id less > >>>>>>> than > >>>>>>> minimal id used in active Client.T instances. Unlinked totally > >>>>>>> and > >>>>>>> all > >>>>>>> as one free for garbage collector to finish them. > >>>>>>> > >>>>>>> I see no transition. > >>>>>>> > >>>>>>> dd > >>>>>>> > >>>>>>> On Thu, 2008-03-13 at 12:45 -0400, Tony Hosking wrote: > >>>>>>>> Realizing that conservative roots collectors like ours can > >>>>>>>> retain > >>>>>>>> data > >>>>>>>> unnecessarily. If a thread stack refers to a page that holds a > >>>>>>>> reference that (transitively) refers to some otherwise dead > >>>>>>>> object > >>>>>>>> then that object will be retained. We say that the page is > >>>>>>>> pinned. > >>>>>>>> Smartening up the collector to ignore dead objects in pinned > >>>>>>>> pages > >>>>>>>> is > >>>>>>>> possible, but not currently implemented. > >>>>>>>> > >>>>>>>> Antony Hosking | Associate Professor | Computer Science | > >>>>>>>> Purdue > >>>>>>>> University > >>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> On Mar 13, 2008, at 12:28 PM, Dragi?a Duri? wrote: > >>>>>>>> > >>>>>>>>> [17:25:19] 117 users, 1 ops and 1 unresolved; > >>>>>>>>> minId = > >>>>>>>>> 18617; thrs = 120 > >>>>>>>>> [17:25:19] started = 663; cleaned = 452; nilled = > >>>>>>>>> 543; > >>>>>>>>> joined = 543 > >>>>>>>>> [17:25:19] Dangling thrs-a-b-n = 1 > >>>>>>>>> [17:25:19] Live queue end is 18621 (head = 18616), > >>>>>>>>> maxClean = 18611, cleaned = 16583 > >>>>>>>>> > >>>>>>>>> About one hour uptime, after I added RTCollector.Collect() > >>>>>>>>> call > >>>>>>>>> every 20 > >>>>>>>>> seconds. > >>>>>>>>> > >>>>>>>>> dd > >>>>>>>>> > >>>>>>>>> On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking wrote: > >>>>>>>>>> Yes. RTCollectorSRC.FinishCollection; > >>>>>>>>>> RTCollectorSRC.StartCollection. This will finish the current > >>>>>>>>>> (possibly minor) collection, and start a major collection. > >>>>>>>>>> The > >>>>>>>>>> major > >>>>>>>>>> collection must finish before you can guarantee garbage has > >>>>>>>>>> been > >>>>>>>>>> freed, so you may want to RTCollectorSRC.FinishCollection. > >>>>>>>>>> This > >>>>>>>>>> sequence is packaged as RTCollector.Collect. > >>>>>>>>>> > >>>>>>>>>> Antony Hosking | Associate Professor | Computer Science | > >>>>>>>>>> Purdue > >>>>>>>>>> University > >>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? wrote: > >>>>>>>>>> > >>>>>>>>>>> Can I force "major collection" ? > >>>>>>>>>>> > >>>>>>>>>>> dd > >>>>>>>>>>> > >>>>>>>>>>> On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: > >>>>>>>>>>>> Indeed! Can you devise a testcase? Note that the > >>>>>>>>>>>> generational > >>>>>>>>>>>> collector will retain old objects for some time until a > >>>>>>>>>>>> major > >>>>>>>>>>>> collection occurs, > >>>>>>>>>>>> > >>>>>>>>>>>> Antony Hosking | Associate Professor | Computer Science | > >>>>>>>>>>>> Purdue > >>>>>>>>>>>> University > >>>>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> And not only that, their stackbase is also set to NIL, > >>>>>>>>>>>>> meaning > >>>>>>>>>>>>> their > >>>>>>>>>>>>> stack is not regarded in any way during future > >>>>>>>>>>>>> collections - > >>>>>>>>>>>>> meaning > >>>>>>>>>>>>> all > >>>>>>>>>>>>> local variables are forgotten once apply method returns. > >>>>>>>>>>>>> > >>>>>>>>>>>>> That way, "spuriousness" is not an issue, once thread > >>>>>>>>>>>>> returns? > >>>>>>>>>>>>> > >>>>>>>>>>>>> dd > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: > >>>>>>>>>>>>>> They should get unlinked from the global ring. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Antony Hosking | Associate Professor | Computer > >>>>>>>>>>>>>> Science | > >>>>>>>>>>>>>> Purdue > >>>>>>>>>>>>>> University > >>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | > >>>>>>>>>>>>>> USA > >>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> What happens to thread stacks once threads are > >>>>>>>>>>>>>>> Join-ed? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> dd > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> This is probably a result of conservatism for > >>>>>>>>>>>>>>>> references > >>>>>>>>>>>>>>>> from > >>>>>>>>>>>>>>>> thread > >>>>>>>>>>>>>>>> stacks which results in spurious retention. It > >>>>>>>>>>>>>>>> may be > >>>>>>>>>>>>>>>> necessary > >>>>>>>>>>>>>>>> to be > >>>>>>>>>>>>>>>> more careful about what references are held on the > >>>>>>>>>>>>>>>> stack > >>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>> threads implementation (and elsewhere in your > >>>>>>>>>>>>>>>> code). > >>>>>>>>>>>>>>>> Until we > >>>>>>>>>>>>>>>> diagnose the place where objects are being > >>>>>>>>>>>>>>>> retained it > >>>>>>>>>>>>>>>> will be > >>>>>>>>>>>>>>>> hard to > >>>>>>>>>>>>>>>> pinpoint. You should realize that weak refs are a > >>>>>>>>>>>>>>>> problem > >>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>> many > >>>>>>>>>>>>>>>> language implementations, not just Modula-3. I > >>>>>>>>>>>>>>>> look > >>>>>>>>>>>>>>>> forward > >>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>> hearing more definitively from you. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Antony Hosking | Associate Professor | Computer > >>>>>>>>>>>>>>>> Science | > >>>>>>>>>>>>>>>> Purdue > >>>>>>>>>>>>>>>> University > >>>>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN > >>>>>>>>>>>>>>>> 47907 | > >>>>>>>>>>>>>>>> USA > >>>>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Problem in my case can be thoroughness of GC... > >>>>>>>>>>>>>>>>> Because > >>>>>>>>>>>>>>>>> it's > >>>>>>>>>>>>>>>>> linked > >>>>>>>>>>>>>>>>> list of WeakRef-ed objects, it's always first > >>>>>>>>>>>>>>>>> element in > >>>>>>>>>>>>>>>>> list to > >>>>>>>>>>>>>>>>> become > >>>>>>>>>>>>>>>>> unreachable first - all others are reachable at > >>>>>>>>>>>>>>>>> least > >>>>>>>>>>>>>>>>> through > >>>>>>>>>>>>>>>>> previous > >>>>>>>>>>>>>>>>> (in list) WeakRef-ed object. So, behaviour I > >>>>>>>>>>>>>>>>> have > >>>>>>>>>>>>>>>>> observed > >>>>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>> because of some thousands of freeable objects, > >>>>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>> one > >>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>> missed > >>>>>>>>>>>>>>>>> too > >>>>>>>>>>>>>>>>> often. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I have other object being cleaned and counted... > >>>>>>>>>>>>>>>>> There > >>>>>>>>>>>>>>>>> also > >>>>>>>>>>>>>>>>> cleanup > >>>>>>>>>>>>>>>>> invocation looks like it's lagging too much. > >>>>>>>>>>>>>>>>> I'll > >>>>>>>>>>>>>>>>> come > >>>>>>>>>>>>>>>>> with > >>>>>>>>>>>>>>>>> numbers > >>>>>>>>>>>>>>>>> after I have more tests. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> dd > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking > >>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>> Sounds like we need some sort of testcase. > >>>>>>>>>>>>>>>>>> Would > >>>>>>>>>>>>>>>>>> you > >>>>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>> able > >>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>> devise one? It will be hard to make it > >>>>>>>>>>>>>>>>>> deterministic, > >>>>>>>>>>>>>>>>>> but > >>>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>>> least we > >>>>>>>>>>>>>>>>>> should see a non-zero cleanup count. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Antony Hosking | Associate Professor | > >>>>>>>>>>>>>>>>>> Computer > >>>>>>>>>>>>>>>>>> Science | > >>>>>>>>>>>>>>>>>> Purdue > >>>>>>>>>>>>>>>>>> University > >>>>>>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN > >>>>>>>>>>>>>>>>>> 47907 | > >>>>>>>>>>>>>>>>>> USA > >>>>>>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 > >>>>>>>>>>>>>>>>>> 5484 > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? > >>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> I have single linked list that I use to send > >>>>>>>>>>>>>>>>>>> messages to > >>>>>>>>>>>>>>>>>>> many > >>>>>>>>>>>>>>>>>>> threads. > >>>>>>>>>>>>>>>>>>> It's end is locked for addition, and current > >>>>>>>>>>>>>>>>>>> end > >>>>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>> given to > >>>>>>>>>>>>>>>>>>> each > >>>>>>>>>>>>>>>>>>> new > >>>>>>>>>>>>>>>>>>> client connecting. This way, all client are > >>>>>>>>>>>>>>>>>>> going > >>>>>>>>>>>>>>>>>>> towards > >>>>>>>>>>>>>>>>>>> end of > >>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>> list so it's head becomes unreferenced and > >>>>>>>>>>>>>>>>>>> goes > >>>>>>>>>>>>>>>>>>> away > >>>>>>>>>>>>>>>>>>> by > >>>>>>>>>>>>>>>>>>> GC. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Except it does not. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> I've added WeakRef cleanup and sequential > >>>>>>>>>>>>>>>>>>> id's > >>>>>>>>>>>>>>>>>>> so I > >>>>>>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>>>> record > >>>>>>>>>>>>>>>>>>> maximum > >>>>>>>>>>>>>>>>>>> freed id for checking. No single cleanup > >>>>>>>>>>>>>>>>>>> happens. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> I'll cross check my code and count my > >>>>>>>>>>>>>>>>>>> references > >>>>>>>>>>>>>>>>>>> from > >>>>>>>>>>>>>>>>>>> application > >>>>>>>>>>>>>>>>>>> side, > >>>>>>>>>>>>>>>>>>> but maybe someone else has similar problems? > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> dd > >>>>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>>>> Dragi?a Duri? > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>> Dragi?a Duri? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>> Dragi?a Duri? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> -- > >>>>>>>>>>>>> Dragi?a Duri? > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> Dragi?a Duri? > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> Dragi?a Duri? > >>>>>>>>> > >>>>>>>> > >>>>>>> -- > >>>>>>> Dragi?a Duri? > >>>>>>> > >>>>>> > >>>>> -- > >>>>> Dragi?a Duri? > >>>> > >>> -- > >>> Dragi?a Duri? > >> > > -- > > Dragi?a Duri? > -- Dragi?a Duri? From hosking at cs.purdue.edu Fri Mar 14 02:16:16 2008 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 13 Mar 2008 21:16:16 -0400 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <1205449510.32152.171.camel@hias.m3w.org> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> <1205425734.32152.144.camel@hias.m3w.org> <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> <1205430199.32152.152.camel@hias.m3w.org> <1205444774.32152.158.camel@hias.m3w.org> <5649BA54-5B2F-423C-AEF7-65FF09CF3763@cs.purdue.edu> <1205446178.32152.162.camel@hias.m3w.org> <4DD89DC7-FA3C-4B61-A2A1-5C8488CF3A3B@cs.purdue.edu> <1205446902.32152.168.camel@hias.m3w.org> <2EA2C202-F053-4AE8-BA60-E51A1A339C6B@cs.purdue.edu> <1205449510.32152.171.camel@hi! as.m3w.org> Message-ID: <3F319520-7A7E-4DE3-A467-3712EBB8AA8F@cs.purdue.edu> http://www.cs.purdue.edu/homes/hosking/papers/ismm06.pdf On Mar 13, 2008, at 7:05 PM, Dragi?a Duri? wrote: > Is your work on GC covered somewhere else except source, comments and > this list? > > dd > > On Thu, 2008-03-13 at 18:35 -0400, Tony Hosking wrote: >> Ah, the fact that you needed to be aggressive suggests that >> conservatism may be the culprit. I keep meaning one day to make sure >> that we do not can objects on pinned pages unless we know that object >> is definitely alive. >> >> On Mar 13, 2008, at 6:21 PM, Dragi?a Duri? wrote: >> >>> I see... What bugs me now is a fact about my flow... I am already >>> doing >>> this, and there is always few thousand objects uncleaned from of >>> type >>> Message... I'll follow this through, to crosscheck everything again. >>> >>> What is nice, after I made this agressive RTCollector.Collect >>> "policy", >>> my memory footprint remains same for many hours... It was swelling >>> before. >>> >>> Thanks, >>> dd >>> >>> On Thu, 2008-03-13 at 18:15 -0400, Tony Hosking wrote: >>>> No, when a page is pinned because one of its objects is directly >>>> referenced from a thread stack then all objects on that page, both >>>> live and dead, are retained. The collector does not move objects >>>> to >>>> or from pinned pages. The objects on the page can only be freed >>>> when >>>> the page is no longer pinned by a direct reference from any thread >>>> stack. Moral of the story is to NIL references aggressively in >>>> local >>>> variables. >>>> >>>> On Mar 13, 2008, at 6:09 PM, Dragi?a Duri? wrote: >>>> >>>>> When object becomes unreachable, and it sits on page with other >>>>> reachable objects, it's pinned? >>>>> >>>>> What I asked is... As garbage collector moves objects around, >>>>> would >>>>> it >>>>> pack reachable objects over unreachable ones in these pages? Thus >>>>> kicking them out of pages where they're pinned? >>>>> >>>>> dd >>>>> >>>>> On Thu, 2008-03-13 at 18:03 -0400, Tony Hosking wrote: >>>>>> I'm not sure I understand what you mean? >>>>>> >>>>>> On Mar 13, 2008, at 5:46 PM, Dragi?a Duri? wrote: >>>>>> >>>>>>> Are objects on pages prepacked/moved? >>>>>>> >>>>>>> On Thu, 2008-03-13 at 17:41 -0400, Tony Hosking wrote: >>>>>>>> The thing about conservatism is that even if you NIL *all* >>>>>>>> references >>>>>>>> to an object that just happens to lie on a pinned page, that >>>>>>>> object >>>>>>>> (and objects reachable from it) will be retained until the page >>>>>>>> is no >>>>>>>> longer pinned. >>>>>>>> >>>>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>>>> Purdue >>>>>>>> University >>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mar 13, 2008, at 1:43 PM, Dragi?a Duri? wrote: >>>>>>>> >>>>>>>>> I understand transitiveness :), but I've made big effort to >>>>>>>>> escape >>>>>>>>> that. >>>>>>>>> >>>>>>>>> Lifetime of thread equals lifetime of Client.T instance, and >>>>>>>>> that >>>>>>>>> instance is kept in thread's closure, and also in local >>>>>>>>> variable. >>>>>>>>> Both >>>>>>>>> are set to NIL before thread termination - all terminations >>>>>>>>> and >>>>>>>>> joins >>>>>>>>> happen and are counted and shown. >>>>>>>>> >>>>>>>>> Pointers to Message objects are kept in two places. Hub.T >>>>>>>>> keeps >>>>>>>>> pointer >>>>>>>>> to list tail, and each Client.T contains pointer to it's >>>>>>>>> current >>>>>>>>> element. That pointer is forwarded as Message's are added to >>>>>>>>> end. No >>>>>>>>> temporary variable, not even WITH substitution is used. And >>>>>>>>> these >>>>>>>>> threads are ones dying with their Client.T's. >>>>>>>>> >>>>>>>>> "head" is new mechanism used to unlink all elements with id >>>>>>>>> less >>>>>>>>> than >>>>>>>>> minimal id used in active Client.T instances. Unlinked totally >>>>>>>>> and >>>>>>>>> all >>>>>>>>> as one free for garbage collector to finish them. >>>>>>>>> >>>>>>>>> I see no transition. >>>>>>>>> >>>>>>>>> dd >>>>>>>>> >>>>>>>>> On Thu, 2008-03-13 at 12:45 -0400, Tony Hosking wrote: >>>>>>>>>> Realizing that conservative roots collectors like ours can >>>>>>>>>> retain >>>>>>>>>> data >>>>>>>>>> unnecessarily. If a thread stack refers to a page that >>>>>>>>>> holds a >>>>>>>>>> reference that (transitively) refers to some otherwise dead >>>>>>>>>> object >>>>>>>>>> then that object will be retained. We say that the page is >>>>>>>>>> pinned. >>>>>>>>>> Smartening up the collector to ignore dead objects in pinned >>>>>>>>>> pages >>>>>>>>>> is >>>>>>>>>> possible, but not currently implemented. >>>>>>>>>> >>>>>>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>>>>>> Purdue >>>>>>>>>> University >>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mar 13, 2008, at 12:28 PM, Dragi?a Duri? wrote: >>>>>>>>>> >>>>>>>>>>> [17:25:19] 117 users, 1 ops and 1 unresolved; >>>>>>>>>>> minId = >>>>>>>>>>> 18617; thrs = 120 >>>>>>>>>>> [17:25:19] started = 663; cleaned = 452; nilled = >>>>>>>>>>> 543; >>>>>>>>>>> joined = 543 >>>>>>>>>>> [17:25:19] Dangling thrs-a-b-n = 1 >>>>>>>>>>> [17:25:19] Live queue end is 18621 (head = 18616), >>>>>>>>>>> maxClean = 18611, cleaned = 16583 >>>>>>>>>>> >>>>>>>>>>> About one hour uptime, after I added RTCollector.Collect() >>>>>>>>>>> call >>>>>>>>>>> every 20 >>>>>>>>>>> seconds. >>>>>>>>>>> >>>>>>>>>>> dd >>>>>>>>>>> >>>>>>>>>>> On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking wrote: >>>>>>>>>>>> Yes. RTCollectorSRC.FinishCollection; >>>>>>>>>>>> RTCollectorSRC.StartCollection. This will finish the >>>>>>>>>>>> current >>>>>>>>>>>> (possibly minor) collection, and start a major collection. >>>>>>>>>>>> The >>>>>>>>>>>> major >>>>>>>>>>>> collection must finish before you can guarantee garbage has >>>>>>>>>>>> been >>>>>>>>>>>> freed, so you may want to RTCollectorSRC.FinishCollection. >>>>>>>>>>>> This >>>>>>>>>>>> sequence is packaged as RTCollector.Collect. >>>>>>>>>>>> >>>>>>>>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>>>>>>>> Purdue >>>>>>>>>>>> University >>>>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mar 13, 2008, at 10:22 AM, Dragi?a Duri? wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Can I force "major collection" ? >>>>>>>>>>>>> >>>>>>>>>>>>> dd >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote: >>>>>>>>>>>>>> Indeed! Can you devise a testcase? Note that the >>>>>>>>>>>>>> generational >>>>>>>>>>>>>> collector will retain old objects for some time until a >>>>>>>>>>>>>> major >>>>>>>>>>>>>> collection occurs, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Antony Hosking | Associate Professor | Computer Science | >>>>>>>>>>>>>> Purdue >>>>>>>>>>>>>> University >>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | >>>>>>>>>>>>>> USA >>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mar 13, 2008, at 9:56 AM, Dragi?a Duri? wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> And not only that, their stackbase is also set to NIL, >>>>>>>>>>>>>>> meaning >>>>>>>>>>>>>>> their >>>>>>>>>>>>>>> stack is not regarded in any way during future >>>>>>>>>>>>>>> collections - >>>>>>>>>>>>>>> meaning >>>>>>>>>>>>>>> all >>>>>>>>>>>>>>> local variables are forgotten once apply method returns. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> That way, "spuriousness" is not an issue, once thread >>>>>>>>>>>>>>> returns? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> dd >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote: >>>>>>>>>>>>>>>> They should get unlinked from the global ring. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Antony Hosking | Associate Professor | Computer >>>>>>>>>>>>>>>> Science | >>>>>>>>>>>>>>>> Purdue >>>>>>>>>>>>>>>> University >>>>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | >>>>>>>>>>>>>>>> USA >>>>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mar 13, 2008, at 6:16 AM, Dragi?a Duri? wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> What happens to thread stacks once threads are >>>>>>>>>>>>>>>>> Join-ed? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> dd >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> This is probably a result of conservatism for >>>>>>>>>>>>>>>>>> references >>>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>>> thread >>>>>>>>>>>>>>>>>> stacks which results in spurious retention. It >>>>>>>>>>>>>>>>>> may be >>>>>>>>>>>>>>>>>> necessary >>>>>>>>>>>>>>>>>> to be >>>>>>>>>>>>>>>>>> more careful about what references are held on the >>>>>>>>>>>>>>>>>> stack >>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> threads implementation (and elsewhere in your >>>>>>>>>>>>>>>>>> code). >>>>>>>>>>>>>>>>>> Until we >>>>>>>>>>>>>>>>>> diagnose the place where objects are being >>>>>>>>>>>>>>>>>> retained it >>>>>>>>>>>>>>>>>> will be >>>>>>>>>>>>>>>>>> hard to >>>>>>>>>>>>>>>>>> pinpoint. You should realize that weak refs are a >>>>>>>>>>>>>>>>>> problem >>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>> many >>>>>>>>>>>>>>>>>> language implementations, not just Modula-3. I >>>>>>>>>>>>>>>>>> look >>>>>>>>>>>>>>>>>> forward >>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>> hearing more definitively from you. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Antony Hosking | Associate Professor | Computer >>>>>>>>>>>>>>>>>> Science | >>>>>>>>>>>>>>>>>> Purdue >>>>>>>>>>>>>>>>>> University >>>>>>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN >>>>>>>>>>>>>>>>>> 47907 | >>>>>>>>>>>>>>>>>> USA >>>>>>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Mar 12, 2008, at 4:21 AM, Dragi?a Duri? wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Problem in my case can be thoroughness of GC... >>>>>>>>>>>>>>>>>>> Because >>>>>>>>>>>>>>>>>>> it's >>>>>>>>>>>>>>>>>>> linked >>>>>>>>>>>>>>>>>>> list of WeakRef-ed objects, it's always first >>>>>>>>>>>>>>>>>>> element in >>>>>>>>>>>>>>>>>>> list to >>>>>>>>>>>>>>>>>>> become >>>>>>>>>>>>>>>>>>> unreachable first - all others are reachable at >>>>>>>>>>>>>>>>>>> least >>>>>>>>>>>>>>>>>>> through >>>>>>>>>>>>>>>>>>> previous >>>>>>>>>>>>>>>>>>> (in list) WeakRef-ed object. So, behaviour I >>>>>>>>>>>>>>>>>>> have >>>>>>>>>>>>>>>>>>> observed >>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>> because of some thousands of freeable objects, >>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>> one >>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>> missed >>>>>>>>>>>>>>>>>>> too >>>>>>>>>>>>>>>>>>> often. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I have other object being cleaned and counted... >>>>>>>>>>>>>>>>>>> There >>>>>>>>>>>>>>>>>>> also >>>>>>>>>>>>>>>>>>> cleanup >>>>>>>>>>>>>>>>>>> invocation looks like it's lagging too much. >>>>>>>>>>>>>>>>>>> I'll >>>>>>>>>>>>>>>>>>> come >>>>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>>>> numbers >>>>>>>>>>>>>>>>>>> after I have more tests. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> dd >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>> Sounds like we need some sort of testcase. >>>>>>>>>>>>>>>>>>>> Would >>>>>>>>>>>>>>>>>>>> you >>>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>> able >>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>> devise one? It will be hard to make it >>>>>>>>>>>>>>>>>>>> deterministic, >>>>>>>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>> least we >>>>>>>>>>>>>>>>>>>> should see a non-zero cleanup count. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Antony Hosking | Associate Professor | >>>>>>>>>>>>>>>>>>>> Computer >>>>>>>>>>>>>>>>>>>> Science | >>>>>>>>>>>>>>>>>>>> Purdue >>>>>>>>>>>>>>>>>>>> University >>>>>>>>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN >>>>>>>>>>>>>>>>>>>> 47907 | >>>>>>>>>>>>>>>>>>>> USA >>>>>>>>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 >>>>>>>>>>>>>>>>>>>> 5484 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Mar 11, 2008, at 4:03 AM, Dragi?a Duri? >>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I have single linked list that I use to send >>>>>>>>>>>>>>>>>>>>> messages to >>>>>>>>>>>>>>>>>>>>> many >>>>>>>>>>>>>>>>>>>>> threads. >>>>>>>>>>>>>>>>>>>>> It's end is locked for addition, and current >>>>>>>>>>>>>>>>>>>>> end >>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>> given to >>>>>>>>>>>>>>>>>>>>> each >>>>>>>>>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>>>>>> client connecting. This way, all client are >>>>>>>>>>>>>>>>>>>>> going >>>>>>>>>>>>>>>>>>>>> towards >>>>>>>>>>>>>>>>>>>>> end of >>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> list so it's head becomes unreferenced and >>>>>>>>>>>>>>>>>>>>> goes >>>>>>>>>>>>>>>>>>>>> away >>>>>>>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>>>> GC. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Except it does not. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I've added WeakRef cleanup and sequential >>>>>>>>>>>>>>>>>>>>> id's >>>>>>>>>>>>>>>>>>>>> so I >>>>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>>> record >>>>>>>>>>>>>>>>>>>>> maximum >>>>>>>>>>>>>>>>>>>>> freed id for checking. No single cleanup >>>>>>>>>>>>>>>>>>>>> happens. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I'll cross check my code and count my >>>>>>>>>>>>>>>>>>>>> references >>>>>>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>>>>>> application >>>>>>>>>>>>>>>>>>>>> side, >>>>>>>>>>>>>>>>>>>>> but maybe someone else has similar problems? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> dd >>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Dragi?a Duri? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Dragi?a Duri? >>>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> Dragi?a Duri? >>>>>> >>>>> -- >>>>> Dragi?a Duri? >>>> >>> -- >>> Dragi?a Duri? >> > -- > Dragi?a Duri? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Fri Mar 14 07:45:57 2008 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Fri, 14 Mar 2008 07:45:57 +0100 Subject: [M3devel] WeakRef mechanism alive and well? cm3-cvshead In-Reply-To: <3F319520-7A7E-4DE3-A467-3712EBB8AA8F@cs.purdue.edu> References: <1205222589.32152.27.camel@hias.m3w.org> <1205310098.32152.43.camel@hias.m3w.org> <54251EAB-69ED-4C78-A8F5-EB8E2F1F7C17@cs.purdue.edu> <1205403396.32152.96.camel@hias.m3w.org> <7BF90305-6004-4561-B628-820668C2C8E9@cs.purdue.edu> <1205416588.32152.127.camel@hias.m3w.org> <1205418137.32152.139.camel@hias.m3w.org> <750B3EA0-9187-4576-A898-8F7424DF57C6@cs.purdue.edu> <1205425734.32152.144.camel@hias.m3w.org> <3A278DBC-4D08-4F3E-BE53-F0532A61A8C4@cs.purdue.edu> <1205430199.32152.152.camel@hias.m3w.org> <1205444774.32152.158.camel@hias.m3w.org> <5649BA54-5B2F-423C-AEF7-65FF09CF3763@cs.purdue.edu> <1205446178.32152.162.camel@hias.m3w.org> <4DD89DC7-FA3C-4B61-A2A1-5C8488CF3A3B@cs.purdue.edu> <1205446902.32152.168.camel@hias.m3w.org> <2EA2C202-F053-4AE8-BA60-E51A1A339C6B@cs.purdue.edu> <1205449510.32152.171.camel@hi! as.m3w.org> <3F319520-7A7E-4DE3-A467-3712EBB8AA8F@cs.purdue.edu> Message-ID: <1205477157.32152.189.camel@hias.m3w.org> [07:29:28] 0 users, 1 ops and 0 unresolved; minId = 44906; thrs = 1 [07:29:28] started = 2201; cleaned = 2199; nilled = 2200; joined = 2200 [07:29:28] Dangling thrs-a-b-n = 0 [07:29:28] Live queue end is 44909 (head = 44906), maxClean = 44897, cleaned = 44897 After all keep-pointers-off-stack, one thing that remained was long living closures... And I killed them all and what happened was total confirmation of your explanation... One Client.T that remained is most probably pinned on page with listener thread's closure object. My understanding of conservativeness of GC with regard to register and stack content was something else... I was thinking it's like this: ProcessStacks will process registers and stacks; each time address-sized, correctly-aligned bit pattern which looks like address of some object GC knows occurs, GC accepts it as it's address - it maybe is not, but conservative thinking will not make risks there. This way, ProcessStacks will know exactly what objects are (maybe, but no risks...) referenced from stack. I know I am missing something here, and that's why I ask, why don't you do it just like this? dd On Thu, 2008-03-13 at 21:16 -0400, Tony Hosking wrote: > http://www.cs.purdue.edu/homes/hosking/papers/ismm06.pdf > > > On Mar 13, 2008, at 7:05 PM, Dragi?a Duri? wrote: > > > Is your work on GC covered somewhere else except source, comments > > and > > this list? > > > > dd > > > > On Thu, 2008-03-13 at 18:35 -0400, Tony Hosking wrote: > > > Ah, the fact that you needed to be aggressive suggests that > > > conservatism may be the culprit. I keep meaning one day to make > > > sure > > > that we do not can objects on pinned pages unless we know that > > > object > > > is definitely alive. > > > > > > On Mar 13, 2008, at 6:21 PM, Dragi?a Duri? wrote: > > > > > > > I see... What bugs me now is a fact about my flow... I am > > > > already > > > > doing > > > > this, and there is always few thousand objects uncleaned from of > > > > type > > > > Message... I'll follow this through, to crosscheck everything > > > > again. > > > > > > > > What is nice, after I made this agressive RTCollector.Collect > > > > "policy", > > > > my memory footprint remains same for many hours... It was > > > > swelling > > > > before. > > > > > > > > Thanks, > > > > dd > > > > > > > > On Thu, 2008-03-13 at 18:15 -0400, Tony Hosking wrote: > > > > > No, when a page is pinned because one of its objects is > > > > > directly > > > > > referenced from a thread stack then all objects on that page, > > > > > both > > > > > live and dead, are retained. The collector does not move > > > > > objects to > > > > > or from pinned pages. The objects on the page can only be > > > > > freed when > > > > > the page is no longer pinned by a direct reference from any > > > > > thread > > > > > stack. Moral of the story is to NIL references aggressively > > > > > in local > > > > > variables. > > > > > > > > > > On Mar 13, 2008, at 6:09 PM, Dragi?a Duri? wrote: > > > > > > > > > > > When object becomes unreachable, and it sits on page with > > > > > > other > > > > > > reachable objects, it's pinned? > > > > > > > > > > > > What I asked is... As garbage collector moves objects > > > > > > around, would > > > > > > it > > > > > > pack reachable objects over unreachable ones in these pages? > > > > > > Thus > > > > > > kicking them out of pages where they're pinned? > > > > > > > > > > > > dd > > > > > > > > > > > > On Thu, 2008-03-13 at 18:03 -0400, Tony Hosking wrote: > > > > > > > I'm not sure I understand what you mean? > > > > > > > > > > > > > > On Mar 13, 2008, at 5:46 PM, Dragi?a Duri? wrote: > > > > > > > > > > > > > > > Are objects on pages prepacked/moved? > > > > > > > > > > > > > > > > On Thu, 2008-03-13 at 17:41 -0400, Tony Hosking wrote: > > > > > > > > > The thing about conservatism is that even if you NIL > > > > > > > > > *all* > > > > > > > > > references > > > > > > > > > to an object that just happens to lie on a pinned > > > > > > > > > page, that > > > > > > > > > object > > > > > > > > > (and objects reachable from it) will be retained until > > > > > > > > > the page > > > > > > > > > is no > > > > > > > > > longer pinned. > > > > > > > > > > > > > > > > > > Antony Hosking | Associate Professor | Computer > > > > > > > > > Science | Purdue > > > > > > > > > University > > > > > > > > > 305 N. University Street | West Lafayette | IN 47907 | > > > > > > > > > USA > > > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 1:43 PM, Dragi?a Duri? wrote: > > > > > > > > > > > > > > > > > > > I understand transitiveness :), but I've made big > > > > > > > > > > effort to > > > > > > > > > > escape > > > > > > > > > > that. > > > > > > > > > > > > > > > > > > > > Lifetime of thread equals lifetime of Client.T > > > > > > > > > > instance, and > > > > > > > > > > that > > > > > > > > > > instance is kept in thread's closure, and also in > > > > > > > > > > local > > > > > > > > > > variable. > > > > > > > > > > Both > > > > > > > > > > are set to NIL before thread termination - all > > > > > > > > > > terminations and > > > > > > > > > > joins > > > > > > > > > > happen and are counted and shown. > > > > > > > > > > > > > > > > > > > > Pointers to Message objects are kept in two places. > > > > > > > > > > Hub.T keeps > > > > > > > > > > pointer > > > > > > > > > > to list tail, and each Client.T contains pointer to > > > > > > > > > > it's current > > > > > > > > > > element. That pointer is forwarded as Message's are > > > > > > > > > > added to > > > > > > > > > > end. No > > > > > > > > > > temporary variable, not even WITH substitution is > > > > > > > > > > used. And > > > > > > > > > > these > > > > > > > > > > threads are ones dying with their Client.T's. > > > > > > > > > > > > > > > > > > > > "head" is new mechanism used to unlink all elements > > > > > > > > > > with id less > > > > > > > > > > than > > > > > > > > > > minimal id used in active Client.T instances. > > > > > > > > > > Unlinked totally > > > > > > > > > > and > > > > > > > > > > all > > > > > > > > > > as one free for garbage collector to finish them. > > > > > > > > > > > > > > > > > > > > I see no transition. > > > > > > > > > > > > > > > > > > > > dd > > > > > > > > > > > > > > > > > > > > On Thu, 2008-03-13 at 12:45 -0400, Tony Hosking > > > > > > > > > > wrote: > > > > > > > > > > > Realizing that conservative roots collectors like > > > > > > > > > > > ours can > > > > > > > > > > > retain > > > > > > > > > > > data > > > > > > > > > > > unnecessarily. If a thread stack refers to a page > > > > > > > > > > > that holds a > > > > > > > > > > > reference that (transitively) refers to some > > > > > > > > > > > otherwise dead > > > > > > > > > > > object > > > > > > > > > > > then that object will be retained. We say that > > > > > > > > > > > the page is > > > > > > > > > > > pinned. > > > > > > > > > > > Smartening up the collector to ignore dead objects > > > > > > > > > > > in pinned > > > > > > > > > > > pages > > > > > > > > > > > is > > > > > > > > > > > possible, but not currently implemented. > > > > > > > > > > > > > > > > > > > > > > Antony Hosking | Associate Professor | Computer > > > > > > > > > > > Science | > > > > > > > > > > > Purdue > > > > > > > > > > > University > > > > > > > > > > > 305 N. University Street | West Lafayette | IN > > > > > > > > > > > 47907 | USA > > > > > > > > > > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 13, 2008, at 12:28 PM, Dragi?a Duri? wrote: > > > > > > > > > > >