[M3devel] possible cygwin createprocess/fork cost measurements..

[Jay]

Olaf,
No, not every process create, creates a shell. AND ESP. NOT a sh/bash ONE! :)
Merely replacing sh with cmd nets some perf win.
The remaining perf cost is apparently the vfork/exec vs. spawn.
I'm working on it.. (actually, not at the moment, but whenever I am home and not at work. :) )
 
My numbers don't all make sense to me so far or at the moment -- oh, maybe one set of perf tests launches more copies of sh than a real build. Anyway, I'm also measuring real builds. I was just really trying to do the "cheaper" up front prototyping/measuring before more real implementing/measuring, though of course I still get caught in refining the throw away work. :)
 
The branch advise helps, but as well, I mean "directory structure"-wise, experimental, possibly throw away work, possibly interesting to keep around for future measuring, but not in any default build. Perhaps that is what you meant -- perhaps a private branch is ok for "personal history" even if it ends up being deleted and never going to another branch? Kind of an abuse, but useful, and maybe affordable? You know, rather than me saving away backups every hour in case I want to go back to an older version of what will still actually be thrown away? Or I could just use my private SVN. Yep. I did that. I'm wondering about making it world readable or somehow sharing it out. It's not chock full of stuff, but, of course, I get to decide what I am allowed to checkin. :)
 
The general dilemna I think I see is that people that "do this sort of stuff" initially develop their command lines in a fancy interactive shell such as bash or cmd. With such features as pipes, redirection, environment variables, or even, gasp, wildcard expansion.
They then move their command lines into a somewhat programmatic environment and expect all those features to keep working.
And thus, various systems that "run command lines" end up wrapping them in shells to retain those fancy features.
It's a bit unfortunate.
 
Though in "native" environments the cost isn't so high. Wrapping in cmd isn't nearly as expensive in native Windows as wrapping in sh on Cygwin, nor I expect is wrapping in sh anywhere else. Though you all should read the Ubuntu Wiki page about changing /bin/sh from bash to ash. I don't think they have numbers, but they did this strictly for perf. While you may need fancy piping, redirection, etc. from /bin/sh, you don't for example need fancy command line editing/history/completion.
 
To a large extent, uses of exec in Quake can't/shouldn't depend on much, since there is some hypothetical portability to cmd and sh, however the Quake code tends to be in target-specific files, which I think ends up being host/native-build specific, as well cmd and sh have an actually useful compatible subset -- pipes, redirection, combining commands with && and ||, so people can rely on having a shell, either one.
 
Anyway, I think I had a plan in mind at least enough for now.
There shall be three implementations of Process. Posix, Cygwin, and Win32. Only Cygwin changes from current.
The Cygwin one will compare workingdirectory and stdin/out/err to the globals, and if they are equal, use spawn instead of fork/exec. Process only adds sh if it gets back a noexec error.
 
The other part, the sh wrapper that Quake always adds, well, for now I set QUAKE_SHELL or whatever  in my environment to cmd. That gets me most of the way there and then measurable.
 
Not sure how to give that benefit to everyone always, probably with the idea of implementing m3_backend and assemble in Modula-3, and then defering to the Quake code if it is exists, and then making sure the Modula-3 version is very viable at least or only for Cygwin so it'll tend not to be replaced.
Really, these functions are just a few lines.
 
Another idea is to add a special prefix character to exec like an explanation point (for "fast" or "direct").
I don't like this use of special characters as flags though. It's the wrong way to design a function interface.
Everything should be named constants or named boolean parameters.
Every special character taken is a character that can't be used as the first character of a file name, as well.
 
The heuristic to check the command line for "special characters" and deem them worthy of a sh wrapper isn't terrible either.
This could be limited to Cygwin as well, like there could be Process.IsShSlow() (yeah yeah, bad choice of names, it doesn't belong in a public place Process, it would be Quake) that is false on all other platforms and gates the heuristic.
 
A batch mode in cm3cg where it runs over an entire directory is also a good idea, along with a switch to it to have it exec as directly. That would also work as well. It MIGHT speed up other systems too, but I'd have to measure it, and I doubt it'd be much. I suspect the process create cost is ONLY signficant on Cygwin.
 
It really bugs me 1) that there aren't more .dlls/.libs for use in-proc instead of shelling out to command lines 2) the related licensing unclarity. cm3cg and as really should not cost a process, the compiler should output .objs directly.
 
On Windows, Microsoft nmake has a "double colon batch feature", where instead of saying:
 
.c.o:
  cl -c $<
 
You can say:
.c.o::
  cl -c $<
 
and what this does is instead of running  cl -c foo.c 
  cl -c bar.c 
 
it runs:
   cl -c foo.c bar.c 
 
It can REALLY speed up builds.
Modula-3 already has this behavior for the front end, just not the backend.
 
I have no numbers.
 
 - Jay



> Date: Thu, 20 Mar 2008 09:09:48 +0100> From: wagner at elegosoft.com> To: m3devel at elegosoft.com> Subject: Re: [M3devel] possible cygwin createprocess/fork cost measurements..> > Quoting Jay <jayk123 at hotmail.com>:> > > spawn(nowait) + waitpid is about the same as spawn(wait), and is > > close to system() in inflexibility, but system() has an implicit > > shell wrapper and so is slow.> > To eliminate the shell wrapper you could use System.ExecuteList> from package sysutils in M3 and> > q_exec( cmd ) --> int (since CM3 d5.6.0)> > in quake (see quake manual on WWW), or does any Process.Create> on Windows invoke a shell on NT386GNU? I wouldn't think so> (but haven't checked).> > > It seems that spawn is faster than CreateProcess at running .exes > > that use cygwin1.dll. That is a little odd. It must be something to > > do with the child startup code talking to its parent or something > > and doing so more efficiently when the immediate parent uses Cygwin. > > I might dig into that. Like, a dynamic choice between CreateProcess > > and spawn could be more efficient, but so far I'm not sure even > > using CreateProcess here is "easy enough".> >> > Is there anywhere in the cm3 tree appropriate to checkin such > > experimental code?> > I can put it in my private svn...> > Yes, just create a branch:> > cvs tag -b devel_process_perf_opt m3core ...> cvs up -r devel_process_perf_opt m3core ...> > and you are then working on the branch, until you issue> > cvs up -A m3core ...> > Hope this helps,> > Olaf> -- > Olaf Wagner -- elego Software Solutions GmbH> Gustav-Meyer-Allee 25 / Gebäude 12, 13355 Berlin, Germany> phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95> http://www.elegosoft.com | Geschäftsführer: Olaf Wagner | Sitz: Berlin> Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://m3lists.elegosoft.com/pipermail/m3devel/attachments/20080320/420d39fb/attachment-0002.html>