Re: parsing a file for analysis
On Sat, 26 Feb 2011 16:29:54 +0100, Andrea Crotti wrote:
> Il giorno 26/feb/2011, alle ore 06.45, Rita ha scritto:
>
>> I have a large text (4GB) which I am parsing.
>>
>> I am reading the file to collect stats on certain items.
>>
>> My approach has been simple,
>>
>> for row in open(file):
>> if "INFO" in row:
>> line=row.split()
>> user=line[0]
>> host=line[1]
>> __time=line[2]
>> ...
>>
>> I was wondering if there is a framework or a better algorithm to read
>> such as large file and collect it stats according to content. Also, are
>> there any libraries, data structures or functions which can be helpful?
>> I was told about 'collections' container. Here are some stats I am
>> trying to get:
>>
>> *Number of unique users
>> *Break down each user's visit according to time, t0 to t1 *what user
>> came from what host.
>> *what time had the most users?
>>
>> (There are about 15 different things I want to query)
>>
>> I understand most of these are redundant but it would be nice to have a
>> framework or even a object oriented way of doing this instead of
>> loading it into a database.
>>
>>
>> Any thoughts or ideas?
>
> Not an expert, but maybe it might be good to push the data into a
> database, and then you can tweak the DBMS and write smart queries to get
> all the statistics you want from it.
>
> It might take a while (maybe with regexp splitting is faster) but it's
> done only once and then you work with DB tools.
>
This is the sort of job that is best done with awk.
Awk processes a text file line by line, automatically splitting each line
into an array of words. It uses regexes to recognise lines and trigger
actions on them. For example, building a list of visitors: assume there's
a line containing "username logged on", you could build a list of users
and count their visits with this statement:
/logged on/ { user[$1] += 1 }
where the regex, /logged on/, triggers the action, in curly brackets, for
each line it matches. "$1" is a symbol for the first word in the line.
--
martin@ | Martin Gregorie
gregorie. | Essex, UK
org |
--
http://mail.python.org/mailman/listinfo/python-list
Re: Purely historic question: VT200 text graphic programming
On Thu, 10 Mar 2011 20:31:11 +, Grant Edwards wrote: > You tricked me by saying only DEC VAX/VMS programmers would know what it > was. In fact, many, many Unix programmers knew about curses (and still > do) and very few VMS programmers ever did. C wasn't very widely used > under VMS, and VMS had it's own screen formatting and form handling > libraries. > >From the context the "only DEC VAX/VMS programmers" remark applied to the VT-100. However, the OP is wrong about that - VT-100s were well-known and popular devices in the 8-bit microprocessor world too, together with assorted clones. In addition, many other terminals had a VT-100 emulation mode. IIRC all the Wyse terminals had that. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Purely historic question: VT200 text graphic programming
On Fri, 11 Mar 2011 04:39:02 -0600, GrayShark wrote: >> I think the order went the other way -- I think most of the ANSI >> sequences were inherited from the VT52/VT100 terminals. > > Are you implying ascii came after the VT52/VT110 terminals? VT52 is a > ascii code based piece of shit, including the backspace/return character > set (which Windows still honors like it's a deity). > He said ANSI, not ASCII. Any, yes I think VT100 preceeded ANSI escape codes too. ANSI codes are very similar to VT100 escape codes. I don't remember seeing ANSI codes before the MS-DOS ANSI driver was introduced but I'd used VT100s long before that on assorted minis and 8-bit micros. > Working from the old days, the ascii set used 0->127 (zero -> seventh > bit) to represent special keys and sounds, letters, numbers and common > punctuations. The eight bit was reserved for error correct. Remember > this was a serial 'printer' console language. It need updating as much > as the QWERTY keyboard does. > > Sometime after 1975, graphical glyphs were added, using the eighth bit. > Some VT models supported the extended ascii, as did IBM-DOS. > > Strange the history one remembers. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Purely historic question: VT200 text graphic programming
On Fri, 11 Mar 2011 11:52:13 +, Jorgen Grahn wrote: > On Thu, 2011-03-10, Martin Gregorie wrote: >> On Thu, 10 Mar 2011 20:31:11 +, Grant Edwards wrote: >> >>> You tricked me by saying only DEC VAX/VMS programmers would know what >>> it was. In fact, many, many Unix programmers knew about curses (and >>> still do) and very few VMS programmers ever did. C wasn't very widely >>> used under VMS, and VMS had it's own screen formatting and form >>> handling libraries. >>> >> From the context the "only DEC VAX/VMS programmers" remark applied to >> the VT-100. However, the OP is wrong about that - VT-100s were >> well-known and popular devices in the 8-bit microprocessor world too, >> together with assorted clones. In addition, many other terminals had a >> VT-100 emulation mode. IIRC all the Wyse terminals had that. > > But he wrote VT-200, not VT-100. I assumed he meant those (vt200) had > some exotic graphics mode. The VT-xxx series was pretty heterogenous, > although most of us think of them as more or less fancy VT-100s. > You're right - he did say VT-200. Can't remember using one. However, I did buy a used VT-103 at some point and dumped it fairly rapidly as it had no manuals and I couldn't get it to work as a terminal (no wonder - I've since found out that it was really a standalone box with an LSI-11/23 crammed into the VT-100 box). It got swapped for a Wyse 120 - an excellent terminal with a white phosphor rather than green. Grayshark was right too: the ANSI control code standard preceded the VT-100 - I live and learn - and because of that there was no commonality between VT-50/52 and VT-100 escape codes. Details here: http://vt100.net/vt_history BTW, there was no such thing as a VT-200 - there was a VT-220 text terminal (which I think the OP was remembering) and the VT-240 and 241 terminals, which were totally different graphics terminals that accepted Tektronics graphics commands: comparing a VT-220 to a VT-240/241 would be like comparing an Epson dot-matric printer to an HP 7485 plotter! -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Purely historic question: VT200 text graphic programming
On Fri, 11 Mar 2011 15:53:20 +, Grant Edwards wrote: >> I think the order went the other way -- I think most of the ANSI >> sequences were inherited from the VT52/VT100 terminals. > > Ah. I didn't mean to imply order, only that the vt200 used ANSI escape > sequences with some extensions. > I got that wrong: ANSI defined the VT-100 control codes. See above. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Purely historic question: VT200 text graphic programming
On Fri, 11 Mar 2011 19:32:53 +, Grant Edwards wrote: > On 2011-03-11, Martin Gregorie wrote: > >> BTW, there was no such thing as a VT-200 - there was a VT-220 text >> terminal (which I think the OP was remembering) and the VT-240 and 241 >> terminals, which were totally different graphics terminals that >> accepted Tektronics graphics commands: comparing a VT-220 to a >> VT-240/241 would be like comparing an Epson dot-matric printer to an HP >> 7485 plotter! > > The 220 and 240/241 weren't fundamentally different display technologies > they way a dot-matrix differs from a pen-plotter. Both were raster-scan > CRT tubes (AFAICT, they used identical CRT tubes and driver hardwar). > When used in text mode, the 240 wasn't really any different than the > 220. But, the 240 also supported a graphics mode that allowed apps to > draw using vector commands). I remember writing an app using the ReGIS > command set to draw a clock with a moving second hand on a 240. > > Comparing a vt220 to a vt240 is like comparing a black-and-white epson > 9-pin dot-matrix printer that can't do graphics with a balck-and-white > epson 9-pin dot-matrix printer than can do graphics. Sorry if I wasn't clear: I was intending to compare APIs rather than the display mechanisms - I am aware that both text terminals and vector graphics terminals are raster devices, not vector like oscilloscopes. What I was getting at is that the API used to cause graphics or text to be output on a dot-matrix printer is totally unlike that used to draw to same representations on a pen plotter. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Purely historic question: VT200 text graphic programming
On Sat, 12 Mar 2011 17:11:37 +, Grant Edwards wrote: > The point I was trying to make was that the 240 was a > superset of the 220, and could be used identically as the 220 was used. > Fair enough: I bow to hands-on experience. My source - http://vt100.net/vt_history - Says "There is no VT200 as such; the VT220 is a text terminal, while the VT240 and VT241 are graphics terminals, supporting Digital’s ReGIS graphics and Tektronix vector graphics." which I read to mean that a 240/241 wouldn't accept the same command set as the 220. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Purely historic question: VT200 text graphic programming
On Fri, 11 Mar 2011 23:00:25 -0800, Dennis Lee Bieber wrote: > Note the last sentence starting "The name ANSI..." > Indeed. I'm pretty certain that the first time I met the term "ANSI" was in connection with the DOS add-on 'ANSI driver' that allowed programs to control the display by emitting escape codes - without it the DOS screen was treated as a glass teletype. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: [OT] Free software versus software idea patents
On Thu, 14 Apr 2011 13:50:24 -0700, Westley Martínez wrote: > Also, why aren't Opera and Google criticized for their proprietary > browsers (Chrome is essentially a proprietary front-end)? Is it because > their browsers follow web standards, or is it because we have demonized > Microsoft? > Personally, I could care less whether a web browser is proprietary or not. What matters to me is whether its standards-compliant enough to interoperate successfully with standards-compliant browsers. If you're going to argue about web standards its better to criticise non-compliant servers since those will piss off a greater number of people. I think the only real evil is to set out to make a non-standards- compliant server and then design client software that seeks to lock in people to your server. FWIW I'm not certain that is anything that MS deliberately set out to do. Judging from stories over the years of lost code (the Win98 New Year problem) and the general slackness of their project management, its equally possible its simply the result of out of control, anarchic and undocumented software development. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: [OT] Free software versus software idea patents
On Fri, 15 Apr 2011 08:01:42 +1000, Chris Angelico wrote: > On Fri, Apr 15, 2011 at 7:36 AM, Martin Gregorie > wrote: >> I think the only real evil is to set out to make a non-standards- >> compliant server and then design client software that seeks to lock in >> people to your server. FWIW I'm not certain that is anything that MS >> deliberately set out to do. > > IE 6. Why is it still around? Because MS deliberately set out to > encourage companies to use its features (mainly in intranet > applications). > Fair point, though I should point out that IIRC its contemporary rivals were often as bad when it came to standards compliance. I don't remember many standards-compliant browsers before the first release of Opera. Never ascribe to malice things that can equally be achieved by incompetence. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: [OT] Free software versus software idea patents
On Thu, 14 Apr 2011 15:23:01 -0700, Westley Martínez wrote: > On Fri, 2011-04-15 at 08:01 +1000, Chris Angelico wrote: >> On Fri, Apr 15, 2011 at 7:36 AM, Martin Gregorie >> wrote: >> > I think the only real evil is to set out to make a non-standards- >> > compliant server and then design client software that seeks to lock >> > in people to your server. FWIW I'm not certain that is anything that >> > MS deliberately set out to do. >> >> IE 6. Why is it still around? Because MS deliberately set out to >> encourage companies to use its features (mainly in intranet >> applications). >> >> ChrisA > > I always thought it was because sys admins don't bother to upgrade. > Its both: a lot of company intranets that rely on IE6 quirks and have never been upgraded and, evidently, there are still office PCs in use that are old enough to have been bought with it installed and never upgraded. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: The Modernization of Emacs
Twisted wrote:
> This seems to be a closer analogy with emacs versus normal Windows
> text editors.
>
Windows text editors are not normal: most are devoid of all but the most
primitive functions and are further hampered by having an interface that
required frequent time wasting hand transfers from keyboard to mouse
because, if they provide keyboard equivalents at all, these are
remarkably unmemorable and/or undocumented.
Ask anybody who used early versions of Word and you'll hear just how
much faster Word for DOS 5 was than any of its Windows incarnations.
This was because all commands were keystrokes so a skilled typist could
keep both hands on the keyboard all the time. WinWord's interface is a
clunker by comparison.
As for documentation, lets look at vi. Not a great editor, but every
*nix variation has it installed and any fool can learn to use it in
about 2 hours flat and it does at least have good pattern matching.
Can't find its documentation? Never heard of manpages ("man vi") or
apropos ("apropos vi")? My copy of vi understands ":help" and tells you
so if you start it without any arguments.
Amongst its benefits are that you can do anything its capable of by
using only a standard QUERTY keyboard plus ESC - no function keys, etc
are needed - which can save your bacon if somebody misconfigured your
console or the computer is dieing. Beyond that it has user-configurable
KEY BINDINGS so you can also set it up to suit both yourself and that
fancy keyboard in front of you.
> Arguably even the weird controls are superior in some
> way -- but only if you got used to them,
>
You mean that Alt-Esc or Windows-e aren't weird? How long is it since
you tried to use Windows with a dead mouse? There are a set of arcane
keystrokes to replace anything you can do with a mouse but I bet 99% of
windows users don't know any of them apart from TAB and CTRL-ALT-DEL.
> The above applies equally to vi and its derivatives, if not more so --
> vi is like taking that same already-wacky car with the two separate
> throttles and adding, in a fit of quaint nostalgia, the need to
> actually crank-start its engine. ;)
>
If you can't learn enough vi to get by with in half a morning you're
probably well out of your depth here on comp.lang.java.programmer.
Oh, I forgot: you don't read manuals so you ARE out of your depth.
--
martin@ | Martin Gregorie
gregorie. | Essex, UK
org |
--
http://mail.python.org/mailman/listinfo/python-list
Re: The Modernization of Emacs
Sascha Bohnenkamp wrote: >> Windows text editors are not normal: most are devoid of all but the most >> primitive functions and are further hampered by having an interface that >> required frequent time wasting hand transfers from keyboard to mouse >> because, if they provide keyboard equivalents at all, these are >> remarkably unmemorable and/or undocumented. > > well ultra-edit, textpad, source-insight etc. pp are better than that > (and run on windows) I said MOST, not all! To your list I'd add PFE and a Windows port of microEmacs, which has almost nothing in common with EMACS except some key bindings. But to return to your point: how many Windows users actually install the editors we've listed? I bet most never get past Wordpad. I've even found people using Word, of all things, to edit BAT files and program source. I'd give long odds that Windows users who use editors other than Wordpad are using the one that came with whatever IDE they've installed, simply because integrated editors are much more common in Windows-only IDEs that they are on *nixen. My guess is that this is because the standard editors (Wordpad, edlin) are so bad. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: The Modernization of Emacs
Bjorn Borud wrote: > [Martin Gregorie <[EMAIL PROTECTED]>] > | > | As for documentation, lets look at vi. Not a great editor, but every > | *nix variation has it installed and any fool can learn to use it in > | about 2 hours flat and it does at least have good pattern matching. > > there's also the "info" system in Emacs, which not only covers Emacs > itself, but usually also a lot of documentation available for Emacs > extensions and other programs. again, this predates a lot of things > that people are used to today, so just because it seems (and sometimes > is) a bit more fiddly, it must necessarily be inferior. > I thought it might be in "info", like most GNUish things but I couldn't check because I don't have it installed. > for instance, Linux has come a long way in addressing the needs of > desktop users, yet some people refuse to use Linux because it doesn't > behave *exactly* like Windows (as if that was a worthwhile goal) and > they are too lazy or don't think they can manage, to learn a new > system. > Yep, and the same people think a command line is to be avoided at all costs. "I mean, its so /last century/ and you can't do anything useful with it anyway". Obligatory OT comment: right now I have two xterm sessions open with which I've been writing a Swing/JDBC app using nowt but a bash shell, cvs, microEmacs and (of course) J2SE. I don't need no steenking IDE. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: The Modernization of Emacs: terminology buffer and keybinding
BartlebyScrivener wrote: > On Jun 22, 3:47 pm, Twisted <[EMAIL PROTECTED]> wrote: > >> If it requires years of mastery, it is clunky > > Well, now you keep harping on this, but it's just not true. > > I use vim myself, but for purposes of this argument it doesn't matter. > If you take the Vim tutorial and use the help (which appears in a > split window anytime you want it), you can use Vim like any other text > editor within a day or so, especially if you use Cream, which is set > up to hold your Windows hands and act like any other Windows text > editor on the surface. But if you use Vim for YEARS you get better and > faster and more efficient precisely BECAUSE of its arcane > capabilities. If you are going to keep your hands on the keyboard > where they belong, if you REALLY want to go fast, then there's no > alternative to having complex key commands, which become second nature > over time, and take the place of repetitive, totally inefficient > mousing around. > > You might enjoy this. Especially the link to an old essay called > "Interface Zen" > > http://tinyurl.com/2da3om > A good reference. Thanks. I like Interface Zen - much sense there. However, there's a case he missed, probably through never using a CAD system. All the good ones can be driven either by mouse, or by non-chorded key sequences or any combo the user likes. The essence of CAD is very accurate pointing but all too many mice move slightly when clicked. Fortunately a decent CAD system offers the possibility of purely pointing with the mouse and doing everything else with the other hand on the keyboard. The result is both fast and extremely accurate. An interface design point that nobody has yet mentioned here is that there are four different types of user that map onto a grid: casual dedicated untrained 1 2 expert 3 4 I first ran across grid this in "Design of Man-Computer Dialogs" by James Martin and its been very useful in systems interface design. The problem with almost all current GUIs is that they are aimed at types 1 and 3 users - this certainly applies to Windows, OS X and Gnome desktops with the emphasis on type 1. vi and microEmacs, OTOH, are aimed at type 3 and 4 users. Where does emacs fit on this grid? My guess would be 3 and 4. Its very difficult indeed to design an interface that suits both untrained and expert users. About the closest I've seen have been keyboard driven menu structures which have been designed so the user can build their own "command sequences" that traverse menu levels without showing the menus, as long as items are selected correctly from each level. Many CAD systems approximate to this but I've yet to see a graphical desktop, IDE, or editor menu structure that came close. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: The Modernization of Emacs
Joel J. Adamson wrote: > Xerox PARC (not Apple nor MIcrosoft) excelled in helping computers fit > in to how people already lived, not the other way around. > I've never got my hands on a genuine Xerox. About the nearest to that I managed was an ICL PERQ back in 1980, with a portrait-mode black and white screen and a three button mouse. That was the first GUI I saw (next was an Apple Lisa in 1984). The PERQ was dead easy to use after about 5 minutes instruction. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: The Modernization of Emacs: terminology buffer and keybinding
Twisted wrote: > On Jun 23, 10:36 am, Martin Gregorie <[EMAIL PROTECTED]> > wrote: > > Actually, what I prefer in (2D and 3D) visual design apps is the > ability to snap to some kind of grid/existing vertices, and to zoom in > close. Then small imprecisions in mouse movement can be rendered > unimportant. > That might work for visual design apps, but it doesn't for CAD, where you may want to point to an arbitrary position with a (scaled) accuracy of microns. The fact remains that mechanical mice do jump when you click them, though optical mice are better in this respect. > The problem of course being the complete exclusion of type 1 users. > Totally untrue. They are the people that all the standard GUIs are designed for and some (e.g Mackintosh) are very fast to learn. The exclusion is down to the way that the purveyors of GUIs keep spreading bullshit about how any untrained person can use a computer and never mess it up or loose data. Thats not true and never can be: a computer is the most complex device the average person will own or use and is likely to retain that title for the foreseeable future. I grant you that type 2 users are rare, but I think flight simulators may fit this case when used for training. > One with a bog-standard UI but also a "console" or command prompt, > scripting language, and customizable bindings? > Not really. What's needed is a single interface that can be used by anybody from beginner to expert and that, in the case of an error, shows precisely where it got, what caused the action to fail to complete and that allows the user to continue from that point without having to undo/redo the bits that were successful. Its not easy, but it can be done. > ROM BASICs and QBasic (on any really ancient microcomputer, and old > pre-Windows PCs, respectively; the former came with printed manuals > and you could just run prepackaged software from disks very easily; > Hang on: you don't read manuals. You object to using tutorials and to buying books, so its a bit precious to claim this example. > * The word processor with the usual interface where I can define > logical styles, then change them in only one place and affect every > pre-existing occurrence retroactively. > Thats been in Word since DOS days and is part of OpenOffice. Its called a "style sheet". The only WPs I've used that didn't use them were Wordperfect, WordStar, DEC WPS+ and the Wang dedicated WP systems. All were horrid to varying degrees, with Wordperfect and Wordstar tying for worst. > * The word processor with the usual interface where I can also view an > underlying markup representation and hand-hack that, > You're thinking of Wordperfect and its 'Reveal Codes' function. That was the worst idea I've ever seen in a WP, matched only by the illogically arranged set of 12 function keys, each with 4 shifts. > and which likely has the capabilities of the first two as a direct > consequence of the implied architecture. > It didn't. 'Reveal codes' could only let you inspect the current document. Unfortunately it was essential to use it because some input sequences could completely mess up the formatting and the only way to recover was via 'Reveal codes'. The effect was similar to making a data entry clerk use a hex editor on a database to fix keyboarding errors. NOTE: I'm not talking about secretaries using WordPerfect. Those that used it hated it. I'm talking about the experience of IT professionals writing structured text, e.g. system specifications. > * The operating system where you can do powerful stuff with a command > line and a script or two, but can also get by without either. (Windows > fails the former. Linux fails the latter.) > Here you're talking about two different interfaces again. The nearest I've seen to good solutions at OS level were the CL interface provided by ICL's VME mainframes and IBM's AS/400 systems. The latter was particularly good, with a single unified text screen interface which provided help screens, menus and a command line. You could search for and find commands via a menu structure, and then use form filling to provide the arguments, with help available at any stage. If you were writing a script the entire menu and form filling system was available from within the text editor - but when you hit ENTER the completed command was written into your script instead of being executed. Both OS/400 and VME were very consistent in the way they assigned command names and argument keywords. In both OSen it was possible to think "if there's a command to do this it will be called BLAHBLAH", type the name, hit the command completion key and have the argument entry screen pop up ready to be filled in. BTW, in both
Re: The Modernization of Emacs: terminology buffer and keybinding
Twisted wrote: > At least Windows 3.1 had most apps have the same keys for the vast > majority of commands, and those were the right keys. Emacs has all the > applications have the vast majority of their commands use the same > WRONG keys. Including whatever you'd use to rebind them. And the help > you'd use to find out what the damn keys are in the first place. ;) > You're mis-remembering this. Apple, first with the Lisa and then with the Mackintosh, had extremely consistent menus, menu shortcuts and other key assignments. It was possible to teach almost anybody to use them in 15 minutes flat. A major reason for the consistency was the Programmer's Toolbox, a piece of ROM that contained all the stuff an application needed to handle keyboard, mouse and menus. It was there and easy to use, so of course all applications programmers used it. Windows 3 and 3.1 were the first usable Windows versions. Windows 1 and 2 were a bad jokes. Win/286 worked but had no applications. Win 3.x worked a lot better. However, it lacked any equivalent of the Programmers Toolbox and as a result the applications were anything but consistent. MS applications were self-similar, but other apps used wildly divergent ideas about menu structures, shortcuts and key assignments. Compare 3.x versions of Word with Wordperfect, or the Borland IDEs and this is obvious. MS finally kicked applications providers into more-or-less consistency but that wasn't before Win 95 appeared and they then spoilt the record by arbitrary and capricious menu changes as each version of MS Office appeared. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: The Modernization of Emacs: terminology buffer and keybinding
Twisted wrote: > The manuals came with the computers, at no additional charge. It was a > different time. This isn't going to be true of any separately- > purchased book or user-made printout concerning emacs. Also, the > manuals provided a basic introduction for the beginning user. A > traditional-unix-tool providing anything resembling that would > genuinely shock me. > Oh, so manuals are OK and you'll read them if they are dead trees that came in the same box as the software, but not if they're HOWTOs, online documentation or O'Reilly books? > I distinctly remember Winword circa 2002 not being able to > retroactively change all of a bunch of like-formatted paragraphs > easily. Not without delving into VBscript or something, anyway. > So you didn't read the free but thick and stodgy Word manual? Styles and style sheets have been in Word since Word for DOS 5. Changing a style sheet has always affected all documents that reference it. > > Oh, because the implementation (of "reveal codes" and of everything > else) was awful, not because of any intrinsic flaw in the idea itself. > If a word processor, which by definition is provides a WYSIWYG user interface, can't produce perfectly formatted text by editing a representation of the finished result then its a deeply flawed program and not fit for purpose. By providing 'Reveal codes' and by being designed in such a way as to force its regular use, Wordperfect reveals itself as being no better than nrof or tex - its like expecting a user to write postscript source with a text editor and providing a separate window with a Postscript viewer to see what the final result will look like. > Would you want to edit a Web page without being able to hand-hack the > HTML? > Of course not, but HTML isn't anything to do with WYSIWYG and any system (Coldfusion, Front Page, HTML from Word) that pretends it is WYSIWG is both useless and perpetrating a fraud. > What happened to the guys that did all this stuff after it became > obsolete? > It isn't obsolete despite going back a looong way. The hardware and software was originally developed as Future Series (intended S/360 replacement), was canned in 1970 but resurfaced in the late 80s as System/38. A second generation appeared as AS/400, was renamed to (I think) Z-series and are now known as iSeries servers. Its good, reliable kit and easy to work with if you don't mind programming in RPG. I know of no better "one size fits all" interface design than that provided by the OS/400 operating system. Its still called that. Its a pity the interface style hasn't been emulated by others. > It would be nice if straightforward macro recording was standard in Windows > though. > It was standard with Win 3.1 and 3.11 and it was bloody useless. Most people I know tried it once or twice before giving up and writing .BAT files or putting up with RSI. The problem was that it recorded keystrokes and mouse clicks. Even minor changes to the screen layout made it fail and the macros couldn't be edited or parameterised nor made to prompt for filenames, etc. You can do better with Gnome, thanks to tcl, but I think most people go straight to Ruby or stick to plain vanilla shell scripts. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Portable general timestamp format, not 2038-limited
Steve O'Hara-Smith wrote: > On Mon, 25 Jun 2007 11:17:27 GMT > Roedy Green <[EMAIL PROTECTED]> wrote: > >> On Sun, 24 Jun 2007 18:14:08 -0700, [EMAIL PROTECTED] (Robert Maas, >> see http://tinyurl.com/uh3t) wrote, quoted or indirectly quoted >> someone who said : >> >>> - Stick to astronomical time, which is absolutely consistent but >>> which drifts from legal time? >> depends what you are measuring. IF you are doing astronomy, your >> advice would apply. If you are doing payrolls, you want effectively to >> pretend the leap seconds never happened, just as Java does. > > Which leaves you about 30 seconds out by now - smelly. > Easy solution: always read Zulu time directly from a recognized real-time clock and store the result in a database as a ccyymmddhhmmssfff ASCII string where fff is milliseconds). By "recognized real-time clock) that I mean an atomic clock and distribution network such as GPS or (in the UK or Germany) an MSF low frequency radio broadcast. NTP using tier-1 sources may do the job too. The clock interface may need to be JINI because most suitable receivers have serial interfaces. This is certainly accurate for financial transactions: the UK CHAPS inter-bank network has a Rugby MSF receiver in each bank's gateway computer and uses that for all timestamps. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: The Modernization of Emacs: terminology buffer and keybinding
Twisted wrote: > > First, I didn't claim the ideal WP was necessarily perfectly WYSIWYG. > Maybe I should have clarified my viewpoint. When it comes to programs that operate on the content of textual documents a word processor is WYSIWYG by definition. Anything else is a text editor. You may have a different view but that's mine. > Your quiet change from discussing word processing to discussing > WYSIWYG is interesting. > See above. We were actually discussing text editors whose formatting capabilities (unless they are syntax-sensitive) are generally limited to line wrapping and auto-indentation. You introduced more complex document reformatting - something that I regard as a capability of word processors rather than text editors. > Programming in role-playing game? And I meant my roguelike-filesystem- > interface suggestion at least partly in jest... > RPG is "Report Generating Program" in the context of programming languages. The RPG language is horrid: its a bastardized, fixed column assembler derivative that's been shoehorned into a typical report generator's processing loop. Even PL/1 and COBOL shine as paragons of programming language design by comparison. > If it's so great, why hasn't it, and why hasn't OS/400 managed to > escape from persistent obscurity? > A fair question. I don't know, but it probably has a lot to do with AIX and the UNIX command shell with its great power but lack of consistency in naming, etc. > In other words, the implementation was a dog. That doesn't refute the > basic concept's validity. > True, but doing better would be really hard because of all the information and context that would need to be associated with every mouse click in case it was needed to record a macro. At best it might make macro recording tedious. At worst it could make the whole GUI unresponsive. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Portable general timestamp format, not 2038-limited
Paul Rubin wrote: > Martin Gregorie <[EMAIL PROTECTED]> writes: >>>> pretend the leap seconds never happened, just as Java does. >>> Which leaves you about 30 seconds out by now - smelly. >> Easy solution: always read Zulu time directly from a recognized >> real-time clock > > That's no good, it doesn't let you accurately compute the difference > between timestamps. > I don't recall the OP mentioning time interval computability - just a requirement for sub second accuracy timestamps. > If you want a precise timestamp and you don't > want to deal with leap seconds, TAI is one approach. > TAI? Care to provide a reference? > There is > currently some political pressure to get rid of leap seconds to ease > computer synchronization, but (at least some of) the astronomy > community is opposed; see > Yes, that's just silly, especially because if you're trying to do date-time calculations across historic time or non-western calendars (e.g. Islamic) the minuscule accumulated leap second error is dwarfed by all the other uncertainties. > No do NOT use stratum 1 sources for something like this. > Fair comment. I was thinking about network delays and jitter and should not have forgotten Stratum 1 congestion. Of course, you could always run your own local Stratum 1 clock if accuracy is that important. IIRC the major American interbank networks use GPS as their time standard because its about the only system that can avoid jitter and propagation delays over continental areas without introducing smoothing engines, e.g. ntpd. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Portable general timestamp format, not 2038-limited
Paul Rubin wrote: > Same one already given: http://cr.yp.to/proto/utctai.html Nope - you referenced leap seconds, not TAI and not that URL Thanks for the reference, though. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Portable general timestamp format, not 2038-limited
[EMAIL PROTECTED] wrote: > On Jun 25, 6:46 pm, Paul Rubin <http://[EMAIL PROTECTED]> wrote: >> TAI really does seem like the most absolute--if you are a user in >> orbit or on Mars, then UTC timestamps will seem pretty meaningless and >> artificial. > > TAI makes sense for clocks on the surface of the earth (at least until > ion trap clocks and picosecond intercomparison become routine, at > which point not even TAI tells what time it is for you), but clocks > off the surface of the earth tick at rates which already differ > nonlinearly from TAI by measurable amounts. > True. The first direct demonstration of relativistic time dilation was made in 1971 with three HP cesium beam atomic clocks. One stayed in the lab, while the other were shipped round the world in opposite directions on commercial jet flights. When the clocks were compared afterwards the errors in the traveling clocks agreed with theory within experimental error. See: http://hyperphysics.phy-astr.gsu.edu/hbase/relativ/airtim.html for more detail. This shows the clocks don't have to be moving at interplanetary speeds to be significantly affected. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Date<->UNIX timestamp mapping (was Portable general timestamp format, not 2038-limited)
Paul Rubin wrote: > Martin Gregorie <[EMAIL PROTECTED]> writes: >>> Same one already given: http://cr.yp.to/proto/utctai.html >> >> Nope - you referenced leap seconds, not TAI and not that URL > > Oh whoops, I thought I put that url further up in the thread. > I remember grumbling to myself about having to look for it twice. > Maybe I'm just confused. Anyway it's pretty interesting stuff, > as is the Wikipedia article someone else linked to. > Thinking of interesting date & time related stuff, there's another document I remember seeing a while back - probably around early '98. It was an ASCII configuration file that contained to rules for mapping human readable dates & times to UNIX timestamps after taking account of changes of calendar (e.g. the switch between Julian and Gregorian calendars), the introduction of daylight saving time, etc. I remember that it was mostly comment interspersed with mapping rules and that the comments were vast and fascinating, often including copies of e-mail threads. The file was part of a Linux distro, probably Debian. Some time later, after I set up my first Linux system, I went looking for it without success, probably because by that time (RedHat 6.2) the date mapping rules had become encoded as some sort of binary rule set. I'd very much like to know where I could find a copy of that file. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Portable general timestamp format, not 2038-limited
Roedy Green wrote: > > To add to the confusion you have GPS, Loran and Julian day also used > as scientific times. > GPS time is UTC time and I'd assume the same is true for Loran. Both are primarily for navigation and so are on Zulu time, which is another name for UTC. Zulu is the international radio word for the letter Z. I've never seen Julian time used outside the world of IBM mainframes. I'd be interested to know who else uses it. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Portable general timestamp format, not 2038-limited
Roedy Green wrote: > On Tue, 26 Jun 2007 13:04:50 +0100, Martin Gregorie > <[EMAIL PROTECTED]> wrote, quoted or indirectly quoted > someone who said : > >> TAI? Care to provide a reference? > > see http://mindprod.com/jgloss/tai.html > Thanks. Your list of NTP servers on http://mindprod.com/jgloss/timesources.html may be a bit out of date: I notice that it doesn't include the European or Oceania time server pools (0.europe.pool.ntp.org, 0.oceania.pool.ntp.org). It may be best to just hold a link to the NTP project servers page, http://support.ntp.org/bin/view/Servers/WebHome -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Portable general timestamp format, not 2038-limited
Roedy Green wrote: > On Sun, 01 Jul 2007 17:47:36 +0100, Martin Gregorie > <[EMAIL PROTECTED]> wrote, quoted or indirectly quoted > someone who said : > >> GPS time is UTC time and I'd assume the same is true for Loran. > > not according to this site that has clocks running on all three. > They are out slightly. > > http://www.leapsecond.com/java/gpsclock.htm > A useful reference: thanks. Interesting that GPS and Loran are slightly out from Zulu. I'll ask round and see if this is something that ATP pilots are aware of: as a glider pilot and GPS user I'd never heard of it. So far the deviations from UTC are probably not enough to affect navigation significantly, but I wonder if banks using networks that timestamp transactions with GPS time know about it. Of course the deviation can't affect disputes where the timestamps are being used to decide event sequences within a single network. However, there could be legal implications if absolute time is important or if the timestamps are being compared across different networks, e.g SWIFT vs CHAPS or Fedwire. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Portable general timestamp format, not 2038-limited
Peter J. Holzer wrote: > On 2007-07-03 05:12, Scott David Daniels <[EMAIL PROTECTED]> wrote: >> TOPS-20 did an interesting format which suggest an interesting variant: >> Tops-20: 36-bit (the machine word size) fixed-bit representation >>of days since a given moment (the first Photographic >>plates of the sky). The "binary point" was at the middle >>of the word; the low order 18 bits were the time of day >>(GMT), the high-order 18 bits were the days-since date. >> I think there's a definite practical advantage in storing dates as a day count from a base date and providing a standard set of procedures/methods to convert it to and from a human-readable format. It makes all sorts of date calculations much easier. For instance, if there's a requirement to produce statements dated for the last day of the current month the pseudo code is simply: - convert the date to ccyymmdd format - add 1 to the month, adjusting the year and century to fix year roll-over and set the day to 1 - convert back to day count and subtract 1 - the result, output in readable form is the last day of the month irrespective of month length, leap years, etc. I don';t think it matters what the base date is, though the Astronomical base date [12 noon on 1 JAN -4712 (4713 BC)] may be as good as any. Other bases I've seen (apart from UNIX date) are ICL 1900 mainframes, which set day zero as 31 Dec 1899 and held the time separately. ICL 2900 systems held the date and time as microseconds since 00:00:00 1900-01-01 in a 64 bit word, which is also easy to deal with and allows the same set of date arithmetic operations as a straight day number. BTW, be sure to distinguish Julian Day and Modified Julian Day (used by astronomers from the "Julian Date" that [used to] be used by IBM mainframes. The former is a day count but the latter is day within year (yyddd). JD and MJD are described in: http://tycho.usno.navy.mil/mjd.html -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Portable general timestamp format, not 2038-limited
Ilya Zakharevich wrote: > [A complimentary Cc of this posting was sent to > greg > <[EMAIL PROTECTED]>], who wrote in article <[EMAIL PROTECTED]>: >> Ilya Zakharevich wrote: >>> In pedantic mode: negative timestamps make sense with Big Bang as the >>> epoch as well. (AFAIU, the current way of thinking is that it was >>> "just too hot" before the big bang, it is not that "there was >>> nothing".) >> If Stephen Hawking is right, the shape of the universe >> is such that there isn't any time "before" the big bang >> at all. It's like asking what's north of the North Pole. > > I do not remember any statement like this - even from 70s... Could > you provide a reference? There were conjectures about "initial > singularity", but I do not recollect them related to SH. > Its in "A Short History of Time". Sorry I can't quote chapter or page, but a friend borrowed my copy and lent me Dawkins "Climbing Mount Improbable" before vanishing, never to be seen since. Not an equal exchange: I preferred ASHOT to CMI. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Portable general timestamp format, not 2038-limited
Ilya Zakharevich wrote: > [A complimentary Cc of this posting was sent to > Martin Gregorie > <[EMAIL PROTECTED]>], who wrote in article <[EMAIL PROTECTED]>: >> Its in "A Short History of Time". Sorry I can't quote chapter or page, >> but a friend borrowed my copy and lent me Dawkins "Climbing Mount >> Improbable" before vanishing, never to be seen since. Not an equal >> exchange: I preferred ASHOT to CMI. > Oops - I should have written "A Brief History of Time". It was the first edition, so I don't know if it was altered/edited out of later versions. > I would prefer a reference to a peer-reviewed paper. ;-) > Sure, but I don't think you'll find one. It was in a descriptive, rather than rigorous, passage. But then, the book famously had only one equation in it. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Portable general timestamp format, not 2038-limited
Ilya Zakharevich wrote: > My point is that attributing something to SH due to it appearing in > ABHoT is like attributing it to you since it was mentioned in your > post... > OK, so who should it be attributed to? -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: The Importance of Terminology's Quality
On Tue, 24 Jun 2008 18:42:15 -0400, John W Kennedy wrote: > David Combs wrote: >> passing >> *unnamed* functions as args (could Algol 60 also do something like that, >> via something it maybe termed a "thunk") > > No, the "thunks" were necessary at the machine-language level to > /implement/ ALGOL 60, but they could not be expressed /in/ ALGOL. > Are you sure about that? The first time I ran across the term "thunking" was when Windows 3 introduced the Win32S shim and hence the need to switch addressing between 16 bit and 32 bit modes across call interfaces. That was called "thunking" by Microsoft and even they would surely admit it was a kludge. I used Algol 60 on an Elliott 503 and the ICL 1900 series back when it was a current language. The term "thunking" did not appear in either compiler manual nor in any Algol 60 language definition I've seen. A60 could pass values by name or value and procedures by name. That was it. Call by name is what is now referred to as reference passing. I should also point out that Algol 60 was initially written as a means for communicating algorithms between people. Compiler implementations came later. In consequence the language did not define links to libraries or i/o methods. Both features were compiler specific - for instance the Elliott introduced 'input' and 'print' reserved words and syntax while the 1900 compilers used function calls. The Elliott approach was more readable. Algol 60 did not have 'functions'. It had procedures which could be declared to return values or not. A procedure that returned a value was equivalent to a function but the term 'function' was not used. Similarly it did not have a mechanism for declaring anonymous procedures. That, like the incorporation of machine code inserts, would have been a compiler-specific extension, so it is a terminological mistake to refer to it without specifying the implementing compiler. -- martin@ | Martin Gregorie gregorie. | org | Zappa fan & glider pilot -- http://mail.python.org/mailman/listinfo/python-list
Re: The Importance of Terminology's Quality
On Sat, 16 Aug 2008 21:46:18 -0400, John W Kennedy wrote: > Martijn Lievaart wrote: >> On Thu, 14 Aug 2008 18:33:30 -0400, John W Kennedy wrote: >> >>> Actually, I was thinking of the 1401. But both the 1620 and the 1401 >>> (without the optional Advanced Programming Feature) share the basic >>> omission of any instruction that could do call-and-return without >>> hard-coding an adcon with the address of the point to be returned to. >>> (The Advanced Programming Feature added a 1401 instruction, Store >>> B-address Register, that, executed as the first instruction of a >>> subroutine, could store the return-to address.) >> >> Rgh >> >> Don't. Bring. Back. Those. Nightmares. Please. >> >> The 1401 was a decent enough processor for many industrial tasks -- at >> that time -- but for general programming it was sheer horror. > > But the easiest machine language /ever/. What? Even easier than ICL 1900 PLAN or MC68000 assembler? That would be difficult to achieve. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: The Importance of Terminology's Quality
On Sun, 17 Aug 2008 22:30:35 -0400, John W Kennedy wrote: > I said "machine language" and I meant it. > OK - I haven't touched that since typing ALTER commands into the console of a 1903 running the UDAS executive or, even better, patching the executive on the hand switches. I was fascinated, though by the designs of early assemblers: I first learnt Elliott assembler, which required the op codes to be typed on octal but used symbolic labels and variable names. Meanwhile a colleague had started on a KDF6 which was the opposite - op codes were mnemonics but all addresses were absolute and entered in octal. I always wondered about the rationale of the KDF6 assembler writers in tackling only the easy part of the job. > Even shops that used assembler nevertheless frequently did bug fixes as > machine-language patches, rather than take the time to run the assembler > again. (SPS, the non-macro basic assembler, ran at about 70 lines a > minute, tops.) > Even a steam powered 1901 (3.6 uS for a half-word add IIRC) running a tape based assembler was faster than that. It could just about keep up with a 300 cpm card reader. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: The Importance of Terminology's Quality
On Thu, 21 Aug 2008 09:11:48 -0500, Rob Warnock wrote: > You're assuming that all machines *have* some sort of "boot ROM". Before > the microprocessor days, that was certainly not always the case. The > "boot ROM", or other methods of booting a machine without manually > entering at least a small amount of "shoelace" code [enough the *load* > the real bootstrap], was a fairly late invention. > Quite. I never knew how to boot the Elliott 503 (never got closer to the console than the other side of a plate glass window). However, I dealt with that aspect of ICL 1900s. They had ferrite core memory and NO ROM. When you hit Start this cleared the memory and then pulsed a wire that wrote the bootstrap into memory and executed it. The wire wove through the cores and wrote 1 bits to the the right places to: - set word 8 (the PC) to 20 - set 25 words from 20 as bootstrap instructions to boot from disk. Then it started the CPU running. On the 1902 this sequence often didn't work, so a good operator knew the 25 words by heart and would toggle them in on hand switches, set PC to 20 and hit the GO switch. > > -Rob > > p.s. Similarly, the DEC PDP-8 & PDP-11 were also originally booted by > manually toggling the console switches in order to deposit a few > instructions into memory, and then the starting address was toggled in > and "Start" was pushed. It was only later that a boot ROM became > available for the PDP-11 (as an expensive option!) -and only much later > still for the PDP-8 series (e.g., the MI8E for the PDP-8/E). > > - > Rob Warnock <[EMAIL PROTECTED]> > 627 26th Avenue http://rpw3.org/> San Mateo, CA 94403 > (650)572-2607 -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: The Importance of Terminology's Quality
On Fri, 22 Aug 2008 22:56:09 +, sln wrote: > On Thu, 21 Aug 2008 09:11:48 -0500, [EMAIL PROTECTED] (Rob Warnock) wrote: > >>[EMAIL PROTECTED]> wrote: >>*IS* raw machine code, *NOT* assembler!! > [snip] > > I don't see the distinction. > Just dissasemble it and find out. > There's a 1:1 relationship between machine code and assembler. Unless its a macro-assembler, of course! > > Each op is a routine in microcode. > That is machine code. Those op routines use machine cycles. > Not necessarily. An awful lot of CPU cycles were used before microcode was introduced. Mainframes and minis designed before about 1970 didn't use or need it and I'm pretty sure that there was no microcode in the original 8/16 bit microprocessors either (6800, 6809, 6502, 8080, 8086, Z80 and friends). The number of clock cycles per instruction isn't a guide either. The only processors I know that got close to 1 cycle/instruction were all RISC, all used large lumps of microcode and were heavily pipelined. By contrast the ICL 1900 series (3rd generation mainframe, no microcode, no pipeline, 24 bit word) averaged 3 clock cycles per instruction. Motorola 6800 and 6809 (no microcode or pipelines either, 1 byte fetch) average 4 - 5 cycles/instruction. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: The Importance of Terminology's Quality
On Sat, 23 Aug 2008 00:06:28 -0400, John W Kennedy wrote: > Martin Gregorie wrote: >> Not necessarily. An awful lot of CPU cycles were used before microcode >> was introduced. Mainframes and minis designed before about 1970 didn't >> use or need it > > No, most S/360s used microcode. I never used an S/360. I thought microcode came into the IBM world with S/370 and Future Series (which later reappeared as the AS/400, which I did use). Didn't the S/370 load its microcode off an 8 inch floppy? -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: The Importance of Terminology's Quality
On Sat, 23 Aug 2008 21:22:05 -0400, John W Kennedy wrote: > Martin Gregorie wrote: >> On Sat, 23 Aug 2008 00:06:28 -0400, John W Kennedy wrote: >> >>> Martin Gregorie wrote: >>>> Not necessarily. An awful lot of CPU cycles were used before >>>> microcode was introduced. Mainframes and minis designed before about >>>> 1970 didn't use or need it >>> No, most S/360s used microcode. >> >> I never used an S/360. >> >> I thought microcode came into the IBM world with S/370 and Future >> Series (which later reappeared as the AS/400, which I did use). Didn't >> the S/370 load its microcode off an 8 inch floppy? > > Some did, but not all. The 370/145 was the first, and made a big splash > thereby. > ..snip... Thanks for that. As I said, during most of that era I was using ICL kit. Microcode was never mentioned in the 1900 contect. Hoiwever, they had a very rough approximation called extracodes. though they were closer to software traps than microcode: if hardware didn't implement an op code the OS intercepted it and ran equivalent code. This was used for i/o operations and for FP instructions on boxes that didn't have FP hardware. As a result all boxes executed the same instruction set. Some opcodes might be very slow on some hardware but it would execute. The 2900 series had huge amounts of microcode - it even defined both memory mapping and opcodes. You could run 1900 code (24 bit words, fixed length instructions, ISO character codes) simultaneously with 'native' code (8 bit bytes, v/l instructions, EBCDIC) with each program running under its usual OS (George 3 for 1900, VME/B for 1900). The only other systems I'm aware of that could do this were the big Burroughs boxes (6700 ?), which used a byte-based VM for COBOL and a word- based VM for FORTRAN and Algol 60) and IBM AS/400 (OS/400 could run S/34 code alongside S/38 and AS/400 code). AFAICT Intel virtualisation doesn't do this - all code running under VMware or any of the other VMs is still running in a standard Intel environment. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: The Importance of Terminology's Quality
On Mon, 01 Sep 2008 12:04:05 -0700, Robert Maas, http://tinyurl.com/uh3t wrote: >> From: George Neuner <[EMAIL PROTECTED]> A friend of mine had an >> early 8080 micros that was programmed through the front panel using >> knife switches > > When you say "knife switches", do you mean the kind that are shaped like > flat paddles? > Pedantic correction: "Knife switch" is the wrong term. These are high current switches, typically used in the sort of heavy duty circuit where the wiring hums when power is on or in school electrical circuits so even the back of the class can see whether the switch is open or closed. In these a copper 'blade' closes the contact by being pushed down into a narrow, sprung U terminal that makes a close contact with both sides of the blade. Like this: http://www.science-city.com/knifeswitch.html What you're talking is a flat handle on a SPST or DPST toggle switch. It is often called a paddle switch and mounted with the flats on the handle horizontal. Like this, but often with a longer handle: http://www.pixmania.co.uk/uk/uk/1382717/art/radioshack/spdt-panel-mount- paddle-s.html -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: The Importance of Terminology's Quality
On Mon, 01 Sep 2008 20:48:23 -0400, George Neuner wrote: > I don't know the correct term, but what I was talking about was a tiny > switch with a 1/2 inch metal handle that looks like a longish grain of > rice. We used to call them "knife" switches because after hours > flipping them they would feel like they were cutting into your fingers. > That sounds like a sub-minature SPDT toggle switch with a normal handle. Cheap as chips, which is probably why they were used on that front panel. Like this by any chance? http://www.maplin.co.uk/images/300/fh00a_ff70m.jpg -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Is Unladen Swallow dead?
On Wed, 17 Nov 2010 23:51:01 +0100, Alexander Kapps wrote: > On 17.11.2010 23:09, John Nagle wrote: >> On 11/17/2010 12:49 PM, John Ladasky wrote: >>> On Nov 16, 2:30 pm, laspi wrote: >>>> Is Unladen Swallow dead? >>> >>> No, it's just resting. >> >> For those who don't get that, The Monty Python reference: >> "http://www.mtholyoke.edu/~ebarnes/python/dead-parrot.htm"; > > Thank you John for making my light enough Wallet even lighter, now I > have to go and buy the original English version. Seems the German > translation sucks (misses a lot) and my copy lacks the original dub. > While you're at it, pick up the video of "Monty Python and the Holy Grail". the project name, Unladen Swallow, is a reference to the film. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: How to read such file and sumarize the data?
On Wed, 17 Nov 2010 13:45:58 -0800, huisky wrote: > Say I have following log file, which records the code usage. I want to > read this file and do the summarize how much total CPU time consumed for > each user. > Two points you should think about: - I don't think you can extract CPU time from this log: you can get the process elapsed time and the number of CPUs each run has used, but you can't calculate CPU time from those values since you don't know how the process spent waiting for i/o etc. - is the first (numeric) part of the first field on the line a process id? If it is, you can match start and stop messages on the value of the first field provided that this value can never be shared by two processes that are both running. If you can get simultaneous duplicates, then you're out of luck because you'll never be able to match up start and stop lines. > Is Python able to do so or say easy to achieve this?, anybody can give > me some hints, appricate very much! > Sure. There are two approaches possible: - sort the log on the first two fields and then process it with Python knowing that start and stop lines will be adjacent - use the first field as the key to an array and put the start time and CPU count in that element. When a matching stop line is found you, retrieve the array element, calculate and output or total the usage figure for that run and delete the array element. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Program, Application, and Software
On Thu, 18 Nov 2010 01:16:34 +, MRAB wrote: > I'd probably say that a "script" is a program which is normally not > interactive: you just set it up, start it, and let it do its work (a > "batch script", for example). It's also written in a language primarily > designed for convenience rather than speed (Want to manipulate large > chunks of text? Fine! :-)). > I use 'script' to refer to programs written in languages that don't have a separate compile phase which must be run before the program can be executed. IOW Python and Perl programs are scripts aloing with programs written as awk, Javascript and bash scripts. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Program, Application, and Software
On Thu, 18 Nov 2010 15:41:51 +, Grant Edwards wrote: > On 2010-11-18, Martin Gregorie wrote: >> On Thu, 18 Nov 2010 01:16:34 +, MRAB wrote: >> >>> I'd probably say that a "script" is a program which is normally not >>> interactive: you just set it up, start it, and let it do its work (a >>> "batch script", for example). It's also written in a language >>> primarily designed for convenience rather than speed (Want to >>> manipulate large chunks of text? Fine! :-)). >>> >> I use 'script' to refer to programs written in languages that don't >> have a separate compile phase which must be run before the program can >> be executed. IOW Python and Perl programs are scripts aloing with >> programs written as awk, Javascript and bash scripts. > > I use 'script' to refer to programs that are used to automate things > that would otherwise be done by a person typing commands. > Its a long-standing UNIXism - it applies to any language implementation where you can add "#!/path/to/interpreter" as the first line of the source file, make the file executable with chmod and run it by treating the file name as a command. > IOW, what I write using Python aren't scripts. They're programs. > Anything with one or more executable lines is a program: #!/usr/bin/python print "\nHello, Python World\n" -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Program, Application, and Software
On Fri, 19 Nov 2010 00:07:05 +, Steven D'Aprano wrote: > On Thu, 18 Nov 2010 14:21:47 +, Martin Gregorie wrote: > >> I use 'script' to refer to programs written in languages that don't >> have a separate compile phase which must be run before the program can >> be executed. IOW Python and Perl programs are scripts aloing with >> programs written as awk, Javascript and bash scripts. > > > You're mistaken then about Python, because it does have a separate > compilation phase that runs before the program can be executed. Where do > you think the .pyc files come from, and what did you think the compile() > function did? It just happens automatically, rather than manually. > Notice my mention of the hash-bang line? That makes it quite clear that I did not mean what you're reading into "separate compile phase which must be run" because hash-bang means you can feed source into the run-time execution system, and also means that by definition the source contains or references everything a program needs to run and that the source code translation step, if there is one, is fast enough for the time it takes to be negligible. Those aren't necessarily the case with a separate compilation step. .pyc files are an optional extra: in a *nix environment they are not produced if you use a hash-bang line to execute the source file: they only appear if you run the source with a "python file.py" command. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Program, Application, and Software
On Fri, 19 Nov 2010 01:43:28 +0100, Alexander Kapps wrote: > What difference does it make? Is 'print "Hello"' a program or a script? > Are you saying, that it depends on whether you have to manually call > some compiler? Thats the way the term 'script' is usually used in the UNIX/Linux world. In that environment you'd call awk and Perl scripting languages but C and Java are known as compiled languages. The size of the source isn't relevant: if you can mark a source file as executable and simply run it its a 'script' while if it needs a separate preparatory step to generate a separate executable its just a source file for a compiled language. The distinction doesn't seem to be used in a Windows environment. Indeed, it doesn't make sense there since executables are limited to .BAR or .CMD files, which are directly interpreted by the command processor, and .EXE or .COM files, which must be compiled before they can be run. AFAIK there's no way you can mark anything else, such as an awk, Perl or Python source file, as executable since there is no 'executable' attribute in any Windows filing system. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: building a web interface
On Sat, 20 Nov 2010 14:40:16 -0800, Shel wrote: > Hello, > > I am pretty new to all this. I have some coding experience, and am > currently most comfortable with Python. I also have database design > experience with MS Access, and have just created my first mySQL db. > > So right now I have a mySQL db structure and some Python code. My end > goal is to create a browser-based interactive fiction/game thing. My > code is currently just using dummy data rather than pulling in data from > the db, but I think/hope it won't be too big of a deal to interact with > the db through Python (famous last words...). > Wrong approach. If you're going to use a database, start with writing a program that puts your data into it alongside the program that reads and uses the data. Do it incrementally: if you are writing an adventure game a good place to start would be the 'place' table and the part of the game builder that adds/amends/removes a place from the database as well as enough of the game program to show the place description and use the doors to move from place to place. This way you'll have that working before you start adding weapons, monsters, etc. or try to implement fights. You'll also learn that writing place descriptions isn't as easy as it sounds: if there are several doors into a room the description must make sense no matter which way you come into it. Its quite hard at first to avoid descriptions like "Further into the forest. Its getting darker" which may be OK coming from a road but is pretty stupid if you're on your way out of the forest toward the road. http://www.kitebird.com/articles/pydbapi.html which tells you what you need to know about accessing SQL databases. > My main problem right now is how to do the web interface. I don't know > much about web architecture, unfortunately. I think I need a CGI script? > You're biting off quite a lot at once. It may be easier to first write a command line Python program and get it working in that form. Once you have the game/application logic working you can adapt it to run as a CGI script. Development and debugging will be easier that way. If you're not familiar with this style of adventure game, look here: http://www.rickadams.org/adventure/ You'll find various downloadable examples on the downloads page as DOS executables and in C or Fortran source code. Have a play with it to see what even this very basic user interaction can do - you'll have fun and learn useful stuff. > What I would really like is to find a GUI tool to design the interface > that would have customizable templates or drag-and-drop elements or > something, so I wouldn't have to code much by hand. Something easy, if > such a tool exists. > If you're doing a textual adventure of something like it a tool won't gain you much, since you only need a few skeleton web pages than your program can select and drop text or images into. > Also would prefer something that I would be able to install/use without > having much access to the server where the site will be hosted (on a > university server that I don't have control over). I don't fully > understand how a lot of the tools I have been looking at work, but it > seems like they're often things where you'd have to get an administrator > to do stuff on the server (is that right?), which I would prefer to > avoid. > CGI and Python should be OK providing the server has Python available and the admins will let you use CGI, but that's not a good development platform. You really need a web server you can start, stop and fiddle with once you get to the point of putting your code in a server. Running a simple local web server on your computer would be a better place to start: once its running you simply point your browser at localhost:80 and send it the URL of the initial page of your application. Search on "Python web server" for details of building or downloading a simple one. You'll also find stuff about interfacing Python programs to a web server. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: building a web interface
On Sat, 20 Nov 2010 17:20:53 -0800, Shel wrote: > Sorry I wasn't clear about the db part. Most of the data has already > been written and/or generated and tested with the code. I do plan to > finish all the database stuff before going to the front end, but am just > thinking ahead about how to do the interface. > That sounds good. Sorry if I was repeating stuff you already know, but it wasn't obvious what you knew about care & feeding of an RDBMS. I'll just add two comments on databases: - Decompose the database design to 3NF form and make sure all prime and foreign keys have indexes. This is stuff that previous experience shows self-taught Access users don't do. Not doing it will bite you hard on performance as soon as the tables exceed a few rows in size. Fixing it later can force major changes to the programs as well. - If you haven't looked at it yet, find out about the EXPLAIN verb and what its output means. Use it on all queries that your online program issues and take notice of how rearranging the query and/or adding/changing indexes affects the cost of the query. Lower cost queries mean higher performance and hence faster response times. What I meant to add last night is that, if your application is to be used by more than a single user at a time a prime consideration is how you will recognise input received from each user and how you'll store their context between interactions with them in the same session and keep each session's context separate. The web server doesn't do this, so this managing session context is the application's responsibility. Common methods are to use a session cookie and/or to store session context in the database. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: building a web interface
a really simple 'do almost nothing' version of 1 while that in turn can be developed and tested on your desktop using 2 and later be dropped into the web server with 3 as its interface. I have an in-house copy of Apache that I'd use to develop your type of program. Its used for all my website development so that nothing gets loaded onto my public sites until its been properly checked out here. You can do the same if you can find and install a really simple web server that would run on your PC together with a local copy of MySQL - of course! Given this setup you can use your usual web browser to talk to the local web server. If you can run all that you won't need 2 because you can have your simple web server and program running in a console window on your desktop PC while you hammer it from your web browser. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Comparison with False - something I don't understand
On Mon, 06 Dec 2010 09:54:46 -0800, Dennis Lee Bieber wrote: > On Mon, 06 Dec 2010 00:14:11 -0800, Paul Rubin > declaimed the following in gmane.comp.python.general: > > >> exceptions that fixed the issue. Are there any languages out there >> with resumable exceptions? Escaping to a debugger doesn't really count >> as > > Visual BASIC 6 > > -=-=-=-=- > On Error GoTo line > REM Actions to sort out the error RESUME > Enables the error-handling routine that starts at line > specified in the required line argument. The line argument is any line > label or line number. If a run-time error occurs, control branches to > line, making the error handler active. The specified line must be in the > same procedure as the On Error statement; otherwise, a compile-time > error occurs. > Any BASIC that implements ON ERROR (i.e. just about all of them) will do this, not just VB. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Scanning directories for new files?
On Tue, 21 Dec 2010 14:17:40 -0500, Matty Sarro wrote: > Hey everyone. > I'm in the midst of writing a parser to clean up incoming files, remove > extra data that isn't needed, normalize some values, etc. The base files > will be uploaded via FTP. > How does one go about scanning a directory for new files? For now we're > looking to run it as a cron job but eventually would like to move away > from that into making it a service running in the background. > Make sure the files are initially uploaded using a name that the parser isn't looking for and rename it when the upload is finished. This way the parser won't try to process a partially loaded file. If you are uploading to a *nix machine You the rename can move the file between directories provided both directories are in the same filing system. Under those conditions rename is always an atomic operation with no copying involved. This would you to, say, upload the file to "temp/ myfile" and renamed it to "uploaded/myfile" with your parser only scanning the uploaded directory and, presumably, renaming processed files to move them to a third directory ready for further processing. I've used this technique reliably with files arriving via FTP at quite high rates. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Nothing to repeat
On Sun, 09 Jan 2011 16:49:35 +, Tom Anderson wrote:
>
> Any thoughts on what i should do? Do i have to bite the bullet and apply
> some cleverness in my pattern generation to avoid situations like this?
>
This sort of works:
import re
f = open("test.txt")
p = re.compile("(spam*)*")
for line in f:
print "input line: %s" % (line.strip())
for m in p.findall(line):
if m != "":
print "==> %s" % (m)
when I feed it
===test.txt===
a line with no match
spa should match
spam should match
so should all of spaspamspammspammm
and so should all of spa spam spamm spammm
no match again.
===test.txt===
it produces:
input line: a line with no match
input line: spa should match
==> spa
input line: spam should match
==> spam
input line: so should all of spaspamspammspammm
==> spammm
input line: and so should all of spa spam spamm spammm
==> spa
==> spam
==> spamm
==> spammm
input line: no match again.
so obviously there's a problem with greedy matching where there are no
separators between adjacent matching strings. I tried non-greedy
matching, e.g. r'(spam*?)*', but this was worse, so I'll be interested to
see how the real regex mavens do it.
--
martin@ | Martin Gregorie
gregorie. | Essex, UK
org |
--
http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: The good, the bad, and the ugly!
On Wed, 19 Jan 2011 12:45:22 -0800, Patty wrote: > - Original Message - > From: "geremy condra" To: > On Wed, Jan 19, 2011 at 10:22 AM, wrote: >> >> Now I think I understand a little better where you all are coming from >> -- I am a Unix person and I guess I expected to have to learn GUI's >> using whatever is provided for me by default. Which isn't a bad thing. >> And if I had to add additional software - and learn that - so be it. I >> am using a Windows XP system and a Windows 7 system presently. Some day >> I would like to switch out the Windows XP for Unix. > > Just dual boot, it isn't hard. > IME you'll find that networking a Windows box to an older, slower PC thats rescued from the attic will be much more useful than a single dual-boot arrangement. Linux will run at a usable speed on a PC with 512 MB RAM and an 866 MHz P3, though some things, such as logging in, will be slow with a graphical desktop (runlevel 5), but if it has more RAM or you run an X-server on another PC, which could be running Windows, you'll execute commands, including graphical ones - provided you have X.11 forwarding enabled, a lot faster. The Linux box can also be headless if you haven't a screen and keyboard to spare. In short, Linux will run well on a PC that can't run anything more recent than Win98 at an acceptable speed. It doesn't need a lot of disk either - anything more than 30 GB will do. However, an optical drive is needed for installation. You can install Fedora from a CD drive provided the box is networked so it can retrieve most of its packages over the net, but using a DVD drive would be easier for a first install. > True. I have a Compaq Presario that is so old hardware-wise that I > don't think it could handle Unix or Linux. > What speed and type of CPU does it use? How much RAM? What's about disk and optical drives? FWIW my house server is an IMB Netvista that is at least 10 years old - 866MHz P3, 512 GB RAM, LG DVD drive, new 160GB hdd and runs Fedora 13. It is a bit slow at runlevel 5 (graphical desktop) when driven from its own console, but I usually access it over the house net from a more modern Core Duo laptop that runs Fedora 14. The NetVista is more than adequate for web and RDBMS development (Apache and PostgreSQL) in Python or Java and very fast for C compilation. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Part of RFC 822 ignored by email module
On Thu, 20 Jan 2011 12:55:44 -0500, Bob Kline wrote:
> On 1/20/2011 12:23 PM, Carl Banks wrote:
>> On Jan 20, 7:08 am, Bob Kline wrote:
>>> I just noticed that the following passage in RFC 822:
>>>
>>> The process of moving from this folded multiple-line
>>> representation of a header field to its single line
>>> represen- tation is called "unfolding". Unfolding is
>>> accomplished by regarding CRLF immediately followed
>>> by a LWSP-char as equivalent to the LWSP-char.
>>>
>>> is not being honored by the email module. The following two
>>> invocations of message_from_string() should return the same value, but
>>> that's not what happens:
>>>
>>> >>> import email
>>> >>> email.message_from_string("Subject: blah").get('SUBJECT')
>>> 'blah'
>>> >>> email.message_from_string("Subject:\n blah").get('SUBJECT')
>>> ' blah'
>>>
>>> Note the space in front of the second value returned, but missing from
>>> the first. Can someone convince me that this is not a bug?
>> That's correct, according to my reading of RFC 822 (I doubt it's
>> changed so I didn't bother to look up what the latest RFC on that
>> subject is.)
>>
>> The RFC says that in a folded line the whitespace on the following line
>> is considered a part of the line.
>
> Thanks for responding. I think your interpretation of the RFC is the
> same is mine. What I'm saying is that by not returning the same value
> in the two cases above the module is not "regarding CRLF immediately
> followed by a LWSP-char as equivalent to the LWSP-char."
>
That's only a problem if your code cares about the composition of the
whitespace and this, IMO is incorrect behaviour. When the separator
between syntactic elements in a header is 'whitespace' it should not
matter what combination of newlines, tabs and spaces make up the
whitespace element.
--
martin@ | Martin Gregorie
gregorie. | Essex, UK
org |
--
http://mail.python.org/mailman/listinfo/python-list
Re: Part of RFC 822 ignored by email module
On Thu, 20 Jan 2011 16:25:52 -0500, Bob Kline wrote: > On 1/20/2011 3:48 PM, Martin Gregorie wrote: >> That's only a problem if your code cares about the composition of the >> whitespace and this, IMO is incorrect behaviour. When the separator >> between syntactic elements in a header is 'whitespace' it should not >> matter what combination of newlines, tabs and spaces make up the >> whitespace element. > > That would be true for what the RFC calls "structured" fields, but not > for the others (such as the Subject header). Subject text comparisons should work correctly if you were to split the subject text using the 'whitespace' definition and then reassemble it using a single space in place of each whitespace separator. Its either that or assuming that all MUAs use the same line length and all use a line split of "CRLF " - the whitespace that's needed to align the continuation with the test on the first subject line. Many MUAs will do that, but its unlikely that all will. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Part of RFC 822 ignored by email module
On Thu, 20 Jan 2011 17:58:36 -0500, Bob Kline wrote: > Thanks. I'm not sure everyone would agree that it's OK to collapse > multiple consecutive spaces into one, but I'm beginning to suspect that > those more concerned with preserving as much as possible of the original > message are in the minority. It sounds like my take-home distillation > from this thread is "yes, the module ignores what the spec says about > unfolding, but it doesn't matter." I guess I can live with that. > I've been doing stuff in this area with the JavaMail package, though not as yet in Python. I've learnt that you parse the headers you can extract values that work well for comparisons, as database keys, etc. but are not guaranteed to let you reconstitute the original header byte for byte. If preserving the message exactly as received the solution is to parse the message to extract the headers and MIME parts you need for the application to carry out its function, but keep the original, unparsed message so you can pass it on. The other gotcha is assuming that the MUA author read and understood the RFCs. Very many barely glanced at RFCs and/or misunderstood them. Consequently, if you use strict parsing you'll be surprised how many messages get rejected for having invalid headers or MIME headers. Fot instance, the mistakes some MUAs make when outputting To, CC and BCC headers with multiple addresses have to be seen to be believed. If the Python e-mail module lets you, set it to use lenient parsing. If this isn't an option you may well find yourself having to fix up messages before you can parse them successfully. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: client server socket interaction (inetd)
On Thu, 17 Feb 2011 08:14:36 -0800, Tim wrote: > Hi, I have an inetd service on freebsd that calls a program (daemon.py) > with which I want the remote user to communicate. I can call daemon.py > from the command line on the host machine and it works fine. > > What I don't understand is how to make my remote client script actually > communicate. If I'm understanding correctly, the code below just takes a > message and sends it to inetd and writes the stdout from the process to > the client. > > How can I modify the code to send a response back? > The code you've shown would appear to be doing what you've specified, though only you can know whether this is what you intended. Each time you run the client it: - connects to the server - sends a request - reads the response(s) - closes the socket and exits. If you run it a second time it should do the same again. Is this the case? An inetd server should be started when a connection request is received. It should read requests, sending a response to each request in turn, until the connection is closed, when it will be stopped by inetd. Without seeing the code for the server and the corresponding inetd configuration line its not possible to say more. BTW, I prefer xinetd to inetd - its configuration is much more modular and easier to understand. If freebsd supports xinetd it may make life easier if you use it rather than inetd. Here's the outline > of what I want to do: > (1) client sends the message to the server (client -> inetd -> > daemon.py), > (2) client receives output back from the server, (3) client user > responds to a question from the remote process (4) client continues to > receive output back. > > where 2-3-4 happen as needed by the remote process. Cries out for a > while loop doesn't it? I just don't know what to put in it. Currently I > just have steps 1 and 2 working. Client sends one message and gets all > output back. If server asks a question, processes deadlock with server > waiting and client unable to respond. thanks, > --Tim Arnold > > # imports, constants set > #def line_buffer(sock): > #code to yield the string response in #blocks of 1024 bytes > > def client(ip,port,message): > sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM) > sock.connect((ip,port)) > sock.send(message) > for line in line_buffer(sock): > print line > sock.close() > > if __name__ == '__main__': > message = ' '.join(sys.argv[1:]) ) > print 'working... %s %s' % (SERVER_IP,SERVER_PORT) > client(SERVER_IP,SERVER_PORT,message) print 'done.' -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: client server socket interaction (inetd)
On Thu, 17 Feb 2011 13:02:08 -0800, Tim wrote: > But. The server may encounter a problem > during the process and ask the user for more information like > 'abort/retry' or something like that. > Servers never ask the client for information: they are strictly request/ response handlers. To do what you're asking about, you'll need to change both the client and server: - the client must analyse every response and, if necessary, ask for more input from the user. IMO it would be best to design it so it can accept several commands, with 'quit' being one of them. Either that or all request/response pairs must be designed so that the client can always send a single request, output the result and exit no matter what the server returns. - the server must always return a response to every possible request or log a failure reason, preferably to the system log which is /var/log/messages in Linux, and die. The server must thoroughly validate requests and return an adequate error message if errors were found. I'd strongly suggest that every request generates a single (newline terminated?) response message because that makes error detection much easier. If the response is multi-line, send it as a single line, possibly as a comma-separated list, and let the client format it for display I tend to precede every request and response message with a fixed length byte count: this way the recipient process always knows how many bytes to read and can report errors if the received length is wrong. Using an 'end of message marker' (NEWLINE unless a message can legally contain internal newlines) can also be useful. I tend to put an error flag in the response message because this simplifies the client. I also design all messages to be plain ASCII because this makes debugging a lot easier. If things really turn to worms you can use wireshark to watch the traffic and read it straight off the screen without needing to decode binary values, etc. So, an invalid request and response might look like this: Request: "0013,WHOIS,elvis\n" Response: "0030,ERROR,Unknown request: WHOIS\n" > The inetd configuration is: > myservice stream tcp nowait root /local/daemon.py daemon.py > >From what I recall about inetd that should be OK - I guess it must be since you can get one request through, and I assume (you didn't say) that a second request is also accepted without restarting the server. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: client server socket interaction (inetd)
On Fri, 18 Feb 2011 07:23:35 -0800, Tim wrote: > Thanks for helping me to understand. I don't know if you're familiar > with LaTeX, but that's part of what daemon.py calls via subprocess, and > that is the underlying process I wanted the user to be able to interact > with. > I've used nroff but never LaTeX - however, my understanding is that its essentially a batch process. > When LaTeX encounters a problem it stops processing, asks the user what > to do (like abort/retry, kind-of), and does whatever the user says. The > daemon.py script handles that okay from the command line, but if I'm > understanding you this will be much harder if not impossible to > accomplish from a client communicating over a socket. > Are you sure it wouldn't be better to end the run, returning suitable error messages to the user, who would correct the input file(s) and rerun LaTex? > I thought that maybe I needed to fork the socket (somehow) so it could > listen after the message is sent, but it looks like it will be more > complex. > The easier way my be to give the user a login to a captive shell that's restricted to running LateX, an editor and little else apart from a results viewer. Bash and most decent editors can be restricted so the user can't start a second, less restricted shell, and the range of programs it can execute can be restricted by adjusting with $PATH. You may also be able to find a menu program that can impose many of those restrictions by replacing the login's shell by the menu program. I do this as a matter of course: my menu program may get released one of these days but it's not currently on a fit state to publish. If you want to continue down the server path you should investigate popen. It is part of the os module. You could design the server so a suitable request runs LaTeX via popen, captures the stdout and stderr output, if any, and returns that and the LaTeX termination status code to the client as a response message. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: An amazing one-minute bit of fun at the interactive prompt
On Sun, 20 Feb 2011 22:42:17 +1100, Ben Finney wrote: > Raymond Hettinger writes: > >> Compute π ± e by counting Mandlebrot set iterations :-) > > Very cool! I love π nerdery. > > > Raymond Hettinger writes: > >> > Compute ð ± e by counting Mandlebrot set iterations :-) >> >> That should be: pi plus-or-minus e > > It was in my reader. Perhaps your server has encoding trouble? Same here (Pan reader, Fedora 14). -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
pyodbc problem
I have a small problem: I can't get pyodbc to connect to a PostgreSQL
database. All it does is spit out a malformed error message. When I run
this:
==
import pyodbc
dsn = pyodbc.dataSources()
print "Data sources:%s" % dsn
conn = pyodbc.connect(dsn="marchive")
print "ODBC connection: %s" % conn
==
I get this:
==
[k...@zappa python]$ python dsncheck.py
Data sources:{'ma': 'PostgreSQL', 'marchive': 'PostgreSQL'}
Traceback (most recent call last):
File "dsncheck.py", line 6, in
conn = pyodbc.connect(dsn="marchive")
pyodbc.Error: ('0', '[0] [unixODBC]c (0) (SQLDriverConnectW)')
==
so the data source exists and is evidently found by connect(), whiuch
seems unable to make sense of it.
The pgsql module works well, but I would prefer to use ODBC because its
not so closely bound to a single RDBMS. unixODBC, which I understand
underlies pyodbc, works OK too:
==
[k...@zappa python]$ isql marchive marchive n/a
+---+
| Connected!|
| |
| sql-statement |
| help [tablename] |
| quit |
| |
+---+
SQL> select count(*) from address;
+-+
| count |
+-+
| 32 |
+-+
SQLRowCount returns 1
1 rows fetched
SQL> quit
==
Questions:
- Why does pyodbc blow up when its apparently trying to talk to unixODBC?
- What does this mean:
pyodbc.Error: ('0', '[0] [unixODBC]c (0) (SQLDriverConnectW)')
I'm new to Python, though not to Linux, PostgreSQL, ODBC, C or Java.
My rig and software:
Linux: Fedora 12 Where I'm running Python and ODBC,
Lenovo Thinkpad R61i: Core Duo.
Fedora 10 Where PostgreSQL is installed,
IBM NetVista: Pentium III.
PostgreSQL: 8.3.8
Python: 2.6.2
pyodbc 2.1.5-3.fc12 ) By yum from RH repository
unixODBC2.2.14-11.fc12 )
--
martin@ | Martin Gregorie
gregorie. | Essex, UK
org |
--
http://mail.python.org/mailman/listinfo/python-list
Re: Why is there no platform independent way of clearing a terminal?
On Wed, 28 Jul 2010 09:01:38 -0700, Jonathan Hartley wrote: > Also, is it crazy to imagine that if colorama was pushed through to > completion (ie. to support a majority of the relevant ANSI codes) then > Python's stdlib curses module, unmodified, would suddenly just work on > Windows? (after a call to 'colorama.init()') > Curses was originally built on top of termcap, a much simpler and more primitive library. Its abilities are: - read the termcap database to find and load the capability list of the terminal identified as the TERM environment variable. - write a named capability to the screen. If the capability isn't defined for the current terminal its silently ignored. - fill in x, y and write the cursor movement string - thats about it. Its easy enough to find termcap databases. They're just text files containing a block of attribute values for each terminal, normally including ANSI terminals, Linux consoles, X-terms etc. They are also fairly easy to parse, even in vanilla C, so parsing one on Python should be a breeze. Termcap format specs and lists of standard attribute names are easy enough to find too. IOW, if you extend colorama to read its codes from a termcap database you'll have a basic but easily portable character console handler that can at least clear the screen, move the cursor and, if the screen has the capability, change the colour of characters or make them bold/dim/blink etc. To give you an idea of how complex this is, I've re-implemented termcap in Java for my own purposes. It amounts to 1100 lines of fairly well spaced and commented Java or about 340 statements, which were estimated by counting lines containing semicolons. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Why is python not written in C++ ?
On Mon, 02 Aug 2010 15:54:52 -0700, sturlamolden wrote: > On 3 Aug, 00:27, Paul Rubin wrote: > >> Certain folks in the functional-programming community consider OO to be >> a 1980's or 1990's approach that didn't work out, and that what it was >> really trying to supply was polymorphism. C++ programs these days >> apparently tend to use template-based generics rather than objects and >> inheritance for that purpose. > > It avoids virtual function calls at the expense of unreable code and > errors that are nearly impossible to trace. It seems many thinks this is > a good idea because Microsoft did this with ATL and WTL. There are also > those who thinks template metaprogramming is a good idea. But who uses a > C++ compiler to dumb to unroll a for loop? In my experience, trying to > outsmart a modern compiler is almost always a bad idea. > >> I have the impression that Ada has an undeservedly bad rap because of >> its early implementations and its origins in military bureaucracy. > > It is annyingly verbose, reminds me of Pascal (I hate the looks of it), > and is rumoured to produce slow bloatware. > And don't forget Ariane 5 ;) > This had nothing to do with Ada per se and everything to do with shortcuts: - re-use of an Ariane 4 component without changing its parameters to match the Ariane 5 flight profile - not removing Ariane 4-only code intended to handle launch holds and not used at all in Ariane 5 - a programming fault that allowed an exception code to be sent to another component as if it was valid data. Inadequate testing allowed all these to be flown without finding the errors. Bottom line: All this would still have happened regardless of the programming language used. However, don't just listen to me: read the final report on Ariane 501 here: http://www.di.unito.it/~damiani/ariane5rep.html -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Python "why" questions
On Fri, 13 Aug 2010 19:14:44 +0200, Thomas Jollans wrote:
> "Where it all started" is that 0-based indexing gives languages like C a
> very nice property: a[i] and *(a+i) are equivalent in C. From a language
> design viewpoint, I think that's quite a strong argument. Languages
> based directly on C (C++, Objective C, ...) can't break with this for
> obvious reasons, and other language designers/implementers imitated this
> behaviour without any good reason to do so, or not to do so. In
> higher-level languages, it doesn't really matter. 1-based indexing might
> seam more intuitive.
>
In a higher level language 1-based indexing is just as limiting as 0-
based indexing. What you really want is the ability to declare the index
range to suit the problem: in Algol 60 it is very useful to be able to
declare something like:
real sample[-500:750];
and Algol 68 went even further:
flex [1:0] int count
where the array bounds change dynamically with each assignment to
'count'. Iteration is supported by the lwb and upb operators which return
the bounds of an array, so you can write:
for i from lwb count to upb count do
--
martin@ | Martin Gregorie
gregorie. | Essex, UK
org |
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python "why" questions
On Mon, 16 Aug 2010 12:33:51 +1200, Gregory Ewing wrote:
> Ian Kelly wrote:
>> On Fri, Aug 13, 2010 at 11:53 AM, Martin Gregorie
>> wrote:
>
>>> real sample[-500:750];
>
>> Ugh, no. The ability to change the minimum index is evil.
>
> Not always; it can have its uses, particularly when you're using the
> array as a mapping rather than a collection.
>
Say you have intensity data captured from an X-ray goniometer from 160
degrees to 30 degrees at 0.01 degree resolution. Which is most evil of
the following?
1) real intensity[16000:3000]
for i from lwb intensity to upb intensity
plot(i/100, intensity[i]);
2) double angle[13000];
double intensity[13000];
for (int i = 0; i < 13000; i++)
plot(angle[i], intensity[i]);
3) struct
{
double angle;
double intensity
} measurement;
measurement m[13000];
for (int i = 0; i < 13000; i++)
plot(m[i].angle, m[i].intensity);
4) double intensity[13000];
for (int i = 0; i < 13000; i++)
plot((16000 - i)/100, intensity[i])
To my mind (1) is much clearer to read and far less error-prone to write,
while zero-based indexing forces you to use code like (2), (3) or (4),
all of which obscure rather than clarify what the program is doing.
--
martin@ | Martin Gregorie
gregorie. | Essex, UK
org |
--
http://mail.python.org/mailman/listinfo/python-list
Re: 79 chars or more?
On Tue, 17 Aug 2010 13:00:51 +1000, James Mills wrote: > Roy, under normal circumstances I would agree with you and have a > different opinion. However being vision impaired restricts the available > width (irregardless of the width of the monitor) of text I'm able to > view at once. > I'm with James here because: 1) ssh terminal windows generally come up as 24 x 80 2) at 24 x 80 I can get more ssh terminal windows on the desktop with minimal overlap than I can do with longer/wider windows. BTW, James, would it be a bore to add a space after the two hyphens at the top of your sig, i.e. use "-- ", not "--"? That way most news readers will automatically remove your sig when replying to your post. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Python "why" questions
On Tue, 17 Aug 2010 10:22:27 -0500, Robert Kern wrote: > On 8/16/10 11:10 PM, Steven D'Aprano wrote: >> On Mon, 16 Aug 2010 22:56:20 -0500, Robert Kern wrote: >> >>> On 8/16/10 9:29 PM, Roy Smith wrote: >>>> In article, >>>>Lawrence D'Oliveiro wrote: >>>> >>>>> In message, Roy Smith >>>>> wrote: >>>>> >>>>>> 5) real intensity[160.0 : 30.0 : 0.01] >>>>> >>>>> How many elements in that array? >>>>> >>>>> a) 2999 >>>>> b) 3000 >>>>> c) neither of the above >>>> >>>> c) neither of the above. More specifically, 13,001 (if I counted >>>> correctly). >>> >>> 13000, actually. Floating point is a bitch. >>> >>> [~/Movies] >>> |1> import numpy >>> >>> [~/Movies] >>> |2> len(numpy.r_[160.0:30.0:-0.01]) >>> 13000 >> >> >> Actually, the answer is 0, not 13000, because the step size is given as >> 0.01, not -0.01. >> >>>>> import numpy >>>>> len(numpy.r_[160.0:30.0:-0.01]) >> 13000 >>>>> len(numpy.r_[160.0:30.0:0.01]) >> 0 > > Roy wasn't using numpy/Python semantics but made-up semantics (following > Martin Gregorie's made-up semantics to which he was replying) which > treat the step size as a true size, not a size and direction. The > direction is determined from the start and stop parameters. It's an > almost-reasonable design. That wasn't a made-up example: AFAICR and ignoring a missing semi-colon it was an Algol 68 snippet. The semantics of the for statement and the use of lwb and upb operators to extract the bounds from a 1-dimensional array are correct A68, but OTOH its a very long time since I last programmed in that language. I used that rather than Python because Algol 68 supports the non-zero lower bound and treats the array limits as attributes of the array. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Iterative vs. Recursive coding
On Thu, 19 Aug 2010 14:12:44 -0700, Baba wrote: > Level: Beginner > > exercise source: > http://ocw.mit.edu/courses/electrical-engineering-and-computer- science/6-00-introduction-to-computer-science-and-programming-fall-2008/ assignments/pset3.pdf > > I am looking at the first problem in the above assignment. The > assignemnt deals, amongst others, with the ideas of iterative vs. > recursive coding approaches and i was wondering what are the advantages > of each and how to best chose between both options? > > I had a go at the first part of the exercise and it seems that i can > handle it. However i think my Recursive version can be improved. By the > way, is my code actually proper recursive code? > No, it isn't. By way of a hint, here are two versions of the classic example of recursion: calculating factorials. Recursion can be quite a trick to get your mind round at first, so compare the two and follow through their operation step by step... - def i_factorial(n): "Iterative factorial calculation" f = 1; for n in range(1, n+1): f = f * n return f def r_factorial(n): "Recursive factorial calculation" if (n > 1): return n * r_factorial(n - 1) else: return 1 print i_factorial.__doc__ for n in range(1,10): print "%d! = %d" % (n, i_factorial(n)) print r_factorial.__doc__ for n in range(1,10): print "%d! = %d" % (n, r_factorial(n)) - In case you're wondering "print i_factorial.__doc__" prints the docstring out of the i_factorial subroutine. You probably haven't covered that yet, but run it and see. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: A question to experienced Pythoneers
On Fri, 20 Aug 2010 17:16:14 +0200, Bruno Desthuilliers wrote: > Rony a écrit : >> It looks like I forgot to specify that the product is a totaly new >> product build from scratch, not an upgrade from an existing product. > > > Still the advice to first find out what went wrong with the previous > project is a very sensible one. Technical problems do exist, but from > experience they rarely are the main source of failure. > And be sure to look into the estimation and costing methods the company is using. IOW are you sure whether design/build/test phases are overrunning a sensible estimate or if the initial costing and estimation is underestimating the project size? -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Iterative vs. Recursive coding
On Fri, 20 Aug 2010 16:22:44 -0700, Baba wrote:
> For the purposes of learning programming i think it's a must to
> understand Recursion so thanks all for your help!
>
That depends on the language and/or hardware. COBOL wouldn't understand
recursion if hit on the head with a recursion brick and early computer
hardware (those without a stack) made it VERY hard work. If you don't
follow this, look at the CODASYL language specification for COBOL or the
hardware design of ICL 1900 or IBM System/360 mainframes (which are still
the heart of the financial world) and work out how to implement a
recursive function for any of them. Its not easy but it can be done.
> Ok so now i hope you all agree that my code is also correct :)
>
Yes: it runs and does what I'd expect. A good result.
A basic skill for a programmer is to understand the specification for a
piece of code and write test cases. This is a set of inputs (both
expected/sensible and totally out of order) and the expected outputs from
each set of inputs. Then you write the code and run it against the test
cases to show that it does what the specification requires.
Never bullshit yourself or anybody else about this conformance to spec:
either you screwed up or, hopefully less often, the designer wrote an
ambiguous or plain wrong requirement. Either way, the problem must be
resolved and the code be made to do what is wanted.
while (results don't match the spec):
Rince, wash, repeat.
--
martin@ | Martin Gregorie
gregorie. | Essex, UK
org |
--
http://mail.python.org/mailman/listinfo/python-list
Re: [Tutor] Arguments from the command line
On Wed, 08 Sep 2010 21:58:49 -0700, Dennis Lee Bieber wrote: > On Thu, 09 Sep 2010 12:38:04 +1200, Lawrence D'Oliveiro > declaimed the following in > gmane.comp.python.general: > >> In message , Hugo >> Arts wrote: >> >> > sys.argv is a list of all arguments from the command line ... >> >> Interesting that Python didnt bother to mimic the underlying POSIX >> convention of passing the command line as arguments to the mainline >> routine. >> > What "mainline routine"... The only programming language(s) I've > ever used that requires there be something called "main" in order to > start a program is the C/C++ family. > Java uses a method defined as "public static void main(String[] args)" > My college COBOL never used multifile assignments, so I'm not sure > if there was a difference between main and linked modules. > There isn't, but command line argument passing is implementation- dependent and is complicated by the ability to define callable sub- programs in the same source file as the main program. The most general method is to use ACCEPT statements. MicroFocus COBOL uses "ACCEPT ... FROM ARGUMENT-NUMBER", AIX COBOL uses a special system call and ICL 2900 COBOL and IBM COBOL/400, where the command line uses function call notation, map the command line arguments into a LINKAGE SECTION. In short: this area of COBOL is a mess. PL/I specifies the main procedure with an OPTIONS(MAIN) clause and declares the integer ARGC_ and pointer ARGV_ variables in it, which are used like their C equivalents. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Ugh! Python 3.1.x and MySQL
On Fri, 10 Sep 2010 09:27:28 -0700, fuglyducky wrote: > Most of the python books coming out now are Py3K. I just started > programming and have a need to access a MySQL database. I would like to > use Python to do this. Unfortunately, I cannot find anyone that has > created anything that allows my to do this. > > I've tried installing an ODBC driver and using sqlalchemy, oursql, and a > few other things with no luck. > > So...just wondering if anyone is aware of any libraries/modules that I > can use to connect to a MySQL DB using Python 3.1.x? > > Ideally, I'd like to be able to this from both x86 and x64 systems (if > that makes any difference). > You don't say what OS you're using, but if you're on a *NIX, take a look at pyodbc: http://code.google.com/p/pyodbc/ This Python module is a wrapper for unixODBC and consequently works with standard MySQL ODBC drivers. iODBC: http://www.iodbc.org/ is similar If you want something that's Windows compatible I can't help: I don't use either Windows or MySQL. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: suggestions please "what should i watch for/guard against' in a file upload situation?"
On Wed, 06 Oct 2010 09:02:21 -0700, geekbuntu wrote: > in general, what are things i would want to 'watch for/guard against' in > a file upload situation? > > i have my file upload working (in the self-made framework @ work without > any concession for multipart form uploads), but was told to make sure > it's cleansed and cannot do any harm inside the system. > Off the top of my head, and assuming that you get passed the exact filename that the user entered: - The user may need to use an absolute pathname to upload a file that isn't in his current directory, so retain only the basename by discarding the rightmost slash and everything to the left of it: /home/auser/photos/my_photo.jpg ===> my_photo.jpg c:\My Photos\My Photo.jpg ===> My Photo.jpg - If your target system doesn't like spaces in names or you want to be on the safe side there, replace spaces in the name with underscores: My Photo.jpg ===>My_Photo.jpg - reject any filenames that could cause the receiving system to do dangerous things, e.g. .EXE or .SCR if the upload target is Windows. This list will be different for each upload target, so make it configurable. You can't assume anything about else about the extension. .py .c .txt and .html are all valid in the operating systems I use and so are their capitalised equivalents. - check whether the file already exists. You need rules about what to do if it exists (do you reject the upload, silently overwrite, or alter the name, e.g. by adding a numeric suffix to make the name unique: my_photo.jpg ===> my_photo-01.jpg - run the application in your upload target directory and put the uploaded file there or, better, into a configured uploads directory by prepending it to the file name: my_photo.jpg ===> /home/upload_user/uploads/my_photo.jpg - make sure you document the process so that a user can work out what has happened to his file and why if you have to reject it or alter its name. > not sure but any suggestions or examples are most welcome :) > There's probably something I've forgotten, but that list should get you going. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Simple database explorer
On Fri, 08 Oct 2010 02:04:39 -0700, dusans wrote: > On Oct 7, 7:49 am, Lawrence D'Oliveiro central.gen.new_zealand> wrote: >> In message >> <21c99273-ed58-4f93-b98a-d9292de5d...@k10g2000yqa.googlegroups.com>, >> dusans wrote: >> >> > - all the others having ODBC drivers... >> >> ODBC seems to be something you use when you can’t use a proper database >> driver. > Its the right answer for a program that needs to be used with many different RDBMSes, especially if you use its metadata access procedures. > yes. first i have been mixing native and odbc drivers. Now i only use > odbc. Odbc is not the best but its good enought for a sql explorer. Dont > know how its on unix thou > Use unixODBC. Its a wrapper that lets Windows ODBC drivers be run on UNIX/ Linux and there's a Python module that provides an interface to unixODBC and a Data Manager utility that configures ODBC data sources. The documentation for the module is poor but the module and utility both work well once you've figured out how to use them. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: "Strong typing vs. strong testing"
On Sun, 10 Oct 2010 21:38:11 +1100, Lie Ryan wrote: > > Virtual Machine in Hardware... isn't that a contradiction? > Nope. Several mainframes did that. Two that I knew well were both British - the ICL 1900 and 2900. The Burroughs x700 series also used hardware virtualisation. Both Burroughs and 2900 series of machines could even run different addressing and virtual hardware architectures, e.g. word-addressing vs. byte addressing, in each virtual machine. The 1900's 'hardware' registers, PC, CCR, etc were the first few words in each program's address space. That was late 60s tech, so scarcely a new concept nowadays. The ICL 2900 and, by extension Multics, since the 2900 hardware's process protection scheme was swiped from Multics, used a different approach - again hardware-implemented virtual per-process registers but in addition 'rings of protection' which meant the OS could affect your code but you couldn't touch it and the OS in turn couldn't touch the hardware drivers etc. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: PEP 249 (database api) -- executemany() with iterable?
On Thu, 14 Oct 2010 16:36:34 +1300, Lawrence D'Oliveiro wrote: > In message <4cb5e659$0$1650$742ec...@news.sonic.net>, John Nagle wrote: > >> Also note that there are some issues with doing a huge volume of >> updates in one MySQL InnoDB transaction. The system has to keep the >> data needed to undo the updates, and there's a limit on the amount of >> pending transaction history that can be stored. > > How does “load data” avoid this? Is that not a transaction too? > Not usually. Its faster because there's no journalling overhead. The loader takes out an exclusive table lock, dumps the data into the table, rebuilds indexes and releases the lock. I can't comment about MySQL (don't use it) but this has been the case on the RDBMS databases I have used. > Seems to me this isn’t going to help, since both old and new tables are > on the same disk, after all. And it’s the disk access that’s the > bottleneck. > There's a lot of overhead in journalling - much more than in applying changes to a table. The before and after images *must* be flushed to disk on commit. In UNIX terms fsync() must be called on the journal file(s) and this is an expensive operation on all OSes because committing a series of small transactions can cause the same disk block to be written several times. However, the table pages can safely be left in the DBMS cache and flushed as part of normal cache operation since, after a crash, the table changes can always be recovered from a journal roll-forward. A good DBMS will do that automatically when its restarted. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: EOF while scanning triple-quoted string literal
On Fri, 15 Oct 2010 17:02:07 +, Grant Edwards wrote: > On 2010-10-15, Steven D'Aprano > wrote: > >> In the Unix world, which includes OS X, text tools tend to have >> difficulty with tabs. Or try naming a file with a newline or carriage >> return in the file name, or a NULL byte. > > How do you create a file with a name that contains a NULL byte? Use a language or program that doesn't use null-terminated strings. Its quite easy in many BASICs, which often delimit strings by preceeding it with a with a byte count, and you hit Ctrl-SPACE by accident -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: EOF while scanning triple-quoted string literal
On Fri, 15 Oct 2010 18:14:13 +, Grant Edwards wrote: > On 2010-10-15, Martin Gregorie wrote: >> On Fri, 15 Oct 2010 17:02:07 +, Grant Edwards wrote: >> >>> On 2010-10-15, Steven D'Aprano >>> wrote: >>> >>>> In the Unix world, which includes OS X, text tools tend to have >>>> difficulty with tabs. Or try naming a file with a newline or carriage >>>> return in the file name, or a NULL byte. >>> >>> How do you create a file with a name that contains a NULL byte? >> >> Use a language or program that doesn't use null-terminated strings. >> >> Its quite easy in many BASICs, which often delimit strings by >> preceeding it with a with a byte count, and you hit Ctrl-SPACE by >> accident > > I don't see what the in-program string representation has to do with it. > The Unix system calls that create files only accept NULL terminated > strings for the path parameter. > Well, obviously you can't have null in a filename if the program is using null-terminated strings. > Are you saying that there are BASIC implementations for Unix that create > Unix files by directly accessing the disk rather than using the Unix > system calls? > I'm saying that the only BASIC implementations I've looked at the guts of have used count-delimited strings. None were on *nixen but its a safe bet that if they were ported to a UNIX they'd retain their count-delimited nature. Another language that will certainly do this is COBOL, which only uses fixed length, and therefore undelimited, strings. The point I'm making is that in both fixed length and counted string representations you can put any character value at all into the string unless whatever mechanism you're using to read in the values recognises something, i.e. TAB, CR, LF, CRLF as a delimiter, and even then the program can generate a string containing arbitrary gibberish. If you then use the string as a file name you can end up with a file that can't be accessed or deleted if the name flouts the OS's file naming conventions. I've done it in the past with BASIC programs and finger trouble under FLEX09 and CP/M. In both cases I had to use a disk editor to fix the file name before the file could be deleted or accessed. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: EOF while scanning triple-quoted string literal
On Fri, 15 Oct 2010 19:59:13 +, Grant Edwards wrote: > > We're talking about Unix. > We're not talking about CP/M, DOS, RSX-11m, Apple-SOS, etc. > That's just your assumption. Track back up the thread and you'll see that the OP didn't mention an OS. He merely said that he was using zlib, and getting unfortunate results when he handled its output, so he could have been using any OS. Rhodri James assumed that the OS was Windows, but it wasn't until the 6th post that Steven D'Aprano mentioned Unix and null characters. I got sucked into the null trap - sorry - because I actually meant to generalise the discussion into ways of getting a range of unwanted characters into a file name and why its unwise to use a filename without checking it for characters the OS doesn't like before creating a file with it. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: PEP 249 (database api) -- executemany() with iterable?
On Sun, 17 Oct 2010 11:27:17 +1300, Lawrence D'Oliveiro wrote: > In message , Martin Gregorie wrote: > >> On Thu, 14 Oct 2010 16:36:34 +1300, Lawrence D'Oliveiro wrote: >> >>> In message <4cb5e659$0$1650$742ec...@news.sonic.net>, John Nagle >>> wrote: >>> >>>> Also note that there are some issues with doing a huge volume of >>>> updates in one MySQL InnoDB transaction. The system has to keep the >>>> data needed to undo the updates, and there's a limit on the amount of >>>> pending transaction history that can be stored. >>> >>> How does “load data” avoid this? Is that not a transaction too? >>> >> Not usually. Its faster because there's no journalling overhead. > > So what happens if there’s an error part way? Do you end up with some > records inserted and others not? > Since the load is usually used to populate an empty table, you simply empty the table and do it again if it doesn't complete for some reason. If you're doing a partial load you'd normally dump the table and/or take a backup beforehand. >> The loader takes out an exclusive table lock, dumps the data into the >> table, rebuilds indexes and releases the lock. > > That’s not acceptable in a situation where other processes are > concurrently making queries on the table. > Obviously, but you're unlikely to be running anything while you're doing a bulk load. If the database is used to support interactive processing you'd normally only use bulk loading during recovery after a disk failure or as part of a database migration or restructuring exercise. In fact the only case I know of where bulk loads are a normal part of the data life cycle is for data warehousing, where bulk loads are the norm. The last month's transactions are removed from the online system and bulk loaded into the data warehouse for analysis. In this case the bulk loads are either done overnight or during a weekend. However, any RDBMS that has been optimised for data warehousing will almost certainly have the ability to segment its fact table. This has two benefits: - you can bulk load a new segment offline and put it online when the load is complete. This can be done during normal operation. - the DBMS can automatically split a query and run parallel copies against each segment before combining the result sets for return to the program that made the query. >>> Seems to me this isn’t going to help, since both old and new tables >>> are on the same disk, after all. And it’s the disk access that’s the >>> bottleneck. >>> >> There's a lot of overhead in journalling - much more than in applying >> changes to a table. The before and after images *must* be flushed to >> disk on commit. In UNIX terms fsync() must be called on the journal >> file(s) and this is an expensive operation on all OSes because >> committing a series of small transactions can cause the same disk block >> to be written several times. However, the table pages can safely be >> left in the DBMS cache and flushed as part of normal cache operation >> since, after a crash, the table changes can always be recovered from a >> journal roll-forward. A good DBMS will do that automatically when its >> restarted. > > And how does this address the point of whether to load the new records > into a new table or the same table? > That's system design choice. There are a limited set of options and its up to the designer to choose one: - if you want to load data into a new table and rename CURRENT->OLD, NEW->CURRENT you have to design all queries to operate on both OLD and CURRENT tables - if your DBMS allows table segmentation and gives the ability to offline segments you can bulk load as I already described. - you can stop the system during a quiet time while you do a bulk load and/or minimise overheads by using very large transaction. - you can bite the bullet, accept the performance hit and simply run a relatively slow load process using small transactions during normal operation. There are different overheads associated with each choice. A good system designer will understand them, know their relative weight for the target hardware and database, and use this information to optimise the design. What I said about transactional and journalling overheads was explanation: unless you know that you'll never understand why you should expect bulk loading to be a lot faster than doing it with transactions designed to logically group related operations or (much, much worse) to use implied commits. This really is the worse of all worlds since it maximises journalling overhead. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: OO and game design questions
On Tue, 19 Oct 2010 19:49:20 -0400, Dave Angel wrote: > Thanks, that is what I was trying to say. In the same sense that > emptying a list makes it quite small, if it's a general purpose object, > you just want to remove all the attributes. > I think a 'place' (to generalise it) is quite a small object in any case. All it needs to contain is a few lists: - a list of exits, which in some implementations might be simply references to other places, but might usefully be objects with two sides, each having an appearance and a link to the place where that side appears. - a list of fixed objects which only serve to describe the place. - a list of mobile objects that actors can pick up and move - a list of actors who happen to be there. plus a describePlace() method and add(), remove() and getItem() methods for each list. It may be possible to use a single list for all types of object, in which case the object itself would be very small indeed. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Print to an IPP printer (pkipplib?)
On Fri, 15 Oct 2010 15:28:10 -0400, Adam Tauno Williams wrote: > I've found the module pkipplib which seems to work well for things like > interrogating an IPP (CUPS) server. But is there a way to send a print > job to an IPP print queue? [and no, the local system knows nothing about > the print architecture so popenlp is not an option]. I just want to > send the data from a file handle to a remote IPP queue as a print job. See RFC 2910. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Unix-head needs to Windows-ize his Python script (II)
On Tue, 26 Oct 2010 02:38:43 -0400, Steve Holden wrote: > > I really don't understand what you are trying to say here. Could you > please explain? I know you to be a capable and sensible person, but this > sounds like nonsense to me, so I must be misunderstanding. > I think he's saying that on a Linux desktop, if you define a launcher for an application the default assumption is that its a graphical application. If so, all you need to do is to tell the launcher the program name, what icon to use and what text to put under it. If the application isn't graphical, you do the same as above and also tell the launcher that the program must run in a console window. Simple. Logical. Concise. I assume that what I've just described applies to OS X and virtually all other graphical desktops: I wouldn't know from personal experience because I don't use them. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: sed equivalent of something easy in python
On Wed, 27 Oct 2010 03:28:16 -0700, Lie wrote: > On Oct 26, 4:04 am, Daniel Fetchinson wrote: > >> (2) made me take care of 2 files instead of 1 from now on. > > Not necessarily: > > $ cat heredoc.sh > #!/usr/bin/env bash > python << 'EOF' > print "hello world" > def foo(): > print "foo()" > foo() > EOF > $ Or even better: $ cat hello #!/usr/bin/python print "hello world" def foo(): print "foo()" foo() $ chmod u+x hello $ hello hello world foo() $ which saves an unnecessary shell invocation. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: What people are using to access this mailing list
On Wed, 03 Nov 2010 13:05:23 +, Grant Edwards wrote: > On 2010-11-03, Paul Rudin wrote: >> John Bond writes: >> >>> On 3/11/2010 11:17 AM, Steven D'Aprano wrote: >>>> On Wed, 03 Nov 2010 08:02:29 +, John Bond wrote: >>>> >>>>> Hope this isn't too O/T - I was just wondering how people read/send >>>>> to this mailing list, eg. normal email client, gmane, some other >>>>> software or online service? >>>> Usenet via my ISP, on comp.lang.python. >>>> >>> Using what client (or web client)? >> >> Emacs, of course :-; > > Slrn, of course. Pan since I'm on Linux. Agent if I was still a Windows user. Its the best news reader I've used. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Python documentation too difficult for beginners
On Thu, 04 Nov 2010 08:03:37 +1100, Cameron Simpson wrote: > On 02Nov2010 04:23, jk wrote: | This > (http://epydoc.sourceforge.net/stdlib/) is what I'm talking | about. > | > | Why aren't the official docs like this, and why has it taken me 2 days > | of searching? All this needs is a search engine behind it and it'd be > | perfect. > > It looks a lot like javadoc. But its weakness is stuff like this: > > http://epydoc.sourceforge.net/stdlib/Canvas.Polygon-class.html > > Automatic docness, no useful information. > You get the same problem in Java and it has exactly the same root: a lazy programmer who can't be arsed to document his code. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Commercial or Famous Applicattions.?
On Tue, 09 Nov 2010 00:47:08 +, brf256 wrote: > Mailman is of course. > ...and don't forget getmail, a better behaved replacement for fetchmail. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Pythonic/idiomatic?
On Wed, 10 Nov 2010 00:11:23 +, Seebs wrote: > On 2010-11-09, Jean-Michel Pichavant wrote: >> One pythonic way to do it, is to use an option parser. > > That seems like massive overkill -- I don't care about any of the other > options. It seems like it'd result in doing more work to get and then > extract the options, and most of that would be discarded instnatly. > I've always used an extended version of getopt() in C. I was so surprised to see that there's nothing equivalent in Java that I wrote my own and was consequently was very pleased to find that Python already has the optparse module. Using it is a no-brainer, particularly as it makes quite a good fist of being self-documenting and of spitting out a nicely formatted chunk of help text when asked to do so. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Commercial or Famous Applicattions.?
On Wed, 10 Nov 2010 13:59:02 +, Nobody wrote: > On Wed, 10 Nov 2010 13:07:58 +0000, Martin Gregorie wrote: > >> FWIW the thing that really irritated me about fetchmail is the way it >> only deletes messages at the end of a session and never cleans up after >> itself. If a session gets timed out or otherwise interrupted the >> messages that were read in it are left in the server's mail marked >> 'read'. Subsequent sessions won't re-read them but won't go expunge >> them either. This leads to a gradual build-up of read but not expunged >> messages in the server's mailbox. > > The --flush option will delete any "seen" messages before retrieving new > messages. > ...and is described as dangerous, can cause message loss in the man page along with a warning to avoid including it as a permanent option - good enough reasons for not using it IMO. > The --fetchall option will retrieve both seen and unseen > messages. > Again, not useful as a permanent option since I don't want to receive duplicated messages. In both cases using them 'as required' requires: (a) monitoring the source mailbox for 'read' message build up (b) when 'read' messages are seen, executing a sequence of stop fetchmail alter configuration run it for one retrieval session change config back restart using the normal configuration That's far too much hassle to be part of SOP. Now, if ESR had fixed fetchmail to tidy up after itself (if getmail can do that, there's no reason why fetchmail can't) or had even added the ability for a daily or weekly cron job to enquire about 'read' messages and, if any are present, tell it to silently expunge them in a special session, I'd probably still use it. Without equivalent fixes its just a buggy bit of software, making getmail a superior replacement because it 'just works'. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: How to test if a module exists?
On Wed, 10 Nov 2010 12:04:01 +, Mark Wooding wrote: > Lawrence D'Oliveiro writes: > >> I see that you published my unobfuscated e-mail address on USENET for >> all to see. I obfuscated it for a reason, to keep the spammers away. >> I'm assuming this was a momentary lapse of judgement, for which I >> expect an apology. Otherwise, it becomes grounds for an abuse complaint >> to your ISP. > > Don't be silly! If anything, using forged email addresses in Usenet > headers is closer to abuse. Note that there's nothing to stop > `new_zealand' from becoming a TLD at some point (the underscore makes it > unlikely, I grant). If you'd mangled the local part, or used an > explicitly reserved TLD such as `example', then there wouldn't be a > problem. > Using the .invalid TLD is a generally accepted way of writing an intentionally fake address for use in USENET. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Commercial or Famous Applicattions.?
On Wed, 10 Nov 2010 17:01:05 +1300, Lawrence D'Oliveiro wrote: > In message , Martin Gregorie wrote: > >> ...and don't forget getmail, a better behaved replacement for >> fetchmail. > > I was just looking this up in the Getmail FAQ, since I didn’t know about > the issues with Fetchmail. > FWIW the thing that really irritated me about fetchmail is the way it only deletes messages at the end of a session and never cleans up after itself. If a session gets timed out or otherwise interrupted the messages that were read in it are left in the server's mail marked 'read'. Subsequent sessions won't re-read them but won't go expunge them either. This leads to a gradual build-up of read but not expunged messages in the server's mailbox. The only fix I discovered was to go in manually with an MUA just after a fetchmail session so it could read and expunge this junk. getmail doesn't cause this problem. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Commercial or Famous Applicattions.?
On Wed, 10 Nov 2010 16:58:06 -0800, alex23 wrote: > Martin Gregorie wrote: >> Now, if ESR had fixed fetchmail [...] > > Did you try submitting patches? Nope. I'd already seen comments that bug reports etc. are ignored and tried getmail. Since that does the needful, why should I bother? -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: is there an Python equivalent for the PHP super globals like $_POST, $_COOKIE ?
On Thu, 11 Nov 2010 13:01:01 -0500, Steve Holden wrote: > Moving from one language to anther is not just a matter of > transliterating the code. Of you try that you will end up with a messy > code base that looks like PHP written in Python. > "The determined Real Programmer can write Fortran programs in any language." - from "Real Programmers don't use Pascal". -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: need some debug-infos on a simple regex
On Fri, 12 Nov 2010 17:21:04 -0800, Martin Kaspar wrote: > hello dear list! > > i'm very new to programming and self teaching myself. I'm having a > problem with a little project. > This doesn't directly help with your problem, but the tool at this URL: http://www.solmetra.com/scripts/regex/ may be useful when you're experimenting with regexes or testing them. Perl regexes are similar enough to Python regexes for this tool to be useful here. Without examples of text that the regex is intended to match its difficult to say more. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
Re: Looking For Tutorial Comparison Of sh - perl - python
On Sat, 13 Nov 2010 22:05:57 -0600, Tim Daneliuk wrote: > I am putting together a comparison & intro to each of sh, perl, python > with a view towards addressing: > Add awk / gawk to that list. IME its often the easiest and most concise way to process a text file, e.g. a log file, while applying different processing rules depending on line content and/or generating summaries. > Before I attack this myself, has anyone done > something along these lines I could piggyback upon? > I haven't seen such a comparison, but that doesn't meant that they don't exist. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | -- http://mail.python.org/mailman/listinfo/python-list
