Changing the Image on a button
I want to be able to toggle if the button has an image or text. For some reason the following code gets the button to have an image but when i push the button i expect the image to go away but it does not. Am I missing something? from Tkinter import * def do(): btn.configure(image = None) root = Tk() img1 = PhotoImage(file="bacon.gif") btn = Button(image = img1, command = do, text = "hello" ) btn.img = img1 btn.pack() root.mainloop() -- http://mail.python.org/mailman/listinfo/python-list
Re: Pythonic way to determine if one char of many in a string
On Feb 15, 9:56 pm, Chris Rebert wrote: > On Sun, Feb 15, 2009 at 9:17 PM, wrote: > > I need to test strings to determine if one of a list of chars is in the > > string. A simple example would be to test strings to determine if they have > > a vowel (aeiouAEIOU) present. > > > I was hopeful that there was a built-in method that operated similar to > > startswith where I could pass a tuple of chars to be tested, but I could not > > find such a method. > > > Which of the following techniques is most Pythonic or are there better ways > > to perform this type of match? > > > # long and hard coded but short circuits as soon as match found > > if 'a' in word or 'e' in word or 'i' in word or 'u' in word or ... : > > > -OR- > > > # flexible, but no short circuit on first match > > if [ char for char in word if char in 'aeiouAEIOU' ]: > > Just use the fairly new builtin function any() to make it short-circuit: > > if any(char.lower() in 'aeiou' for char in word): > do_whatever() > > Cheers, > Chris > > -- > Follow the path of the Iguana...http://rebertia.com If you want to generalize it you should look at sets http://docs.python.org/library/sets.html It seems what you are actually testing for is if the intersection of the two sets is not empty where the first set is the characters in your word and the second set is the characters in your defined string. -- http://mail.python.org/mailman/listinfo/python-list
Re: Changing the Image on a button
On Feb 16, 8:40 am, John Posner wrote: > >> from Tkinter import * > >> > >> def do(): > >> btn.configure(image = None) > >> > >> root = Tk() > >> img1 = PhotoImage(file="bacon.gif") > >> > >> btn = Button(image = img1, command = do, text = "hello" ) > >> btn.img = img1 > >> btn.pack() > >> root.mainloop() > >> > > Try this change: > > from: btn.configure(image = None) > to: img1.blank() > > -John > > E-mail message checked by Spyware Doctor (6.0.0.386) > Database version: 5.11770http://www.pctools.com/en/spyware-doctor-antivirus/ This does in fact clear the image out, however it isn't causing the text to display... Do i have have to create a new button and swap it out? -- http://mail.python.org/mailman/listinfo/python-list
Re: Pythonic way to determine if one char of many in a string
On Feb 15, 11:31 pm, odeits wrote: > On Feb 15, 9:56 pm, Chris Rebert wrote: > > > > > On Sun, Feb 15, 2009 at 9:17 PM, wrote: > > > I need to test strings to determine if one of a list of chars is in the > > > string. A simple example would be to test strings to determine if they > > > have > > > a vowel (aeiouAEIOU) present. > > > > I was hopeful that there was a built-in method that operated similar to > > > startswith where I could pass a tuple of chars to be tested, but I could > > > not > > > find such a method. > > > > Which of the following techniques is most Pythonic or are there better > > > ways > > > to perform this type of match? > > > > # long and hard coded but short circuits as soon as match found > > > if 'a' in word or 'e' in word or 'i' in word or 'u' in word or ... : > > > > -OR- > > > > # flexible, but no short circuit on first match > > > if [ char for char in word if char in 'aeiouAEIOU' ]: > > > Just use the fairly new builtin function any() to make it short-circuit: > > > if any(char.lower() in 'aeiou' for char in word): > > do_whatever() > > > Cheers, > > Chris > > > -- > > Follow the path of the Iguana...http://rebertia.com > > If you want to generalize it you should look at > setshttp://docs.python.org/library/sets.html > > It seems what you are actually testing for is if the intersection of > the two sets is not empty where the first set is the characters in > your word and the second set is the characters in your defined string. To expand on what I was saying I thought i should provide a code snippet: WORD = 'g' * 100 WORD2 = 'g' * 50 + 'U' VOWELS = 'aeiouAEIOU' BIGWORD = 'g' * 1 + 'U' def set_test(vowels, word): vowels = set( iter(vowels)) letters = set( iter(word) ) if letters & vowels: return True else: return False with python 2.5 I got 1.30 usec/pass against the BIGWORD -- http://mail.python.org/mailman/listinfo/python-list
Re: Changing the Image on a button
On Feb 17, 7:14 am, John Posner wrote: > >> > Try this change: > >> > > >> > from: btn.configure(image = None) > >> > to: img1.blank() > >> > > >> > >> This does in fact clear the image out, however it isn't causing the > >> text to display... Do i have have to create a new button and swap it > >> out? > > I knew you were gonna ask that! :-) I haven't worked in this area before, > but I > found this athttp://effbot.org/tkinterbook/button.htm: > > In earlier versions of Tkinter, the image option overrides the text > option. > If you specify both, only the image is displayed. In later versions, you > can > use the compound option to change this behavior. To display text on top > of > an image, set compound to CENTER: > > b = Button(master, text="Click me", image=pattern, compound=CENTER) > > So here's a reworking, in which the text and image are both toggled by > pressing the button: > > ### button that toggles both text and image > from Tkinter import * > > def pushed(): > """callback: button push""" > > global toggle > > if toggle: > btn.config(text="", image=img_empty) > toggle = not toggle > else: > btn.config(text=msg, image=img_good) > toggle = not toggle > > ### main program > toggle = True > msg = "hello" > > root = Tk() > > ### store two versions of an image in global variables: > # 1. original image > img_good = PhotoImage(file="bacon.gif") > > # 2. blank image, created by copying and erasing > img_empty = img_good.copy() > img_empty.blank() > > ### create toggle button > btn = Button(root, compound=CENTER, > text="hello", font="helvetica 14 bold", > image=img_good, > command=pushed) > btn.pack() > > ### go > root.mainloop() > > E-mail message checked by Spyware Doctor (6.0.0.386) > Database version: 5.11780http://www.pctools.com/en/spyware-doctor-antivirus/ Thank you very much for your help! It is now behaving the way I wanted! -- http://mail.python.org/mailman/listinfo/python-list
Re: Pythonic way to determine if one char of many in a string
On Feb 21, 12:47 am, "Gabriel Genellina"
wrote:
> En Sat, 21 Feb 2009 01:14:02 -0200, odeits escribió:
>
>
>
> > On Feb 15, 11:31 pm, odeits wrote:
> >> It seems what you are actually testing for is if the intersection of
> >> the two sets is not empty where the first set is the characters in
> >> your word and the second set is the characters in your defined string.
>
> > To expand on what I was saying I thought i should provide a code
> > snippet:
>
> > WORD = 'g' * 100
> > WORD2 = 'g' * 50 + 'U'
> > VOWELS = 'aeiouAEIOU'
> > BIGWORD = 'g' * 1 + 'U'
>
> > def set_test(vowels, word):
>
> > vowels = set( iter(vowels))
> > letters = set( iter(word) )
>
> > if letters & vowels:
> > return True
> > else:
> > return False
>
> > with python 2.5 I got 1.30 usec/pass against the BIGWORD
>
> You could make it slightly faster by removing the iter() call: letters =
> set(word)
> And (if vowels are really constant) you could pre-build the vowels set.
>
> --
> Gabriel Genellina
set(word) = set{[word]} meaning a set with one element, the string
the call to iter makes it set of the letters making up the word.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Pythonic way to determine if one char of many in a string
On Feb 21, 2:24 pm, rdmur...@bitdance.com wrote:
> odeits wrote:
> > On Feb 21, 12:47=A0am, "Gabriel Genellina"
> > wrote:
> > > En Sat, 21 Feb 2009 01:14:02 -0200, odeits escribi=F3:
>
> > > > On Feb 15, 11:31=A0pm, odeits wrote:
> > > >> It seems what you are actually testing for is if the intersection of
> > > >> the two sets is not empty where the first set is the characters in
> > > >> your word and the second set is the characters in your defined string.
>
> > > > To expand on what I was saying I thought i should provide a code
> > > > snippet:
>
> > > > WORD = 'g' * 100
> > > > WORD2 = 'g' * 50 + 'U'
> > > > VOWELS = 'aeiouAEIOU'
> > > > BIGWORD = 'g' * 1 + 'U'
>
> > > > def set_test(vowels, word):
>
> > > > vowels = set( iter(vowels))
> > > > letters = set( iter(word) )
>
> > > > if letters & vowels:
> > > > return True
> > > > else:
> > > > return False
>
> > > > with python 2.5 I got 1.30 usec/pass against the BIGWORD
>
> > > You could make it slightly faster by removing the iter() call:
> > > letters = set(word)
> > > And (if vowels are really constant) you could pre-build the vowels set.
>
> > set(word) = set{[word]} meaning a set with one element, the string
> > the call to iter makes it set of the letters making up the word.
>
> Did you try it?
>
> Python 2.6.1 (r261:67515, Jan 7 2009, 17:09:13)
> [GCC 4.3.2] on linux2
> Type "help", "copyright", "credits" or "license" for more information.>>>
> set('abcd')
>
> set(['a', 'c', 'b', 'd'])
>
> --RDM
You are in fact correct. Thank you for pointing that out.
--
http://mail.python.org/mailman/listinfo/python-list
Re: count secton of data in list
On Feb 20, 3:45 pm, Emile van Sebille wrote: > brianrpsgt1 wrote: > > > def step1(val): > > data2_row = [] > > > for d1r in data1_row: > > if d1r[1] >= val: > > switch = 0 > > data2_row = d1r[0],d1r[1],d1r[2],switch > > data2_row.append([d1r[0],d1r[1],d1r[2],switch]) > > HTH, > > Emile def count_consecutive(rows): switch = 0 count = 0 for r in rows: if r[-1] == switch: count += 1 else: switch = not switch if count != 0: yield count count = 0 if count != 0: yield count rows = [ ['2009-01-09','13:17:30,96',123456,0], ['2009-01-09','13:17:31,95',123456,0], ['2009-01-09','13:17:32,95',123456,0], ['2009-01-09','13:17:33,95',123456,0], ['2009-01-09','13:17:34,94',123456,1], ['2009-01-09','13:17:35,94',123456,1], ['2009-01-09','13:17:36,94',123456,1], ['2009-01-09','13:17:37,94',123456,1], ['2009-01-09','13:17:38,94',123456,1], ['2009-01-09','13:17:39,94',123456,1], ['2009-01-09','13:17:40,94',123456,1], ['2009-01-09','13:17:41,94',123456,1], ['2009-01-09','13:17:42,95',123456,0], ['2009-01-09','13:17:43,95',123456,0], ['2009-01-09','13:17:44,95',123456,0], ['2009-01-09','13:17:45,95',123456,0] ] for cnt in count_consecutive(rows): print cnt -- http://mail.python.org/mailman/listinfo/python-list
Re: How to open a remote file using python.
On Feb 22, 9:02 pm, "venutaurus...@gmail.com"
wrote:
> On Feb 23, 9:25 am, MRAB wrote:
>
>
>
> > venutaurus...@gmail.com wrote:
> > > Hello all,
> > > I am writing an application where I need to open a shared
> > > file on a remote machine using python script. I tried using the
> > > following function:
>
> > > f = urllib.open("\\remote_machine\\folder1\\file1.doc")
>
> > > I also tried using
>
> > > class urllib.FancyURLopener(...)
>
> > > but didn't work. Can some one help me in this regard.
>
> > What do you mean by "remote machine"? Do you mean you want to open a
> > file that's in a shared folder on a machine that's on the same local
> > network?
>
> > If it's meant to be a Windows filepath then it should be:
>
> > f = open(r"\\remote_machine\folder1\file1.doc")
>
> > (If the file is a Microsoft Word document file, then you won't probably
> > be able to make much sense of its contents using open().)
>
> Thanks to all for your brisk replies:
>
> Yes, my aim is to open a file from another machine in the same
> LAN. It also worked using
>
> >>f = urllib.urlopen("\\\remote_machine\\folder\\file.doc")
>
> But now I also have to copy the same file to the local machine
> in Python. Do I need to follow any protocols for this?
>
> Thank you,
> Venu.
the copy in shutil will work just fine.
import shutil
shutil.copy(remotepath,localpath)
--
http://mail.python.org/mailman/listinfo/python-list
Re: removing duplication from a huge list.
On Feb 26, 9:15 pm, Chris Rebert wrote:
> On Thu, Feb 26, 2009 at 8:49 PM, Benjamin Peterson
> wrote:
> > Shanmuga Rajan gmail.com> writes:
>
> >> f any one suggests better solution then i will be very happy.Advance thanks
> > for any help.Shan
>
> > Use a set.
>
> To expand on that a bit:
>
> counted_recs = set(rec[0] for rec in some_fun())
> #or in Python 3.0:
> counted_recs = {rec[0] for rec in some_fun()}
>
> Cheers,
> Chris
>
> --
> Follow the path of the Iguana...http://rebertia.com
How big of a list are we talking about? If the list is so big that the
entire list cannot fit in memory at the same time this approach wont
work e.g. removing duplicate lines from a very large file. A couple of
things come into play at that point. If order does not matter then I
would suggest looking at some recipes for first sorting the large file
and then iterating through the lines removing duplicates as you go
( if cur_line != last_line: write cur_line; last_line = cur_line )
If order does matter then let me know and I will post a recipe for
that.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Delete all items in the list
On Feb 26, 3:05 am, Clarendon wrote:
> Hi. This must be a simple command but I just can't find it in the
> Phthon manual. How do I delete all items with a certain condition from
> a list? For instance:
>
> L=['a', 'b', 'c', 'a']
>
> I want to delete all 'a's from the list.
> But if L.remove('a') only deletes the first 'a'.
>
> How do you delete all 'a's?
> I would really appreciate your help.
>
> Thanks.
while 'a' in L:
L.remove('a')
not the most efficient but it works
--
http://mail.python.org/mailman/listinfo/python-list
Re: String search
On Feb 27, 2:17 am, "Gabriel Genellina" wrote: > En Fri, 27 Feb 2009 07:33:44 -0200, pranav escribió: > > > Greeting fellow pycoders, > > > I have a script that browses large codes and replaces certain text > > with some other text. Of lately i observed an issue.Some of the > > original text were like > > > > > ,N'#attributes.SOFTPREREQ#' > > > > > My module does scan this code and suggests replacement to this code. > > But when i use the string.replace() method, it just fails. A > > string.find() for above code in the whole file returns code -1. > > Where am i going wrong? I figure it could be something to do with > > character encoding or else. > > Do you mean, you look for those three complete lines with a single > .find(...)? > Hard to guess... maybe there is some withespace at the end of a line, some > additional leading whitespace in the second line, a tab character instead > of spaces, maybe those ' are instead ´ or `, maybe the file does not use > \n as a line termination, maybe the file encoding doesn't match what you > expect... > > If string.find() does return -1, it is because the text you're looking for > is NOT inside the string. No use in questioning that. If you think the > text IS there, get a decent hex editor and examine the file near the > supposed match, character by character, and see WHERE is the difference > (there MUST be a difference). > > -- > Gabriel Genellina Try looking into regular expressions http://docs.python.org/library/re.html -- http://mail.python.org/mailman/listinfo/python-list
Re: removing duplication from a huge list.
On Feb 27, 1:18 am, Stefan Behnel wrote: > bearophileh...@lycos.com wrote: > > odeits: > >> How big of a list are we talking about? If the list is so big that the > >> entire list cannot fit in memory at the same time this approach wont > >> work e.g. removing duplicate lines from a very large file. > > > If the data are lines of a file, and keeping the original order isn't > > important, then the first to try may be to use the unix (or cygwin on > > Windows) commands sort and uniq. > > or preferably "sort -u", in case that's supported. > > Stefan Although this is true, that is more of an answer to the question "How do i remove duplicates from a huge list in Unix?". -- http://mail.python.org/mailman/listinfo/python-list
Re: Looking for a General Method to Configure Tkinter Widgets
On Mar 2, 7:14 am, "W. eWatson" wrote:
> I'm modifying a Tkinter Python program that uses hard coded initial values
> for several widgets. For example, latitude = 40. My plan is to put the names
> and values for configuration purposes into a file. For example, a pseudo
> statement like the one just given. ConfigObj provides a mechanism for it.
>
> I am only at an early stage of learning Tkinter, but it looks like a hang up
> is in the use of control variables passed between widgets and non-Tkinter
> objects that setup up the widget and retrieve the changed values. Roughly,
> the main loop contains code like self.longitude = 40. Non-Tkinter objects
> set up the parameters to the widgets, and when a widget* is complete the
> setup program resets the main loop globals. As I see it, it looks like
> IntVar, etc. used must be hard coded, as in the original program, to
> preserve types like boolean, strings, integers, floats, etc. It's either
> that or use of eval or some like function. Comments?
>
> * For example, in one setup program, I see code like this after its call to
> a dialog returns:
>
> try:
> s = dialog.stopVar.get()
> d = [ int(x) for x in s.split(":") ]
> self.stop_time = datetime.time(d[0],d[1],d[2])
>
> stop_time is a string like "10:30:15".
> --
> W. eWatson
>
> (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time)
> Obz Site: 39° 15' 7" N, 121° 2' 32" W, 2700 feet
>
> Web Page:
I am sorry, I am a bit confused. Is your question how to preserve the
value and/or type of the data in the config file? Or are you having
trouble getting the values from the file to the widget?
--
http://mail.python.org/mailman/listinfo/python-list
Re: how to find all completely connected sub-graphs?
On Mar 2, 10:35 pm, Hyunchul Kim wrote: > Hi, all, > > How can I find all "completely connected subgraphs" in a graph when node > and edge data are available? > > "completely connected subgraph" is a group, all members of which are > connected to each other. > > Thanks, > > Hyunchul Do you mean all of the member are directly connected to each other? -- http://mail.python.org/mailman/listinfo/python-list
Re: how to find all completely connected sub-graphs?
On Mar 3, 12:07 am, Andre Engels wrote:
> On Tue, Mar 3, 2009 at 7:35 AM, Hyunchul Kim wrote:
> > How can I find all "completely connected subgraphs" in a graph when node and
> > edge data are available?
>
> > "completely connected subgraph" is a group, all members of which are
> > connected to each other.
>
> Here is an algorithm I came up with in a few minutes of thinking,
> complexity O(N*SN) with N being the number of nodes in the graph and
> SN the sum of the nodes in each connected subgraph (there may well be
> faster algorithms, but then you'd probably have to ask someone who
> either already knows it, or spends significantly more time on it than
> I did) - in pseudocode, but translating pseudocode into Python is an
> easy thing to do:
>
> Let N be the nodes in the graph.
> A = {emptyset} # that is, a set containing only the empty set in
> the beginning
> foreach node k in N:
> foreach set a in A:
> if k is connected to each node in a:
> add k+{a} to A # in such a way that it's not included in
> the loop for the current node k
>
> The completely connected subgraphs are the subgraphs for which the set
> of nodes in the subgraph is in A.
>
> --
> André Engels, andreeng...@gmail.com
Seems to me the definition of the completely connected graph is:
for a given node N with an edge set of E
the complete graph is the intersection of all of the edge sets
belonging to each element in E
so assuming you have a dictionary that is d[Node] = list(edgeNodes)
for Node, EdgeNodes in d:
connectedGraph = set(EdgeNodes}
connectedGraph.add(Node)
for EdgeNode in EdgeNodes:
EdgeSet = set(d[EdgeNode])
EdgeSet.add(EdgeNode)
connectedGraph.intersectionUpdate( EdgeSet)
yield connectedGraph
Code is untested but i think illustrates my theory.
Regards,
--
http://mail.python.org/mailman/listinfo/python-list
Re: how to find all completely connected sub-graphs?
On Mar 2, 11:26 pm, Hyunchul Kim wrote: > Dear Odeits, > > Yes, I meant directly connected to each other. > > Thanks. > > Hyunchul > > odeits wrote: > > On Mar 2, 10:35 pm, Hyunchul Kim wrote: > > >> Hi, all, > > >> How can I find all "completely connected subgraphs" in a graph when node > >> and edge data are available? > > >> "completely connected subgraph" is a group, all members of which are > >> connected to each other. > > >> Thanks, > > >> Hyunchul > > > Do you mean all of the member are directly connected to each other? > > -- > >http://mail.python.org/mailman/listinfo/python-list > > I would start by creating sets for every node in your graph that are made up of the node and all of its edges. Then you can use the issubset to filter down your collection. http://docs.python.org/library/sets.html The idea is a node find its completely connected subset. once you have done this, you can remove all of those nodes from the universal set and chose another node. repeat until the universal set is empty. Remember the smallest completely connected subset is an individual node. That should be a good enough starting point to figure out an alg that works for you. Cheers. -- http://mail.python.org/mailman/listinfo/python-list
Re: Pickle Problem
On Mar 3, 4:16 am, Fab86 wrote:
> Thanks, this seems like a simpler way to do it.
>
> I plan on recording 200 values to this file from the outcome of 200
> Yahoo searches. Is there any type of loop I can make to do this or do
> I have to have a line like "print >> f, res1.total_results_available"
> 200 times?
>
> Regards,
>
> Fabien
>
> On Mar 3, 12:00 pm, "andrew cooke" wrote:
>
> > maybe the following are simpler as they use print
>
> > if you are using python 2.6 or python 3:
>
> > >>> from __future__ import print_function
> > >>> f = open('myfile.txt', 'w')
> > >>> print(123, file=f)
> > >>> print(456, file=f)
> > >>> f.close()
>
> > alternatively, in python 2.5 or 2.6:
>
> > >>> f = open('myfile.txt', 'w')
> > >>> print >> f, 123
> > >>> print >> f, 456
> > >>> f.close()
>
> > andrew
>
>
How are you getting res1, res2? in most cases you could just use a for
loop
for res in results:
print >> f, res.total_results_available
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to replace the two last digits from an xml file?
On Mar 6, 11:53 am, awel wrote:
> Hi,
>
> I am trying to get a value from an xml file, increment it by 1,
> replace by the new value and write another xml with the new value
> inside.
>
> I have found this code to get the line and it works but I have to do
> everything manualy:
>
> import re
> lines = open("c:\\File1.xml").readlines()
> for i in range(len(lines)):
> if re.search('national.number', lines[i]):
> lines[i] = lines[i].replace
> ('01aaa02','01aaa03')
> open("File2.xml","w").writelines(lines)
>
> The goal I am trying to reach is to create 30 different xml.
>
> Could someone help me?
>
> Thanks
> Al
I would look into actually using an xml parser like lxml or the built
in elementTree. You might run into some problems later on with the
regex matching something unintended. Having said that..
If you want to stick to regex i would pull out the number using a
group, convert the number to a python type, increment then do the
replace.
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to replace the two last digits from an xml file?
On Mar 6, 3:31 pm, odeits wrote:
> On Mar 6, 11:53 am, awel wrote:
>
>
>
> > Hi,
>
> > I am trying to get a value from an xml file, increment it by 1,
> > replace by the new value and write another xml with the new value
> > inside.
>
> > I have found this code to get the line and it works but I have to do
> > everything manualy:
>
> > import re
> > lines = open("c:\\File1.xml").readlines()
> > for i in range(len(lines)):
> > if re.search('national.number', lines[i]):
> > lines[i] = lines[i].replace
> > ('01aaa02','01aaa03')
> > open("File2.xml","w").writelines(lines)
>
> > The goal I am trying to reach is to create 30 different xml.
>
> > Could someone help me?
>
> > Thanks
> > Al
>
> I would look into actually using an xml parser like lxml or the built
> in elementTree. You might run into some problems later on with the
> regex matching something unintended. Having said that..
>
> If you want to stick to regex i would pull out the number using a
> group, convert the number to a python type, increment then do the
> replace.
having dug a little deeper myself into the regular expressions in
python and grouping i stumbled onto this
http://www.amk.ca/python/howto/regex/regex.html#SECTION00052
Right above section 6. Common Problems there is an example of how to
change decimal numbers to hex... with a little tweek i believe it will
do exactly what you want.
Good Luck
--
http://mail.python.org/mailman/listinfo/python-list
Help cleaning up some code
I am looking to clean up this code... any help is much appreciated. Note: It works just fine, I just think it could be done cleaner. The result is a stack of dictionaries. the query returns up to STACK_SIZE ads for a user. The check which i think is very ugly is putting another contraint saying that all of the ni have to be the same. stack = [] rows = self.con.execute(adquerystring,(user,STACK_SIZE)).fetchall() for row in rows: ad = dict() ad['ni'] = row['ni'] ad['adid'] = row['adid'] ad['rundateid'] = row['rundateid'] ad['rundate'] = row['rundate'] if row['city'] is None: ad['city'] = 'None' else: ad['city'] = row['city'] if row['state'] is None: ad['state'] = 'None' else: ad['state'] = row['state'] ad['status'] = row['status'] try: if stack[0]['ni'] != ad['ni']: break; except IndexError: pass stack.append(ad) -- http://mail.python.org/mailman/listinfo/python-list
Re: Help cleaning up some code
On Mar 7, 1:07 pm, Scott David Daniels wrote: > odeits wrote: > > I am looking to clean up this code... any help is much appreciated. > > Note: It works just fine, I just think it could be done cleaner. > > > The result is a stack of dictionaries. the query returns up to > > STACK_SIZE ads for a user. The check which i think is very ugly is > > putting another contraint saying that all of the ni have to be the > > same. > > Well, the obvious way to get your constraint is by changing your SQL, > but if you are going to do it by fetching rows, try: > > FIELDS = 'ni adid rundateid rundate city state status'.split() > ni = UNSET = object() # use None unless None might be the value > stack = [] > rows = self.con.execute(adquerystring, (user,STACK_SIZE)).fetchall() > for row in rows: > ad = dict() > for field in FIELDS: > ad[field] = row[field] > for field in 'city', 'state': > if ad[field] is None: > ad[field] = 'None' > if ni != ad['ni']: > if ni is UNSET: > ni = ad['ni'] > else: > break > stack.append(ad) > > --Scott David Daniels > scott.dani...@acm.org Taking from several suggestions this is what i have come up with for now: for row in ifilter(lambda r: r['ni'] == rows[0]['ni'],rows): ad = dict() keys = row.keys() # if python 2.6 keys = ['ni','adid','rundateid','rundate','city','state','status'] # if python 2.5 for index in row.keys(): if row[index] is None: ad[index] = 'None' else: ad[index] = row[index] stack.append(ad) print row the test to see if the ad is valid is placed in the ifilter so that I dont build the dictionary unnecessarily. and the None special case is fairly simple to read now. The None case would even be irrelevant if i could get the damn xmlrpc to allow null. sigh. anyhow. thanks for all of your input, it is definitely better than it was ;) -- http://mail.python.org/mailman/listinfo/python-list
Re: Help cleaning up some code
On Mar 7, 10:58 pm, odeits wrote:
> On Mar 7, 1:07 pm, Scott David Daniels wrote:
>
>
>
> > odeits wrote:
> > > I am looking to clean up this code... any help is much appreciated.
> > > Note: It works just fine, I just think it could be done cleaner.
>
> > > The result is a stack of dictionaries. the query returns up to
> > > STACK_SIZE ads for a user. The check which i think is very ugly is
> > > putting another contraint saying that all of the ni have to be the
> > > same.
>
> > Well, the obvious way to get your constraint is by changing your SQL,
> > but if you are going to do it by fetching rows, try:
>
> > FIELDS = 'ni adid rundateid rundate city state status'.split()
> > ni = UNSET = object() # use None unless None might be the value
> > stack = []
> > rows = self.con.execute(adquerystring, (user,STACK_SIZE)).fetchall()
> > for row in rows:
> > ad = dict()
> > for field in FIELDS:
> > ad[field] = row[field]
> > for field in 'city', 'state':
> > if ad[field] is None:
> > ad[field] = 'None'
> > if ni != ad['ni']:
> > if ni is UNSET:
> > ni = ad['ni']
> > else:
> > break
> > stack.append(ad)
>
> > --Scott David Daniels
> > scott.dani...@acm.org
>
> Taking from several suggestions this is what i have come up with for
> now:
>
> for row in ifilter(lambda r: r['ni'] == rows[0]['ni'],rows):
> ad = dict()
>
> keys = row.keys() # if python 2.6
> keys =
> ['ni','adid','rundateid','rundate','city','state','status'] # if
> python 2.5
>
> for index in row.keys():
> if row[index] is None:
> ad[index] = 'None'
> else:
> ad[index] = row[index]
> stack.append(ad)
> print row
>
> the test to see if the ad is valid is placed in the ifilter so that I
> dont build the dictionary unnecessarily. and the None special case is
> fairly simple to read now. The None case would even be irrelevant if i
> could get the damn xmlrpc to allow null. sigh. anyhow. thanks for all
> of your input, it is definitely better than it was ;)
For those of you who asked about the SQL the full function is here.
The connection is to a sqlite database with the row_factory set to
sqlite.Row
def get(self,user):
'''
Fetches an ad for USER, assigns the ad and returns a
dictionary that represents an ad
'''
stack = []
aduserquerystring = "SELECT ni, adid, rundateid, rundate,
city, state, status, priority,time FROM ads NATURAL JOIN rundates
NATURAL JOIN newspapers WHERE ( status in (1,3) and user = ?) "
expiredadquerystring = "SELECT ni, adid, rundateid, rundate,
city, state, status, priority,time FROM ads NATURAL JOIN rundates
NATURAL JOIN newspapers WHERE ( status = 1 AND time < DATETIME
('NOW', '-%d MINUTES') ) LIMIT 0,?"%(MINUTES_TO_EXPIRE,)
adquerystring = "SELECT ni, adid, rundateid, rundate, city,
state, status , priority, time FROM ads NATURAL JOIN rundates NATURAL
JOIN newspapers WHERE (status IN (0,2) AND priority IN ( SELECT
priority FROM users NATURAL JOIN groups WHERE user = ? )) ORDER BY
status DESC, priority ASC, ni ASC, time ASC, adid ASC LIMIT 0,?"
rows = self.con.execute(aduserquerystring,(user,)).fetchall()
if len(rows) == 0:
rows = self.con.execute(expiredadquerystring,
(STACK_SIZE,)).fetchall()
if len(rows) == 0:
rows = self.con.execute(adquerystring,
(user,STACK_SIZE)).fetchall()
print user
keys =
['ni','adid','rundateid','rundate','status','city','state']
for row in ifilter(lambda r: r['ni'] == rows[0]['ni'], rows):
ad = dict( )
for key in keys:
if row[key] is None:
ad[key] = 'None'
else:
ad[key] = row[key]
stack.append(ad)
print row
self.con.executemany('UPDATE ads SET user = ?, status = CASE
(status) WHEN 1 THEN 1 WHEN 0 THEN 1 WHEN 2 THEN 3 END WHERE adid = ?',
[(user, ad['adid']) for ad in stack])
self.con.commit()
return stack
--
http://mail.python.org/mailman/listinfo/python-list
Re: Help cleaning up some code
On Mar 8, 4:48 am, "andrew cooke" wrote: > odeits wrote: > > On Mar 7, 1:07 pm, Scott David Daniels wrote: > >> odeits wrote: > >> > I am looking to clean up this code... any help is much appreciated. > >> > Note: It works just fine, I just think it could be done cleaner. > > >> > The result is a stack of dictionaries. the query returns up to > >> > STACK_SIZE ads for a user. The check which i think is very ugly is > >> > putting another contraint saying that all of the ni have to be the > >> > same. > > >> Well, the obvious way to get your constraint is by changing your SQL, > >> but if you are going to do it by fetching rows, try: > > >> FIELDS = 'ni adid rundateid rundate city state status'.split() > >> ni = UNSET = object() # use None unless None might be the value > >> stack = [] > >> rows = self.con.execute(adquerystring, > >> (user,STACK_SIZE)).fetchall() > >> for row in rows: > >> ad = dict() > >> for field in FIELDS: > >> ad[field] = row[field] > >> for field in 'city', 'state': > >> if ad[field] is None: > >> ad[field] = 'None' > >> if ni != ad['ni']: > >> if ni is UNSET: > >> ni = ad['ni'] > >> else: > >> break > >> stack.append(ad) > > >> --Scott David Daniels > >> scott.dani...@acm.org > > > Taking from several suggestions this is what i have come up with for > > now: > > > for row in ifilter(lambda r: r['ni'] == rows[0]['ni'],rows): > > not sure what version of python you're using, but it would be more natural > in recent python to write that as: > > for row in (r for r in rows if r['ni'] == rows[0]['ni']): > > (the () create a generator for you). > > andrew > > > ad = dict() > > > keys = row.keys() # if python 2.6 > > keys = > > ['ni','adid','rundateid','rundate','city','state','status'] # if > > python 2.5 > > > for index in row.keys(): > > if row[index] is None: > > ad[index] = 'None' > > else: > > ad[index] = row[index] > > stack.append(ad) > > print row > > > the test to see if the ad is valid is placed in the ifilter so that I > > dont build the dictionary unnecessarily. and the None special case is > > fairly simple to read now. The None case would even be irrelevant if i > > could get the damn xmlrpc to allow null. sigh. anyhow. thanks for all > > of your input, it is definitely better than it was ;) > > -- > >http://mail.python.org/mailman/listinfo/python-list This function is very time critical so i went with itertools "for efficient looping" and i am runing 2.5 -- http://mail.python.org/mailman/listinfo/python-list
Re: Help cleaning up some code
On Mar 8, 11:31 am, Dennis Lee Bieber wrote:
> On Sat, 7 Mar 2009 23:07:55 -0800 (PST), odeits
> declaimed the following in gmane.comp.python.general:
>
>
>
>
>
> > For those of you who asked about the SQL the full function is here.
> > The connection is to a sqlite database with the row_factory set to
> > sqlite.Row
>
> > def get(self,user):
> > '''
> > Fetches an ad for USER, assigns the ad and returns a
> > dictionary that represents an ad
> > '''
>
> > stack = []
> > aduserquerystring = "SELECT ni, adid, rundateid, rundate,
> > city, state, status, priority,time FROM ads NATURAL JOIN rundates
> > NATURAL JOIN newspapers WHERE ( status in (1,3) and user = ?) "
> > expiredadquerystring = "SELECT ni, adid, rundateid, rundate,
> > city, state, status, priority,time FROM ads NATURAL JOIN rundates
> > NATURAL JOIN newspapers WHERE ( status = 1 AND time < DATETIME
> > ('NOW', '-%d MINUTES') ) LIMIT 0,?"%(MINUTES_TO_EXPIRE,)
> > adquerystring = "SELECT ni, adid, rundateid, rundate, city,
> > state, status , priority, time FROM ads NATURAL JOIN rundates NATURAL
> > JOIN newspapers WHERE (status IN (0,2) AND priority IN ( SELECT
> > priority FROM users NATURAL JOIN groups WHERE user = ? )) ORDER BY
> > status DESC, priority ASC, ni ASC, time ASC, adid ASC LIMIT 0,?"
>
> Not knowing which fields come from which tables (nor which are
> commonly named per the definition of natural join) the first thing that
> strikes me here is that it looks like you could create a view/virtual
> table in the database to consolidate that
>
> ... rundates natural join newspapers ...
>
> And do your tables really share that many columns? Since the
> definition of the natural join is basically an inner/equi-join over
> every column that has the same name in the two tables, you are forcing
> the SQL engine to basically loop over all the columns in first table,
> looking for matching column name in the second table, and computing a
> "t1.colname = t2.colname" for EACH matching column.
>
> If there are many such columns in common, I'd be looking to factor
> them out into a separate table with an integer primary key, and put the
> foreign keys into the original tables.
>
> > rows = self.con.execute(aduserquerystring,(user,)).fetchall()
> > if len(rows) == 0:
>
> I suspect a simple
>
> if not rows:
>
> can be used.
>
> > rows = self.con.execute(expiredadquerystring,
> > (STACK_SIZE,)).fetchall()
> > if len(rows) == 0:
> > rows = self.con.execute(adquerystring,
> > (user,STACK_SIZE)).fetchall()
>
> In the worst case you are using three separate hits on the database
> (and recomputing the same natural joins in many cases each time --
> definitely suggest creating a view for the basic joins to simplify the
> parsing -- if not a physical read-only table, though that would need to
> be refreshed at times).
>
> A one-time database hit:
>
> SQL = """select 1, ni, adid, rundateid, rundate, city,
> state, status, priority, time
> from ads natural join rundates
> natural join newspapers
> where status in (1, 3) and user = :USER
>
> UNION
> select 2, ni, adid, rundateid, rundate, city,
> state, status, priority, time
> from ads natural join rundates
> natural join newspapers
> where status = 1 and time < datetime("now", "-%d
> minutes")
> limit 0, :STACK
>
> UNION
> select 3, ni, adid, rundateid, rundate, city,
> state, status, priority, time
> from ads natural join rundates
> natural join newspapers
> where status in (0, 2) and priority in
> (select priority from users natural join
> groups
> where user = :USER)
> order by status desc, priority, ni, time, adid
> limit 0, :STACK""" % (MINUTES_TO_EXPIRE,)
>
> rows = self.con.execute(SQL,
&g
Re: Help cleaning up some code
On Mar 8, 12:31 pm, Dennis Lee Bieber wrote:
> On Sat, 7 Mar 2009 23:07:55 -0800 (PST), odeits
> declaimed the following in gmane.comp.python.general:
>
>
>
>
>
> > For those of you who asked about the SQL the full function is here.
> > The connection is to a sqlite database with the row_factory set to
> > sqlite.Row
>
> > def get(self,user):
> > '''
> > Fetches an ad for USER, assigns the ad and returns a
> > dictionary that represents an ad
> > '''
>
> > stack = []
> > aduserquerystring = "SELECT ni, adid, rundateid, rundate,
> > city, state, status, priority,time FROM ads NATURAL JOIN rundates
> > NATURAL JOIN newspapers WHERE ( status in (1,3) and user = ?) "
> > expiredadquerystring = "SELECT ni, adid, rundateid, rundate,
> > city, state, status, priority,time FROM ads NATURAL JOIN rundates
> > NATURAL JOIN newspapers WHERE ( status = 1 AND time < DATETIME
> > ('NOW', '-%d MINUTES') ) LIMIT 0,?"%(MINUTES_TO_EXPIRE,)
> > adquerystring = "SELECT ni, adid, rundateid, rundate, city,
> > state, status , priority, time FROM ads NATURAL JOIN rundates NATURAL
> > JOIN newspapers WHERE (status IN (0,2) AND priority IN ( SELECT
> > priority FROM users NATURAL JOIN groups WHERE user = ? )) ORDER BY
> > status DESC, priority ASC, ni ASC, time ASC, adid ASC LIMIT 0,?"
>
> Not knowing which fields come from which tables (nor which are
> commonly named per the definition of natural join) the first thing that
> strikes me here is that it looks like you could create a view/virtual
> table in the database to consolidate that
>
> ... rundates natural join newspapers ...
>
> And do your tables really share that many columns? Since the
> definition of the natural join is basically an inner/equi-join over
> every column that has the same name in the two tables, you are forcing
> the SQL engine to basically loop over all the columns in first table,
> looking for matching column name in the second table, and computing a
> "t1.colname = t2.colname" for EACH matching column.
>
> If there are many such columns in common, I'd be looking to factor
> them out into a separate table with an integer primary key, and put the
> foreign keys into the original tables.
>
> > rows = self.con.execute(aduserquerystring,(user,)).fetchall()
> > if len(rows) == 0:
>
> I suspect a simple
>
> if not rows:
>
> can be used.
>
> > rows = self.con.execute(expiredadquerystring,
> > (STACK_SIZE,)).fetchall()
> > if len(rows) == 0:
> > rows = self.con.execute(adquerystring,
> > (user,STACK_SIZE)).fetchall()
>
> In the worst case you are using three separate hits on the database
> (and recomputing the same natural joins in many cases each time --
> definitely suggest creating a view for the basic joins to simplify the
> parsing -- if not a physical read-only table, though that would need to
> be refreshed at times).
>
> A one-time database hit:
>
> SQL = """select 1, ni, adid, rundateid, rundate, city,
> state, status, priority, time
> from ads natural join rundates
> natural join newspapers
> where status in (1, 3) and user = :USER
>
> UNION
> select 2, ni, adid, rundateid, rundate, city,
> state, status, priority, time
> from ads natural join rundates
> natural join newspapers
> where status = 1 and time < datetime("now", "-%d
> minutes")
> limit 0, :STACK
>
> UNION
> select 3, ni, adid, rundateid, rundate, city,
> state, status, priority, time
> from ads natural join rundates
> natural join newspapers
> where status in (0, 2) and priority in
> (select priority from users natural join
> groups
> where user = :USER)
> order by status desc, priority, ni, time, adid
> limit 0, :STACK""" % (MINUTES_TO_EXPIRE,)
>
> rows = self.con.execute(SQL,
&g
Re: Help cleaning up some code
On Mar 9, 1:06 am, Dennis Lee Bieber wrote:
> On Sun, 8 Mar 2009 19:07:08 -0700 (PDT), odeits
> declaimed the following in gmane.comp.python.general:
>
>
>
> > i get this error when running that query:
>
> > sqlite3.OperationalError: LIMIT clause should come after UNION not
> > before
>
> Well, I did generate that as a bit of off-the-cuff...
>
> Apparently SQL parses the UNION as a higher precedence than LIMIT --
> wanting it to apply to the final results of everything.
>
> And since I'm trying to reduce the number of selects overall, I sure
> don't want to suggest making them subselects...
>
> select first long mess
> ...
> union
> select * from
> (select second long mess
> ...
> limit 0, ?)
> union
> select * from
> (select third long mess
> ...
> limit 0, ?)
>
> Putting the limit last would have been okay if it weren't for the
> lack of a limit on the "first long mess" -- since, as I recall, you have
> the same limit for "second" and "third".
> --
> Wulfraed Dennis Lee Bieber KD6MOG
> wlfr...@ix.netcom.com wulfr...@bestiaria.com
> HTTP://wlfraed.home.netcom.com/
> (Bestiaria Support Staff: web-a...@bestiaria.com)
> HTTP://www.bestiaria.com/
Doing all of this work in the query made me realize that all the
filtering can be done just on the ADS table, so i modified the query
you provided to this :
select adid, rundateid,priority, rundate, ni,city,state
FROM ads NATURAL JOIN rundates NATURAL JOIN newspapers WHERE adid in
(
SELECT * from (
SELECT adid from ads
where status in (1, 3) and user
= :USER LIMIT 0, :STACK
)
UNION
SELECT * from (
SELECT adid from ads
where status = 1 and time <
datetime("now", "-%d minutes") LIMIT 0,:STACK
)
UNION
SELECT * from (
SELECT adid from ads
where status in (0, 2) and
priority in
(
select priority from
users natural join groups where user = :USER
) limit 0,:STACK
)
)
order by status desc, priority, time, adid
limit 0, :STACK
and i have achieved a 4x improvement in speed!!! thanks so much.
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to extract some text?
On Mar 8, 3:50 pm, Oltmans wrote: > On Mar 9, 3:37 am, Chris Rebert wrote: > > > Learn about the methods of the string class > > (str):http://docs.python.org/library/stdtypes.html#id4 > > > You'll probably be most interested in .split() > > OK, thanks I got it. I was trying to use Regex but .split() just > worked like a charm. Thank you ;) > > > > > Cheers, > > Chris > > > -- > > I have a blog:http://blog.rebertia.com > > The regex you were lookingfor was probably something like p = re.compile(r'href='(.*?)') when you p.match(mystring) group(1) will be your url -- http://mail.python.org/mailman/listinfo/python-list
Re: Input data from .txt file and object array problem
On Mar 12, 5:03 am, SamuelXiao wrote:
> I want to input data by using pickle
> First of all, I have a database.txt
> The content is like:
>
> AAA,aaalink
> BBB,bbblink
> CCC,ccclink
> ...,...
>
> AAA,BBB,CCC is Language name, and aaalink,bbblink,ccclink is their
> respective link.
> I want to store data by using pickle. Meanwhile , I got a problem.
> #I have created a class:
> class Lang:
> def __init__(self,lname="",tlink="",alink=""):
> self.lname = lname #lname is the Language
> self.tlink = tlink #tlink is the tutorial link
> self.alink = alink #alink is the a link to school Library
> finding
> the first returned Language book
>
> def alibrary_link(self,alink):
> self.alink = alink
>
> def tutorial_link(self,tlink):
> self.tlink = tlink
>
> def lang_name(self,lname):
> self.lname = lname
>
> def _display(self):
> string = "+++" + \
> "+" + lname \
> "+" + tlink \
> "+" + alink \
> "+"
>
> def Print(self):
> print self._display()
>
> def originData():
> fo = ("/database.txt","r+")
> lines = fo.readlines()
> for line in lines:
> pair = line.split(",")
> temp = Lang();
> temp.lname = pair[0]
> temp.tlink = pair[1]
> temp.alink = findbook(temp.lname)
> #stopping here, because I don't know how to do here...
> #I want to use object array here...
> #Then using pickle to dump the object...
> # Is this work? Or there is another better method to do so?
>
> I hope to create an object array to store all the language and its
> information from database.txt.
> How can I do that? I am a beginner to Python. Any help would be
> appreciated.
check out the csv module for the parsing of the file
http://docs.python.org/library/csv.html
--
http://mail.python.org/mailman/listinfo/python-list
Re: converting a string to a function parameter
On Mar 13, 12:52 am, koranthala wrote: > Hi, > Is it possible to convert a string to a function parameter? > Ex: > str = 'True, type=rect, sizes=[3, 4]' > and I should be able to use it as: > test(convert(str)) and the behaviour should be same as calling test > with those values : > i.e. test(True, type=rect, sizes=[3, 4]) > > I tried eval, but it did not work. And any other mechanism I think > turns out to be creating a full fledged python parser. > > Is there any mechanism with which we can do this straight away? If the string has all of the names you could parse it into a dictionary and pass it as the keyword arguments -- http://mail.python.org/mailman/listinfo/python-list
Re: Input data from .txt file and object array problem
On Mar 12, 9:02 am, SamuelXiao wrote:
> On Mar 12, 11:17 pm, Piet van Oostrum wrote:
>
>
>
> > >>>>> SamuelXiao (S) wrote:
> > >S> I want to input data by using pickle
> > >S> First of all, I have a database.txt
> > >S> The content is like:
> > >S> AAA,aaalink
> > >S> BBB,bbblink
> > >S> CCC,ccclink
> > >S> ...,...
> > >S> AAA,BBB,CCC is Language name, and aaalink,bbblink,ccclink is their
> > >S> respective link.
> > >S> I want to store data by using pickle. Meanwhile , I got a problem.
> > >S> #I have created a class:
> > >S> class Lang:
> > >S> def __init__(self,lname="",tlink="",alink=""):
> > >S> self.lname = lname #lname is the Language
> > >S> self.tlink = tlink #tlink is the tutorial link
> > >S> self.alink = alink #alink is the a link to school Library
> > >finding
> > >S> the first returned Language book
> > >S> def alibrary_link(self,alink):
> > >S> self.alink = alink
> > >S> def tutorial_link(self,tlink):
> > >S> self.tlink = tlink
> > >S> def lang_name(self,lname):
> > >S> self.lname = lname
> > >S> def _display(self):
> > >S> string = "+++" + \
> > >S> "+" + lname \
> > >S> "+" + tlink \
> > >S> "+" + alink \
> > >S> "+"
>
> > This will not do very much. The string is calculated and then thrown
> > away. You will need also a "return string" to make it useful.
>
> > >S> def Print(self):
> > >S> print self._display()
> > >S> def originData():
> > >S> fo = ("/database.txt","r+")
> > >S> lines = fo.readlines()
> > >S> for line in lines:
> > >S> pair = line.split(",")
> > >S> temp = Lang();
> > >S> temp.lname = pair[0]
> > >S> temp.tlink = pair[1]
> > >S> temp.alink = findbook(temp.lname)
>
> > Why do you set these attributes directly while you also have methods for
> > this (like lang_name, tutorial_link)? Or better use
> > temp = Lang(pair[0], pair[1], findbook(temp.lname))
>
> > >S> #stopping here, because I don't know how to do here...
> > >S> #I want to use object array here...
> > >S> #Then using pickle to dump the object...
> > >S> # Is this work? Or there is another better method to do so?
> > >S> I hope to create an object array to store all the language and its
> > >S> information from database.txt.
>
> > I guess you want to put them in a list. Then use
> > objList.append(temp)
> > here and
> > objList = [] before the loop.
>
> > >S> How can I do that? I am a beginner to Python. Any help would be
> > >S> appreciated.
>
> > You can use pickle to store the list is a file after reading.
> > Something like (untested):
>
> > import pickle # or use the faster cPickle module.
>
> > output = open('somefile', 'wb')
>
> > # Pickle object list
> > pickle.dump(objList, output)
>
> > output.close()
>
> > But as this is only textual data (I think) there is not much profit in
> > using pickle. Unless you have other things added in your class. You
> > could also write these things to a simple text file, for example with
> > the csv module.
>
> > --
> > Piet van Oostrum
> > URL:http://pietvanoostrum.com[PGP8DAE142BE17999C4]
> > Private email: p...@vanoostrum.org
>
> Hi, odeits, Piet van Oostrum, thanks for your reply. But I found that
> my school's server have problem with reading csv file. So, I think I
> have to do it another way. By the way, is there any better way to
> store data? Actually, what I want to do is like:
>
> 1. Given a post or an essay, I want to find any language names in the
> post or essay matching with data in my database.txt
> 2. Then insert about the matched word.
>
> My original version is super slow because each time go to and back
> from the library website wastes a lot of times. I hope to use class
> to store the data in database.txt first, then store the book
> information from library. Then visited matched word no need to go to
> search again. Just simply insert about it. But my problem
> is how to store object array and call it out? Thanks for any help.
datafile = open('datafile.txt','rb')
pairs = [line.strip().split(',',1) for line in datafile]
datafile.close()
textfile = open('essay.txt','rb')
text = textfile.read()
textfile.close()
for word,link in pairs:
text = text.replace(word, link) # change this to decorate the link
with the http://mail.python.org/mailman/listinfo/python-list
Re: converting pipe delimited file to fixed width
On Mar 19, 8:51 am, digz wrote:
> Hi,
> I am trying to convert a | delimited file to fixed width by right
> padding with spaces, Here is how I have written the program , just get
> the feeling this can be done in a much better ( python functional )
> way rather than the procedural code i have below . Any help
> appreciated
>
> #!/usr/bin/python
> def rightFill(fillString, toLength, fillChar):
> return fillString+''.join([fillChar for x in range(len
> (fillString),toLength)])
>
> fieldWidth=[ 14, 6, 18, 21, 21,4, 6 ];
>
> file = open("/home/chatdi/input.csv", "r");
> lines = file.readlines()
> file.close()
>
> out = open( "/home/chatdi/ouptut.csv", 'w')
> for line in lines:
> line = line[:-1]
> index = 0
> for token in line.split('|'):
> paddedToken = rightFill(token, fieldWidth[index], ' ' )
> out.write( paddedToken )
> index = index + 1
> out.write("\n")
Caveat: none of the solutions (including mine) deal with the case of
the field being longer than the width. You might want to throw an
exception.
I use the csv module to do the reading of the file, i am going to do
some testing to see if i can set the delimiter to '' so that i can
also use it for the output. For now this should work.
http://docs.python.org/library/csv.html
import csv
fieldWidth=[ 14, 6, 18, 21, 21,4, 6 ];
csvfilepath = 'somefile.csv'
outcsvfilepath = 'someoutfile.csv'
endl = '\n'
f = open(csvfilepath)
outf = open(outcsvfilepath)
csvfile = csv.reader(f,delimiter = '|')
for row in csvfile:
outrow = [ field + fillchar * (width - len(field)) for width, field
in zip(fieldWidth,row)]
outcsv.write( ''.join(outrow))
outcsv.write( endl )
f.close()
outf.close()
--
http://mail.python.org/mailman/listinfo/python-list
Re: Threads not Improving Performance in Program
On Mar 19, 9:50 am, Ryan Rosario wrote:
> I have a parser that needs to process 7 million files. After running
> for 2 days, it had only processed 1.5 million. I want this script to
> parse several files at once by using multiple threads: one for each
> file currently being analyzed.
>
> My code iterates through all of the directories within a directory,
> and at each directory, iterates through each file in that directory. I
> structured my code something like this. I think I might be
> misunderstanding how to use threads:
>
> mythreads = []
> for directory in dirList:
> #some processing...
> for file in fileList:
> p = Process(currDir,directory,file) #class that extends
> thread.Threading
> mythreads.append(p)
> p.start()
>
> for thread in mythreads:
> thread.join()
> del thread
>
> The actual class that extends threading.thread is below:
>
> class Process(threading.Thread):
> vlock = threading.Lock()
> def __init__(self,currDir,directory,file): #thread constructor
> threading.Thread.__init__(self)
> self.currDir = currDir
> self.directory = directory
> self.file = file
> def run(self):
> redirect = re.compile(r'#REDIRECT',re.I)
> xmldoc = minidom.parse(os.path.join(self.currDir,self.file))
> try:
> markup =
> xmldoc.firstChild.childNodes[-2].childNodes[-2].childNodes[-2].childNodes[0].data
> except:
> #An error occurred
> Process.vlock.acquire()
> BAD = open("bad.log","a")
> BAD.writelines(self.file + "\n")
> BAD.close()
> Process.vlock.release()
> print "Error."
> return
> #if successful, do more processing...
>
> I did an experiment with a variety of numbers of threads and there is
> no performance gain. The code is taking the same amount of time to
> process 1000 files as it would if the code did not use threads. Any
> ideas on what I am doing wrong?
Perhabs the bottleneck is the IO. How big are the files you are trying
to parse? Another possible bottleneck is that threads share memory.
Thread construction is also expensive in python, try looking up a
threadPool class. (ThreadPools are collections of threads that do
work, when they finish they go to an idle state untill you give them
more work, that way you aren't constantly creating new threads which
is expensive)
--
http://mail.python.org/mailman/listinfo/python-list
Re: splitting a large dictionary into smaller ones
On Mar 22, 7:32 pm, per wrote: > hi all, > > i have a very large dictionary object that is built from a text file > that is about 800 MB -- it contains several million keys. ideally i > would like to pickle this object so that i wouldnt have to parse this > large file to compute the dictionary every time i run my program. > however currently the pickled file is over 300 MB and takes a very > long time to write to disk - even longer than recomputing the > dictionary from scratch. > > i would like to split the dictionary into smaller ones, containing > only hundreds of thousands of keys, and then try to pickle them. is > there a way to easily do this? i.e. is there an easy way to make a > wrapper for this such that i can access this dictionary as just one > object, but underneath it's split into several? so that i can write > my_dict[k] and get a value, or set my_dict[m] to some value without > knowing which sub dictionary it's in. > > if there aren't known ways to do this, i would greatly apprciate any > advice/examples on how to write this data structure from scratch, > reusing as much of the dict() class as possible. > > thanks. > > large_dict[a] So that I understand the goal, the reason you wish to split the dictionary into smaller ones is so that you dont have to write all of the dictionaries to disk if they haven't changed? Or are you trying to speed up the initial load time? If you are trying to speed up the initial load time I don't think this will help. If the save time is what you are after maybe you want to check out memory mapped files. -- http://mail.python.org/mailman/listinfo/python-list
