Plot all contents in a two dimension list in a loop?
Hi, say I have two lists a and b, which are both in two dimensions. I wanna to plot all curves by using pylab.plot(a[i],b[i]) in a loop for the range of the length(a). I got quite a lot of figures, but only the first one with the plot i want. how to eliminate those blank figures poping out? Rgds Huisky -- http://mail.python.org/mailman/listinfo/python-list
How to read such file and sumarize the data?
Say I have following log file, which records the code usage. I want to read this file and do the summarize how much total CPU time consumed for each user. Is Python able to do so or say easy to achieve this?, anybody can give me some hints, appricate very much! Example log file. ** LSTC license server version 224 started at Sun Dec 6 18:56:48 2009 using configuration file /usr/local/lstc/server_data xyz 15...@trofast3.marin.ntnu.no LS-DYNA_971 NCPU=1 started Sun Dec 6 18:57:40 15...@trofast3.marin.ntnu.no completed Sun Dec 6 19:42:55 xyz 15...@trofast3.marin.ntnu.no LS-DYNA_971 NCPU=2 started Sun Dec 6 20:17:02 15...@trofast3.marin.ntnu.no completed Sun Dec 6 20:26:03 xyz 18...@trofast2.marin.ntnu.no LS-DYNA_971 NCPU=1 started Sun Dec 6 21:01:17 18...@trofast2.marin.ntnu.no completed Sun Dec 6 21:01:28 tanhoi 5...@iimt-tanhoi-w.ivt.ntnu.no LS-DYNA_971 NCPU=1 started Mon Dec 7 09:31:00 5...@iimt-tanhoi-w.ivt.ntnu.no presumed dead Mon Dec 7 10:36:48 sabril 18...@trofast2.marin.ntnu.no LS-DYNA_971 NCPU=2 started Mon Dec 7 13:14:47 18...@trofast2.marin.ntnu.no completed Mon Dec 7 13:24:07 sabril 18...@trofast2.marin.ntnu.no LS-DYNA_971 NCPU=2 started Mon Dec 7 14:21:34 sabril 18...@trofast2.marin.ntnu.no LS-DYNA_971 NCPU=2 started Mon Dec 7 14:28:42 18...@trofast2.marin.ntnu.no killed Mon Dec 7 14:31:48 18...@trofast2.marin.ntnu.no killed Mon Dec 7 14:32:06 -- http://mail.python.org/mailman/listinfo/python-list
Re: How to read such file and sumarize the data?
thank you Martin. You are right. But the elapsed time is also okay for me. And i would like to assume that the total CPU time equals to the number of CPUs multiply the elapsed time. As to the number you mentioned, it is the 'process id', so it will be no problem to identify each job. Huiksy On Nov 18, 12:38 am, Martin Gregorie wrote: > On Wed, 17 Nov 2010 13:45:58 -0800, huisky wrote: > > Say I have following log file, which records the code usage. I want to > > read this file and do the summarize how much total CPU time consumed for > > each user. > > Two points you should think about: > > - I don't think you can extract CPU time from this log: you can get > the process elapsed time and the number of CPUs each run has used, > but you can't calculate CPU time from those values since you don't > know how the process spent waiting for i/o etc. > > - is the first (numeric) part of the first field on the line a process id? > If it is, you can match start and stop messages on the value of the > first field provided that this value can never be shared by two > processes that are both running. If you can get simultaneous > duplicates, then you're out of luck because you'll never be able to > match up start and stop lines. > > > Is Python able to do so or say easy to achieve this?, anybody can give > > me some hints, appricate very much! > > Sure. There are two approaches possible: > - sort the log on the first two fields and then process it with Python > knowing that start and stop lines will be adjacent > > - use the first field as the key to an array and put the start time > and CPU count in that element. When a matching stop line is found > you, retrieve the array element, calculate and output or total the > usage figure for that run and delete the array element. > > -- > martin@ | Martin Gregorie > gregorie. | Essex, UK > org | -- http://mail.python.org/mailman/listinfo/python-list
Re: How to read such file and sumarize the data?
the number before @ is the process id in the linux server and it is identical. So i do NOT think distinguish each job's starting and ending time is difficult in this case. Huisky On Nov 17, 11:38 pm, Tim Harig wrote: > On 2010-11-17, huisky wrote: > > > I want to read this file and do the summarize how much total CPU time > > consumed for each user. > > Is Python able to do so or say easy to achieve this?, anybody can give > > me some hints, appricate very much! > > The question is, is the information you want available in the data. > > > > > Example log file. > > ** > > LSTC license server version 224 started at Sun Dec 6 18:56:48 2009 > > using configuration file /usr/local/lstc/server_data > > xyz 15...@trofast3.marin.ntnu.no LS-DYNA_971 NCPU=1 started Sun Dec 6 > > 18:57:40 > > 15...@trofast3.marin.ntnu.no completed Sun Dec 6 19:42:55 > > xyz 15...@trofast3.marin.ntnu.no LS-DYNA_971 NCPU=2 started Sun Dec 6 > > 20:17:02 > > 15...@trofast3.marin.ntnu.no completed Sun Dec 6 20:26:03 > > xyz 18...@trofast2.marin.ntnu.no LS-DYNA_971 NCPU=1 started Sun Dec 6 > > 21:01:17 > > 18...@trofast2.marin.ntnu.no completed Sun Dec 6 21:01:28 > > tanhoi 5...@iimt-tanhoi-w.ivt.ntnu.no LS-DYNA_971 NCPU=1 started Mon > > Dec 7 09:31:00 > > 5...@iimt-tanhoi-w.ivt.ntnu.no presumed dead Mon Dec 7 10:36:48 > > sabril 18...@trofast2.marin.ntnu.no LS-DYNA_971 NCPU=2 started Mon > > Dec 7 13:14:47 > > 18...@trofast2.marin.ntnu.no completed Mon Dec 7 13:24:07 > > sabril 18...@trofast2.marin.ntnu.no LS-DYNA_971 NCPU=2 started Mon > > Dec 7 14:21:34 > > sabril 18...@trofast2.marin.ntnu.no LS-DYNA_971 NCPU=2 started Mon > > Dec 7 14:28:42 > > 18...@trofast2.marin.ntnu.no killed Mon Dec 7 14:31:48 > > 18...@trofast2.marin.ntnu.no killed Mon Dec 7 14:32:06 > > I see starts, completes, kills, and presumed deads. The question is can > the starts be matched to the completes and kills either from the numbers > before @ or from a combination of the address and NCPU. You will need to > figure out whether or not you want to count the presumed deads in your > calculations. > > Assuming that the starts and stops can be corrilated, it is a simple matter > of finding the pairs and using the datetime module to find the difference > in time between them. -- http://mail.python.org/mailman/listinfo/python-list
Re: How to read such file and sumarize the data?
Thank you very much for your reply.
I think you just count the total number of NCPU used for each user.
And it does NOT show how much time used for each user.
Huisky
On Nov 18, 12:10 am, Steve Holden wrote:
> On 11/17/2010 4:45 PM, huisky wrote:> Say I have following log file, which
> records the code usage.
> > I want to read this file and do the summarize how much total CPU time
> > consumed for each user.
> > Is Python able to do so or say easy to achieve this?, anybody can give
> > me some hints, appricate very much!
>
> > Example log file.
> > **
>
> I'm assuming the following (unquoted) data is in file "data.txt":
>
>
>
> > LSTC license server version 224 started at Sun Dec 6 18:56:48 2009
> > using configuration file /usr/local/lstc/server_data
> > xyz 15...@trofast3.marin.ntnu.no LS-DYNA_971 NCPU=1 started Sun Dec 6
> > 18:57:40
> > 15...@trofast3.marin.ntnu.no completed Sun Dec 6 19:42:55
> > xyz 15...@trofast3.marin.ntnu.no LS-DYNA_971 NCPU=2 started Sun Dec 6
> > 20:17:02
> > 15...@trofast3.marin.ntnu.no completed Sun Dec 6 20:26:03
> > xyz 18...@trofast2.marin.ntnu.no LS-DYNA_971 NCPU=1 started Sun Dec 6
> > 21:01:17
> > 18...@trofast2.marin.ntnu.no completed Sun Dec 6 21:01:28
> > tanhoi 5...@iimt-tanhoi-w.ivt.ntnu.no LS-DYNA_971 NCPU=1 started Mon
> > Dec 7 09:31:00
> > 5...@iimt-tanhoi-w.ivt.ntnu.no presumed dead Mon Dec 7 10:36:48
> > sabril 18...@trofast2.marin.ntnu.no LS-DYNA_971 NCPU=2 started Mon
> > Dec 7 13:14:47
> > 18...@trofast2.marin.ntnu.no completed Mon Dec 7 13:24:07
> > sabril 18...@trofast2.marin.ntnu.no LS-DYNA_971 NCPU=2 started Mon
> > Dec 7 14:21:34
> > sabril 18...@trofast2.marin.ntnu.no LS-DYNA_971 NCPU=2 started Mon
> > Dec 7 14:28:42
> > 18...@trofast2.marin.ntnu.no killed Mon Dec 7 14:31:48
> > 18...@trofast2.marin.ntnu.no killed Mon Dec 7 14:32:06
>
> The line wrapping being wrong shouldn't affect the logic.
>
> $ cat data.py
> lines = open("data.txt").readlines()
> from collections import defaultdict
> c = defaultdict(int)
> for line in lines:
> ls = line.split()
> if len(ls) > 3 and ls[3].startswith("NCPU="):
> amt = int(ls[3][5:])
> c[ls[0]] += amt
> for key, value in c.items():
> print key, ":", value
>
> $ python data.py
> xyz : 4
> tanhoi : 1
> sabril : 6
>
> regards
> Steve
> --
> Steve Holden +1 571 484 6266 +1 800 494 3119
> PyCon 2011 Atlanta March 9-17 http://us.pycon.org/
> See Python Video! http://python.mirocommunity.org/
> Holden Web LLC http://www.holdenweb.com/
--
http://mail.python.org/mailman/listinfo/python-list
Time and date operation
Hi everyone,
Could you anybody shed lights to me? Say I have two dics.
>>> cstart
defaultdict(, {15424: ['Dec', '6', '18:57:40'], 552:
['Dec', '7', '09:31:00'], 15500: ['Dec', '6', '20:17:02'], 18863:
['Dec', '7', '13:14:47'], 18291: ['Dec', '6', '21:01:17'], 18969:
['Dec', '7', '14:28:42'], 18937: ['Dec', '7', '14:21:34']})
>>> ccompl
defaultdict(, {15424: ['Dec', '6', '19:42:55'], 18291:
['Dec', '6', '21:01:28'], 15500: ['Dec', '6', '20:26:03'], 18863:
['Dec', '7', '13:24:07']})
and I need to calculate the difference time if the key value is the
same in both dics.
thanks in advance
Huisky
--
http://mail.python.org/mailman/listinfo/python-list
Read time and date from a text file
Hi,
As a newbie, I posted my question here again.
say i have two dics read from a text file by 'split'.
>>> cstart
defaultdict(, {15424: ['Dec', '6', '18:57:40'], 552:
['Dec', '7', '09:31:00'], 15500: ['Dec', '6', '20:17:02'], 18863:
['Dec', '7', '13:14:47'], 18291: ['Dec', '6', '21:01:17'], 18969:
['Dec', '7', '14:28:42'], 18937: ['Dec', '7', '14:21:34']})
>>> ccompl
defaultdict(, {15424: ['Dec', '6', '19:42:55'], 18291:
['Dec', '6', '21:01:28'], 15500: ['Dec', '6', '20:26:03'], 18863:
['Dec', '7', '13:24:07']})
and I need to calculate the difference time if the key value is the
same in both dics.
Someone suggested me to use the module 'datetime', but I'm still
wondering how to make it work.
I mean how to assign ['Dec','6','21:01:17'] to a 'datetime' object and
then do the datetime operation.
>>>time=datetime.datetime(cstart[18291]) does NOT work.
thanks in advance
Huisky
--
http://mail.python.org/mailman/listinfo/python-list
Re: Read time and date from a text file
On Nov 24, 2:09 pm, Peter Otten <__pete...@web.de> wrote:
> huisky wrote:
> > As a newbie, I posted my question here again.
> > say i have two dics read from a text file by 'split'.
>
> Please don't start a new thread when you are still asking about the same
> topic.
>
> >>>> cstart
>
> > defaultdict(, {15424: ['Dec', '6', '18:57:40'], 552:
> > ['Dec', '7', '09:31:00'], 15500: ['Dec', '6', '20:17:02'], 18863:
> > ['Dec', '7', '13:14:47'], 18291: ['Dec', '6', '21:01:17'], 18969:
> > ['Dec', '7', '14:28:42'], 18937: ['Dec', '7', '14:21:34']})
> >>>> ccompl
>
> > defaultdict(, {15424: ['Dec', '6', '19:42:55'], 18291:
> > ['Dec', '6', '21:01:28'], 15500: ['Dec', '6', '20:26:03'], 18863:
> > ['Dec', '7', '13:24:07']})
>
> I think you should use a normal dict. A default value of 0 doesn't make much
> sense here.
>
> > and I need to calculate the difference time if the key value is the
> > same in both dics.
>
> > Someone suggested me to use the module 'datetime', but I'm still
> > wondering how to make it work.
> > I mean how to assign ['Dec','6','21:01:17'] to a 'datetime' object and
> > then do the datetime operation.
> >>>>time=datetime.datetime(cstart[18291]) does NOT work.
>
> Chris Rebert also suggested that you use the strptime() method. To spell it
> out a bit:
>
> >>> s = " ".join(cstart[18291])
> >>> s
> 'Dec 6 21:01:17'
> >>> datetime.datetime.strptime(s, "%b %d %H:%M:%S")
>
> datetime.datetime(1900, 12, 6, 21, 1, 17)
>
> You can learn about the format codes here:
>
> http://docs.python.org/library/time.html#time.strftime
>
> Note that strptime() assumes 1900 as the year which may lead to errors in
> leapyears and when start and completion time are in different years.
>
> Peter
Thanks a lot, Peter.
It helps a lot!
I see the problem of year. But the question is the source file does
NOT provide the year information.
for instance if i have one record as ['Dec','6','21:01:17'], and the
other as ['Jan','6','21:01:17']
these two records may be in different year. Will it be a problem to
use the 'datetime' do the time difference calculation?
regards
Huisky
Huisky
--
http://mail.python.org/mailman/listinfo/python-list
Re: Read time and date from a text file
On Nov 24, 2:45 pm, Peter Otten <__pete...@web.de> wrote: > huisky wrote: > > I see the problem of year. But the question is the source file does > > NOT provide the year information. > > for instance if i have one record as ['Dec','6','21:01:17'], and the > > other as ['Jan','6','21:01:17'] > > these two records may be in different year. Will it be a problem to > > use the 'datetime' do the time difference calculation? > > You have a problem that is independent of Python. You have to guess the > correct year to get the correct time interval. One approach would be to use > the year from the file's timestamp, then calculate the differences, and > whenever you get a negative interval add one year to the end date. > > This will still result in rare ValueErrors (Feb 29 in non-leapyears) and > some silent miscalculations; I fear you'll have to live with that. > > Peter I see, what i'm doing is only for personal usage. should be no problem at all. thanks a lot, Peter! -- http://mail.python.org/mailman/listinfo/python-list
Interactive mode under DOS?
Hi, I'm trying to use the interactive mode under DOS for Python 2.7. As a newbie, I do NOT know what is the following problem: >>>world_is_flat=1 >>>if world_is_flat: . . . print "be carefule to be not fall out!" File "", line 2 print "be carefule to be not fall out!" ^ IndenatationError : expected an idented block >>> Enybody knows how to fix this simple issue? regards -- http://mail.python.org/mailman/listinfo/python-list
Re: Interactive mode under DOS?
On Oct 24, 1:15 pm, Peter Otten <__pete...@web.de> wrote: > huisky wrote: > > Hi, > > > I'm trying to use the interactive mode under DOS for Python 2.7. As a > > newbie, I do NOT know what is the following problem: > > >>>>world_is_flat=1 > >>>>if world_is_flat: > > . . . print "be carefule to be not fall out!" > > File "", line 2 > > print "be carefule to be not fall out!" > > ^ > > IndenatationError : expected an idented block > > > Enybody knows how to fix this simple issue? > > > regards > > The lines that shall be executed when world_is_flat is True must be indented > more than the if. If you don't follow this rule Python complains and raises > an IndentationError. > > >>> if world_is_flat: > > ... print "be careful" > File "", line 2 > print "be careful" > ^ > IndentationError: expected an indented block > > The easiest way to indent in interactive mode is to hit TAB once: > > >>> if world_is_flat: > > ... print "be careful" > ... > be careful > > By the way, you should not retype your error messages because that is error > prone. Instead cut and paste. I think there is an entry in the system menu > of the dos window to help you do that. > > Peter Excellent reply! thanks very much Peter. -- http://mail.python.org/mailman/listinfo/python-list
